Tech Share on Debugging Lockups with Watchdogs and Ramdumps

Okay, let's get started. Welcome to the first of our Friday tech shares. I see people from four teams, which is exactly what I hoped for. The idea is simple: once a week, someone shares a technique they use in their daily work, and we have an open discussion afterward. No slides are required, and questions are welcome at any time. Today, Yuna and I will talk about debugging lockups, because it's the type of issue where teams waste the most time pointing at each other.

And it's also the type of issue where a little bit of knowledge goes a long way. Most lockups can be narrowed down to the right team in fifteen minutes if you know where to look in the dump. Our goal today is that by the end of this hour, everyone here can at least do that first triage step on their own.

That would make my life much easier. Right now, when the stress lab sees a hang, we basically attach the dump and assign it to kernel by default. I suspect that's not always fair.

It's not, but don't worry, we're not keeping score. Let's go around very quickly so I know who's in the room. Elena, Seojin?

Elena from Platform Framework. I mostly debug system server hangs, so I'm curious where the line is between a framework hang and a kernel lockup.

Seojin, Modem Software. We just had an incident where a debug restart policy ended up in a production build, so I'm here to understand the policies better, honestly.

Great, Elena's question is a perfect starting point. Let's define our terms, because people use 'hang' to mean five different things. At the bottom, we have a hard lockup: a CPU is stuck with interrupts disabled, so it can't even respond to the timer. Above that, a soft lockup: a CPU is stuck in kernel code with interrupts enabled, but it hasn't let the scheduler run anything else for more than twenty seconds, by default.

Then there's a hung task, which is different. The CPU is fine, but a specific task has been sleeping in an uninterruptible state for too long, usually waiting for a lock or for I/O that never completes. The CPU is happily running other things. It's just that one task, and everything waiting for it, is stuck.

And the system server watchdog sits on top of all that, right? When our watchdog fires, it's because a thread in the system server didn't check in for sixty seconds.

Exactly. And here's the key insight: a framework watchdog can be caused by any of the layers below it. If a kernel driver holds a mutex forever, the system server thread that needs it shows up as a hung task, and sixty seconds later, the framework watchdog fires. The framework is the messenger, not necessarily the culprit.

Can I ask about the hardware watchdog? In our reset reasons, we see 'watchdog bark' and 'watchdog bite'. I've always assumed they're the same thing.

Good question, and it's a common confusion. The hardware watchdog is a timer that the kernel has to pet regularly. If it isn't petted in time, first it barks: it raises an interrupt, and the kernel gets one last chance to save its state, usually by triggering a panic and collecting a ramdump. If the kernel can't even handle that interrupt, because the CPU is completely locked, the watchdog bites: it resets the chip immediately, and you often get very little information.

So a bite is a much worse sign than a bark, from a debugging point of view.

In terms of how much data you get, yes. A bite usually means interrupts were disabled on the CPU that should have handled the bark. So the reset reason alone already tells you to look for code that disables interrupts for too long, which narrows things down a lot.

Is there any way to get more information out of a bite? It feels like those are the cases where we need it most.

There are a couple of tricks. On most chipsets, the bootloader can preserve a small region of memory across the reset, and the kernel writes its log there continuously. So after a bite, the next boot can still recover the last few seconds of kernel messages. It's not a full dump, but it often shows which driver was active right before the reset. We call it the last kernel log, and SQE should always attach it.

Noted. I'll make that mandatory in our filing template.

Let me make this concrete with a real case from Orion last spring. The symptom was that the screen froze during a video call, and after about sixty seconds, the device rebooted. SQE filed it as a display issue, and the display team spent a week on it. When I finally looked at the dump, here's the order I followed, and I'd recommend everyone use the same order.

Step one: read the panic reason and the last kernel log lines. In this case, the panic was triggered by the framework watchdog, which killed the system server, and the log said the main thread was blocked. That tells us the framework detected the problem, but not who caused it. Step two: look at what the blocked thread is waiting on. Here, it was waiting in the kernel, inside the binder driver, for a reply from the camera hardware service.

So you follow the chain of who is waiting for whom, until you find the thread that isn't waiting for anything but is still stuck.

Exactly, that's step three. The camera service thread was waiting on a mutex in the camera driver. The crash utility can tell you the owner of a mutex, so I looked up the owner, and it was a kernel worker thread that was running, not sleeping, on CPU five. When I looked at that CPU's stack, it was spinning in a loop inside the image processor driver, waiting for a hardware status bit that never got set.

So in the end, it was the camera driver, not the display. The display just stopped updating because the whole framework was blocked behind it.

Right. And the fix was a timeout on that polling loop plus a hardware reset path for the image processor. The display team had done nothing wrong. With the right order, that triage takes about fifteen minutes. Without it, it took a week and three teams.

A quick question about step two. Binder transactions cross process boundaries. How do you find which thread in the other process is handling the call? In the modem service, we've had cases where we couldn't figure that out.

The binder driver keeps a record of every outstanding transaction, including the source and target threads. In the dump, you can print the binder state, and it shows something like: thread A in process X is waiting for transaction number such-and-such, which is being handled by thread B in process Y. It's one of the most useful things in the whole dump, and very few people know it's there.

That would have saved us days last month. Could you add the exact command to the wiki page?

Absolutely. That's actually one of today's action items.

Now for the discussion part. There's a configuration option that makes the kernel panic, and therefore reboot, as soon as a soft lockup is detected. In our engineering builds, it's on. In user builds, the ones customers get, it's off. Yuna and I disagree on whether it should be on for user builds, and we thought it would be fun to argue about it in public.

My position is that it should be on. Today, when a soft lockup happens on a customer device, the kernel prints a warning and keeps going. Sometimes the device recovers, but often it ends up half-dead: the screen is frozen, the user has to hold the power button, and we get no dump at all. If we panicked immediately, we'd collect a ramdump, the device would reboot in about a minute, and we'd actually be able to fix the root cause.

And my position is that it's too aggressive for customers. Not every soft lockup is fatal. Some are caused by very long but legitimate operations, like a huge memory compaction on a device under pressure. In those cases, the system recovers on its own after a few seconds. If we panic, we turn a short stutter into a full reboot, which is far worse for the user. A reboot also shows up in our quality metrics, while a stutter doesn't.

Isn't that last point a bit of a red flag, though? If we're choosing a policy partly because it keeps a metric looking good, we might be hiding real problems from ourselves.

Fair challenge, and I'll take that back. The metric shouldn't drive the decision. The user experience should. And for the user, a two-second stutter is better than a sixty-second reboot.

From the framework side, I'd add that the half-dead state Yuna described is the worst outcome of all. When the kernel recovers partially, the system server often ends up in a weird state, and we see strange bugs for hours afterward. A clean reboot is sometimes kinder than limping along.

Is there a middle ground? For example, don't panic on the first soft lockup, but if the same CPU is still stuck after, say, sixty seconds, then panic. Short stalls recover on their own, and real lockups get a dump.

I actually like that a lot. It addresses Daniel's concern about legitimate long operations, and it addresses my concern about half-dead devices. We'd need to check whether the kernel supports that kind of escalation out of the box, or whether we'd have to carry a patch.

I can live with that, as long as we roll it out carefully. Let's not change the policy on all devices at once. We could enable it on the internal dogfood population first, compare the reboot rate and the number of useful dumps for a month, and then decide based on data rather than on who argued more convincingly today.

SQE would be happy to help with the comparison. We can mirror the setting in the stress lab too, which would give us a much faster signal than dogfood alone.

Before we close, I want to go back to my original reason for being here. After our incident, I'm nervous about any debug-related setting reaching a production build. How do we make sure the ramdump collection policy itself is correct for each build type?

Good question. In user builds, a full ramdump is disabled, because it can contain personal data, and it's too big to upload anyway. What we collect instead is a minidump: only the kernel log, the registers and stacks of each CPU, and a few key data structures, with all user memory excluded. It's usually a few megabytes, and in my experience, it's enough to triage about seventy percent of lockups.

And for the remaining thirty percent, we usually need the lock owner information, which isn't in the minidump today. That's something I'd like to add: the list of tasks in uninterruptible sleep and what each one is waiting on. It's small, it doesn't contain user data, and it would have made the camera case I described solvable from a minidump alone.

If you add that, please include the binder transaction state as well. Half of the framework hangs I see are blocked binder calls, and that's exactly the information that would point me to the right process.

Noted. We'll need a privacy review for any change to what's collected, even if it's just kernel metadata, so let's propose both additions together in one review.

Let's turn this into action items. Yuna: write a wiki page with the four-step triage order and the exact crash utility commands, including the binder state, by next Wednesday. Me: check whether the kernel supports escalating a soft lockup to a panic after a configurable delay, and draft a proposal for the dogfood experiment. Tom: set up a matching configuration in the stress lab once we have the proposal. Elena and Yuna: draft the minidump additions for privacy review together.

And from SQE, I'll update our triage checklist so testers look at the reset reason and the blocked thread before assigning an issue to a team. If it saves even one wrong assignment a week, it's worth it.

I'll share the release check we're adding for debug settings in carrier configurations. It might be useful for kernel configuration options too.

Thank you all. This was more lively than I expected, in a good way. Next Friday, August 14th, same time, 4 p.m. Korea time in the large meeting room, we'll have a session on the generic kernel image and how vendor modules should be split. That one will probably be even more controversial, so bring your opinions.

I'll be there. And thanks for the binder tip; I'm going to try it on a hang I've been stuck on since Tuesday.

Let me know how it goes. If it works, that's a great example for the wiki page.