Root-Causing a UFS Kernel Panic after the Orion Maintenance Release

Okay, it's two minutes past, so let's get started. Thanks, everyone, for joining on a Monday morning with such short notice. I'm Minji, the SQE lead for the Orion maintenance releases, and I called this meeting because we have a kernel panic that's reproducing in our stress lab, and I don't want it anywhere near the next OTA until we understand it. Let's go around quickly.

I'm Tom, also from SQE. I run the stress lab and I'm the one who has been collecting the ramdumps over the weekend, so if you have questions about test conditions, I'm your person.

Jiwoo Lee, Kernel Storage. I own the UFS driver integration on Orion, including the patches that went into 2.1.0. I've been looking at the dumps since Saturday night.

Hyunsoo Bae, also Kernel Storage. I mostly work on runtime power management for storage, which may turn out to be relevant today.

Hi, Laura Chen from Qorvix customer engineering in Taipei. I support the UFS host controller on your chipset. I've read the initial report but not the dumps yet.

Stefan Brandt, field application engineer at Kestrel Memory, joining from Munich. We supply the UFS storage parts for Orion, and the firmware update in this release came from us, so I suspect I'll be getting some questions.

Thank you all. One ground rule before we start: this meeting is about facts and next experiments, not about who is to blame. We have three components that changed in the same release, so it's very tempting to point at each other. Let's not do that. Tom, can you walk us through what we're seeing?

Sure. The build is ORN.2.1.0, which went to our lab on July 28th. We run a 72-hour stress profile on two hundred devices: heavy file I/O, camera recording, app installs, and frequent screen on and off cycles. Over the first run, eleven devices hit a kernel panic, which is a rate of five and a half percent per 72 hours. On the previous release, 2.0.3, we had zero panics on the same profile.

Just to clarify, are all eleven panics the same signature, or are we looking at several different crashes that happen to show up together?

Good question. Nine of the eleven have the same signature: a NULL pointer dereference in the UFS completion handler, specifically in the function that completes a request after the controller signals it's done. The other two are different; one is a watchdog bite in the display driver and one is an out-of-memory panic. I'd treat those two as separate issues for now.

Agreed. Let's park those two and focus on the nine. Tom, is there any pattern in when they happen?

There is. In seven of the nine, the panic happens within two seconds of the screen turning off, while there's still heavy write traffic in flight. In the other two, the logs are truncated, so we can't say for sure. We also noticed that all nine devices have the same storage firmware version, which is the new one from Kestrel. But to be fair, every device in this run has that firmware, so that alone doesn't prove anything.

Thank you for saying that, Tom. I'd just like to add that the new firmware has been in mass production with two other customers since May, with no reports of this kind of crash. That's not proof either, but it's a data point.

Jiwoo, you've been in the dumps all weekend. What does the analysis tell us so far?

Here's what I can say with confidence. In all nine dumps, the completion handler is processing a request whose tag has already been released. In other words, the controller tells us request number seventeen is done, but by the time we look it up, slot seventeen is empty. The pointer we dereference is the request structure for that empty slot, which is why we get a NULL pointer.

So it's a double completion, or at least a completion arriving after the request was already cleaned up. Do you know who released the tag first?

That's the key question, and I don't have a definitive answer yet. What I can see is that in six of the nine dumps, the UFS error handler ran within the previous few hundred milliseconds. The error handler aborts outstanding requests and releases their tags. So my working hypothesis is a race: the error handler aborts request seventeen and frees the slot, and then a late completion interrupt for the same request arrives from the controller.

And that would explain the screen-off timing. When the screen turns off, we start the runtime suspend sequence for storage. If there's heavy write traffic, the suspend request has to wait, and if it waits too long, it times out. That timeout is one of the triggers for the error handler. So the chain would be: screen off, suspend attempt, timeout, error handler, abort, and then a late completion hits an empty slot.

That's a plausible chain. But why would it start in 2.1.0? What changed in that path?

Two things changed. On the kernel side, we merged a patch that shortened the suspend timeout from five hundred milliseconds to two hundred, to improve standby battery life. And on the firmware side, Kestrel's update enabled a larger write buffer, which means the device can take longer to flush data before it's ready to suspend.

I'd push back slightly on the framing there. A larger write buffer does increase the worst-case flush time, that's true, but it's still well within the specification. According to the standard, the host is supposed to tolerate flush times of up to one second in that state. If the kernel gives up after two hundred milliseconds, I'd argue that's a host-side timing issue rather than a firmware problem.

That's a fair point about the timeout, and honestly, I was the one who proposed the shorter value. In our testing with the old firmware, flushes never took more than eighty milliseconds, so two hundred looked very safe. We clearly didn't retest it with the new firmware. That's on us.

Let me add one thing from the controller side, because I don't think the timeout is the whole story. Even if the timeout is too short, the driver should never crash because of a late completion. The host controller can legitimately report a completion for a request that the software has already aborted. The completion path needs to check whether the tag is still valid before using it. If it doesn't, that's a robustness bug in the driver, regardless of the timing.

I agree with Laura. We may have two problems layered on top of each other: a timeout that's too aggressive, which makes the error handler run far more often, and a missing validity check, which turns a recoverable situation into a panic. Even if we fix the timeout, I'd want the validity check in place, because there are other ways to trigger the error handler.

Okay, this is helpful. We have a hypothesis, but we haven't proven it yet. What experiments would confirm or rule it out, and how quickly can we run them?

From the lab side, I can split the next run into three groups of sixty devices each. Group A: current build, unchanged, as the control. Group B: the timeout restored to five hundred milliseconds. Group C: the old storage firmware with the current kernel. If the hypothesis is right, B and C should both drop to near zero, and A should keep crashing at about the same rate.

That's a clean design. I can have a test build with the restored timeout ready by this afternoon. It's a one-line change, so the risk is very low.

For group C, downgrading the firmware is possible, but please be aware that the downgrade tool resets some of the device's internal health counters. It won't affect the test, but I'd recommend you don't ship those units to anyone afterward. I can send the tool and the old image within the hour.

In parallel, I'd like to add a debug patch to group A that logs every tag allocation and release with a timestamp, and checks validity in the completion path. Instead of panicking, it would print a warning and dump the state. That would give us direct evidence of the race rather than inferring it from the dumps.

Wait, if we change group A, it's no longer a clean control group. I'd rather keep A exactly as shipped, and put the debug patch on a fourth, smaller group. Tom, can we spare forty more devices?

It's tight, but yes. I can borrow forty from the Atlas pre-validation pool for three days, as long as I give them back by Thursday.

If you capture the controller register state in the debug dump, please send it to me directly. I can check whether the controller reported the completion before or after the abort command was acknowledged. That timing will tell us whether this is purely a software race or whether the controller is behaving unexpectedly.

Will do. I'll capture the doorbell register, the task management registers, and the interrupt status for the last fifty events before each warning. If that's not enough, tell me what else you need and I'll add it to the next debug build rather than guessing.

That should be plenty for a first pass. Thank you.

Now, the uncomfortable part. ORN.2.1.0 is scheduled for OTA to the first market wave on August 6th. Given what we've seen, SQE cannot sign off on that build. I'd like to know if anyone disagrees with holding the OTA.

No disagreement from me. A five percent panic rate in the lab is far too high to ship, even if real users don't stress the device the way our lab does. I'd rather explain a delay than explain field crashes.

Same here. And if the experiments confirm the timeout theory, the fix is small, so the delay could be short. We could realistically have a candidate build by the end of the week.

Good. I'll inform the release management team that the August 6th OTA is on hold pending root cause, and that we expect to give a new date on Wednesday. I'd rather give them a realistic date once than move it three times.

There's one more question we should answer before we close this topic. Are any other models affected? The shorter timeout patch went into the common kernel branch, so it's also in the Orion Lite build that's in our lab right now. Orion Lite uses a different storage part, but from the same supplier.

The Orion Lite part uses an older controller generation, and it doesn't have the larger write buffer feature at all, so the flush times should be similar to the old firmware. I'd be surprised if you saw the same problem there. That said, I'd rather you verify it than take my word for it.

Fair enough. Orion Lite has been running the same stress profile since Friday, and so far we have zero storage panics on a hundred devices. I'll keep it running and flag it immediately if that changes.

Good. So for now, the hold applies to Orion only, and Orion Lite stays on its current schedule unless the data tells us otherwise. Let's write that down explicitly so nobody gets confused later.

Let me capture the action items. Hyunsoo: test build with the timeout restored, ready by 4 p.m. today. Jiwoo: debug build with tag tracing and the validity check, by end of day. Stefan: send the firmware downgrade tool and the previous image within the hour. Tom: start the four-group stress run tonight and share interim results on Wednesday morning. Laura: review controller register dumps as soon as Jiwoo sends them. And I'll handle communication with release management. Did I miss anything?

One small addition. Could someone share the exact kernel commit for the timeout change? I'd like to compare it with our reference driver, because our reference value is actually one second, not five hundred milliseconds.

Interesting, I didn't know that. I'll send you the commit right after this call. If the reference is one second, we should probably understand why we ever moved away from it.

And from our side, I'll ask our firmware team for the worst-case flush time distribution with the new buffer size, measured on your configuration. That should tell us how much margin any timeout value actually gives you.

Excellent. Thank you, everyone, for keeping this constructive. We're meeting again on Wednesday, August 5th, at 9 a.m. Korea time, which is 2 a.m. in Munich, so Stefan, I'll send you a written summary instead and we'll call you only if the firmware group looks suspicious. Laura, 8 a.m. Taipei time should work for you.

That works. Talk to you on Wednesday.

Thank you, Minji. I appreciate the summary, and please call me anytime if you need the firmware team involved.

I'll post the run status in the channel every morning, so nobody has to wait until Wednesday to see the trend.