Tracking Down a Hard Lockup during Fold and Unfold Transitions

Good morning, everyone, and thanks for joining on a Saturday. I know it's a public holiday here in Korea, so I appreciate it even more. I'm Hyemi, the SQE lead for Orion Fold. We have a hard lockup during fold transitions that I don't want anywhere near the factory build, which is due on August 28th. Let's do quick introductions.

Woojin Choi, Display Software lead. My team owns the display driver and the logic that switches between the cover screen and the main screen.

Jisu Han, also Display Software. I wrote most of the power management code for the two display pipelines, so if this is a power sequencing issue, it's probably my code.

Lena Hoffmann from Qorvix display engineering, joining from Munich. I support the display controller and the interconnect on your chipset.

Takeshi Mori from Lumina, in Tokyo. We supply both panels on Orion Fold, and I'm responsible for the panel driver IC timing.

Here's what we know. We have six folding robots that open and close devices continuously, about four times a minute, while running a mix of apps. Over the last week, we ran just over forty thousand fold cycles. We had a hundred and twenty-two freezes, so about 0.3 percent per fold. The screen freezes in the middle of the transition, and about ten seconds later the device reboots.

Point three percent per fold sounds small, but a typical user folds the device around eighty times a day. That would mean a freeze roughly every four days for every user. That's clearly a launch blocker.

Exactly. And the hardest part is the reset reason. In every single case, it's a watchdog bite, not a bark. So we have no ramdump. All we have is the last kernel log that survives in the preserved memory region.

As Yuna explained at the tech share last week, a bite usually means the CPU couldn't even handle the bark interrupt. In display code, the most common cause I know of is a CPU getting stuck on a register access, not in a software loop.

Can you explain that for those of us who aren't hardware people? How can reading a register make a CPU hang?

I can take that one. When the CPU reads a register in the display controller, the request travels across the chip's interconnect to that block. If the block's clock is turned off or its power domain is collapsed, nobody answers the request. On some chips, the CPU then waits forever for a response that never comes. It's stuck at the hardware level, so it can't take interrupts, and the watchdog eventually bites.

Woojin, what do the last kernel logs show?

We have usable logs from ninety-eight of the hundred and twenty-two freezes. In every one, the last messages are from the display switching sequence. The pattern is consistent: the main display pipeline starts powering up, the cover display pipeline starts powering down, and then the log simply stops. The last line is usually our debug message saying the cover pipeline's power domain is being turned off.

That fits the theory perfectly. If something tries to read a cover pipeline register right after its power domain goes off, the CPU would hang exactly at that point. The question is what would touch the cover pipeline after we've decided to turn it off.

Could it be related to the panel's tearing effect signal? Our panels send a signal at the start of each frame, and your driver uses it to time the next update. If the cover panel sends one last signal while it's shutting down, the interrupt handler might try to read the controller's status.

That's a very good thought. The interrupt handler does read a status register to check which frame completed. And it runs on whichever CPU receives the interrupt, independently of the switching sequence. So there's a window where the handler could fire after the power-off has started.

But we disable that interrupt before powering down. That's the first thing the power-off sequence does. I checked that code carefully when I wrote it.

Disabling the interrupt stops new interrupts from being delivered, but it doesn't wait for a handler that's already running on another CPU. If the tearing signal arrived a few microseconds before you disabled it, the handler could still be in progress on CPU two while CPU zero turns off the power. You'd need to synchronize, to wait for any in-flight handler to finish, before collapsing the domain.

That's a classic mistake, and I think you're right. Let me check whether we call the synchronization function after disabling the interrupt. If we don't, that's almost certainly our bug.

This is a promising theory, but we're still guessing from the last line of a log. Is there any way to turn this kind of hang into something we can actually see in a dump?

Yes. Our interconnect has a debug feature called access timeout detection. When it's enabled, if a request to a powered-off block doesn't get a response within a set time, the interconnect returns an error to the CPU instead of letting it wait forever. The kernel then gets a bus error exception, which normally leads to a panic with a full register dump, including the exact address that was being accessed.

That would be perfect. We'd go from guessing to knowing exactly which register and which code path caused it. Why isn't it enabled by default?

It adds a small amount of latency to every access through the interconnect, and in production, some customers prefer a silent hang and reboot over a panic with a dump. But for debugging, it's invaluable. I'd recommend enabling it on the robot devices right away. I'll send the configuration this afternoon.

Great. With six robots and a freeze every three hundred folds or so, we should see several dumps within a day. I'll also keep two robots on the original configuration as a control, so we can confirm the timeout detection doesn't change the failure rate in some unexpected way.

On our side, I'd like to measure the timing of the tearing signal during shutdown. I suspect our panel may send one more signal than your driver expects, about sixteen milliseconds after the display-off command. If that's true, it widens the window for the race.

That's useful, but I'd be careful not to treat that as the root cause. Even if the panel sends an extra signal, our driver should handle it safely. The fix has to be on our side regardless.

I agree completely. I'm just trying to explain why it happens so often, so you can test the fix under realistic timing.

Let's assume for a moment that the theory is confirmed by tomorrow. What does the fix look like, and how risky is it?

There are two layers again. First, after disabling the interrupt, we call the synchronization function so any in-flight handler finishes before we power down. That closes the race. Second, as defense in depth, the interrupt handler should check a flag that says the pipeline is powering down, and return immediately without touching any registers.

Both changes are small. Maybe thirty lines total. The risk is in timing: the synchronization call can wait a few hundred microseconds if a handler is running, which slightly extends the fold transition. I don't expect anyone to notice, but we should measure it.

I'd add a third check. Before any register access in the display driver, verify that the power domain is actually on, at least in debug builds. Our reference driver has a helper for this. It won't run in production, but it will catch similar mistakes in other code paths during development.

I like that a lot. It turns a silent hardware hang into a loud software warning, which is exactly what we want during development.

Now the schedule. The factory build cutoff is August 28th. That's thirteen days from today. Walk me through how we get there.

Today and tomorrow: robots run with access timeout detection to confirm the theory. Monday: Jisu has the fix ready for review. Tuesday: a test build goes on all six robots. If we see zero freezes over forty thousand folds, which takes about a week, we'd have confidence by around August 25th. That leaves three days of margin.

Forty thousand folds with zero failures, when the failure rate was one in three hundred, gives us strong statistical confidence. But I'd like to add more robots if possible. Can we borrow the two from the hinge durability lab?

I'll ask them. They're running a mechanical test that doesn't need the latest software, so they may be able to switch two units over.

I'll have the tearing signal measurements by Tuesday, so you can configure one of the robots to fold at the most dangerous timing if needed.

I also need a fallback before I talk to the launch team on Monday. If the theory turns out to be wrong, or the fix doesn't hold up on the robots, what's our plan B? I don't want to walk into that meeting with only one option.

The bluntest option would be to delay powering down the cover pipeline. Instead of turning it off immediately when the main screen comes on, we keep it powered for an extra hundred milliseconds. Any late tearing signal or in-flight handler would finish while the block is still alive. It costs a little power on each fold, but it would almost certainly hide the problem.

'Hide' is the right word, though. A delay narrows the window, but it doesn't close it. Under heavy load, a handler could still be delayed by more than a hundred milliseconds, and you'd be back to a hang, just much more rarely, which would make it even harder to debug in the field.

Agreed. I'd only use the delay together with the real fix, never instead of it. But as a fallback for the factory build, if the synchronization fix shows any side effects, a delay is a much better outcome than a freeze every four days.

So for the launch team, I'll present it like this: plan A is the synchronization fix plus the handler check, validated on the robots by the twenty-fifth. Plan B adds the power-off delay as an extra margin if plan A shows anything unexpected. In both cases, we meet the August 28th cutoff.

If it helps, we can also adjust the panel's shutdown sequence in a future panel firmware update, so it stops sending tearing signals immediately after the display-off command. It won't make the factory build, but it would remove one of the triggers for later production batches.

That's a good long-term improvement. Let's track it separately so it doesn't distract from the factory build. Takeshi, could you open a ticket on your side and share the number with Woojin, so both teams are following the same item?

One more suggestion. This pattern, disabling an interrupt without synchronizing before powering down a block, is easy to repeat in other drivers. It might be worth a quick audit of the camera, video, and touch drivers on Orion Fold, since they also power blocks up and down frequently.

Good point. A simple search for places where we disable an interrupt and then turn off a power domain or clock in the same function would find most candidates. I'll ask Junho's camera team and the touch team to look at their drivers this week.

And let's enable the access timeout detection on all Orion Fold engineering builds until launch, not just on the robots. If there's another one hiding somewhere, I want a dump, not a bite.

Let me summarize. Lena: send the access timeout configuration today. Hyemi, that's me: enable it on all robots tonight and share the first dumps tomorrow. Jisu: confirm whether the synchronization call is missing and prepare the fix, plus the power-domain check helper, by Monday. Woojin: coordinate the driver audit with the camera and touch teams, and measure the transition time with the fix. Takeshi: tearing signal timing measurements by Tuesday.

I'll actually check the code right after this call. If the call is missing, I'll post a message in the channel within the hour, so nobody spends the weekend wondering.

Sounds good. I'll also include instructions for decoding the bus error information in the dump, since the format isn't obvious.

Thank you all, especially for giving up part of your holiday. Our next call is Monday, August 17th, at 5 p.m. Korea time, which is 10 a.m. in Munich and 5 p.m. in Tokyo. We'll review the first dumps and Jisu's fix.

Thank you, Hyemi. See you on Monday. If my measurements are ready earlier, I'll send them to Jisu directly, so the robots can start testing the worst-case timing as soon as possible.

And if the first dumps show an address outside the display controller, please send them to me right away. It would mean another block is involved, and I'd want to bring in the right colleague from our side before Monday.

Thanks, everyone. Fingers crossed that the first dump points exactly where we think it does. And if it doesn't, at least we'll finally be looking at real evidence instead of the last line of a log.