Investigating a System Hang during Long 4K Camera Recording

Good morning, everyone. Let's get started. This is the first triage call for the camera recording hang on Orion. I'm Tom from SQE, and I'll chair today. Soyeon from my team ran the reproduction over the weekend. We also have Kernel and BSP, Qorvix, and Lumicore on the line. Could everyone introduce themselves briefly?

Soyeon Park, SQE. I own the camera stress tests, and I have the dumps and the thermal logs ready.

Yuna Kwak, Kernel and BSP. Some of you were at the tech share on Friday, where I described an old camera hang. I'm a bit worried we're about to see a sequel.

Junho Kim, also Kernel and BSP. I maintain the camera driver stack on Orion, including our changes on top of the Qorvix image signal processor driver.

Ming Huang, Qorvix multimedia team, joining from Shenzhen. I support the image signal processor, the ISP, and its kernel driver.

Kenji Watanabe from Lumicore in Yokohama. We supply the main camera sensor for Orion. I'm a sensor application engineer, so I can answer questions about the sensor's behavior and its output interface.

Thanks. Soyeon, please walk us through what we've seen so far.

Sure. The issue was first reported by a reviewer who was recording a long concert. On Orion, if you record 4K video at sixty frames per second continuously, after somewhere between twenty-five and forty minutes, the preview freezes. The device stops responding to touch and buttons, and about ten to fifteen seconds later, it reboots. The video file being recorded is usually corrupted, so the user loses the whole recording.

Losing the recording is what makes this so painful. A crash at minute thirty-five of a concert is about as bad as it gets for a camera feature.

Exactly. In the lab, we've reproduced it fourteen times out of thirty attempts on ten devices. A few details are important. First, it never happens at 4K thirty frames per second or at 1080p, even after two hours. Second, it happens faster when the room is warm: at thirty degrees Celsius ambient, it took an average of twenty-six minutes; at twenty-two degrees, it took thirty-eight minutes. Third, on the reset reason, twelve were watchdog barks with a ramdump, and two were watchdog bites with no dump.

Do you have the sensor temperature at the time of the freeze? Our sensor reports its internal temperature through a register, and your camera software logs it, I believe.

We do. In every case, the sensor temperature was between seventy-four and seventy-nine degrees at the moment of the freeze. In the non-failing runs, it peaked around seventy-one degrees.

Yuna, Junho, what do the dumps show?

I followed the same order I described on Friday. The panic reason is the hardware watchdog bark. When I look at the CPUs, seven of the eight are waiting to acquire the same spinlock, and they're all spinning with interrupts disabled. The eighth CPU, CPU three, holds that spinlock, and it's inside the ISP driver's error recovery function, polling a hardware status register in a loop.

And that explains why the whole system freezes, not just the camera. Once CPU three holds the lock with interrupts off and never lets go, every other CPU that touches the camera path piles up behind it. Eventually, nothing can pet the watchdog. In the two bite cases, my guess is that the CPU handling the watchdog interrupt was also one of the CPUs spinning with interrupts disabled, so the bark couldn't be handled.

What's the error recovery function waiting for?

It's waiting for the ISP to report that its input pipeline has drained after an error. Just before the freeze, the kernel log shows a burst of errors from the camera serial interface receiver: checksum errors and frame synchronization errors on the link between the sensor and the ISP. The ISP raises an error interrupt, the driver enters recovery, takes the lock, and waits for the pipeline to drain. But the pipeline never drains, and there's no timeout in that loop.

So we have two layers again, like the storage panic last week. There's a trigger, the link errors at high temperature, and there's a software defect, an unbounded wait with interrupts disabled, that turns a recoverable error into a full system hang.

Kenji, does the sensor behave differently at those temperatures?

The sensor is specified to operate up to eighty-five degrees, so seventy-nine is within specification. However, I should be transparent that signal quality on the high-speed output link does degrade as temperature rises. At 4K sixty, the link runs at its highest data rate, which leaves the smallest margin. So I'm not surprised that errors appear only in that mode. But in our reference design, the error rate at seventy-nine degrees is extremely low, perhaps one corrupted packet per hour.

One corrupted packet per hour would be fine. The ISP is designed to drop a bad frame and continue. What we're seeing in these logs is different: dozens of errors within a few milliseconds. That looks more like the link losing synchronization completely, which the ISP treats as a fatal input error.

Complete loss of synchronization would surprise me with the sensor alone. I wonder whether the board layout plays a role. The trace length and the connector on Orion are different from our reference design. Has your hardware team measured the eye diagram on that link at high temperature?

I don't believe so, at least not at seventy-nine degrees. I'll ask them. But I want to be careful not to turn this into a hardware debate that takes a month. Even if the link is marginal, we can't change the board on devices that are already in customers' hands.

Right. Whatever the trigger is, the software has to survive it. The worst acceptable outcome for a link failure should be a few dropped frames, or at most a stopped recording with the file saved properly. A full system hang is never acceptable.

Ming, why doesn't the pipeline drain? Is there a way to force it?

When the input link loses synchronization in the middle of a frame, the ISP's input stage can get stuck waiting for the end of that frame, which never arrives. In that state, the drain never completes. The documented way out is a hardware reset of the ISP input block, followed by reprogramming the pipeline. Our reference driver does that after a timeout of one hundred milliseconds. It looks like the Orion driver lost that timeout at some point.

That's a little embarrassing, but I think you're right. I checked the history this morning. Last year, we restructured the recovery function to support a multi-camera mode, and the timeout was dropped during the rework. It wasn't caught in review because the old code was spread across two functions. I'll own that.

I'd go one step further than restoring the timeout. Polling a hardware register in a loop while holding a spinlock with interrupts disabled is risky even with a timeout, because one hundred milliseconds is a very long time for seven other CPUs to spin. I'd restructure it: mark the pipeline as being in recovery, release the lock, do the wait outside the lock with interrupts enabled, then reset and take the lock again only for the short reprogramming step.

That's a better design, and it's closer to what our newer driver generation does. The only complication is that other parts of the driver must check the 'in recovery' state and avoid touching the hardware during that window. I can share the pattern from our newer driver to save Junho some time.

How do the two options compare in risk and time? I'm thinking about the maintenance release schedule.

Restoring the timeout is a small change, maybe twenty lines, and I could have it ready tomorrow. Yuna's restructuring is better long-term, but it's a few hundred lines and touches locking across the driver. I'd estimate a week to write and another week of testing. My suggestion is to do both in sequence: the timeout fix goes into the September maintenance release as a quick mitigation, and the restructuring goes into October after proper testing.

I can support that, as long as the October change doesn't quietly slip to November. Temporary fixes have a way of becoming permanent.

There's another angle I'd like us to cover. Even with the timeout fix, the ISP reset will interrupt the video stream. What happens to the recording? If the camera app crashes and the file is still corrupted, users will still lose their concert.

Good point. After the reset, the driver reports an error to the camera service. As far as I know, the current app behavior is to stop the recording and show an error. Whether the file is finalized properly depends on the framework's recorder, which isn't my area.

Then we need the framework camera team in the loop. Soyeon, can you test what happens to the file when we inject a link error during recording on a build with the timeout fix? If the file is unplayable, that's a separate issue for the framework team, and it needs to be fixed in the same release.

For error injection, our ISP has a debug register that simulates loss of synchronization on the input link. I'll send Junho the instructions. It's much faster than heating the device in a thermal chamber for half an hour every time.

Ideally, the camera could also reduce the risk before the error happens. When the sensor temperature crosses seventy-five degrees, the camera app could warn the user or step down to 4K thirty frames per second. That's what several other device makers do with our sensor.

That's a product decision, so I'll raise it with the camera product owner rather than decide it here. But I'll include it in the summary as a recommendation from Lumicore.

Before we talk about validation, are any other products exposed? The ISP driver is shared across several of our devices.

The restructured recovery function from last year is used on Orion and on the upcoming Orion Fold. Orion Lite uses an older ISP and a different driver, so it isn't affected. Orion Fold is the bigger concern, because it supports 4K sixty on two of its cameras, and it has less room for heat to escape in the folded position.

Orion Fold also uses a newer revision of our ISP, but the input stage behaves the same way when synchronization is lost, so I would assume it's vulnerable until proven otherwise.

Then the timeout fix has to go into the Orion Fold launch software as well. I'll flag it to Sarah's launch team today, since they're tracking every change that goes into their final builds.

I'd also like to run the original thermal scenario on a few Orion Fold prototypes, folded and unfolded. If the fold position makes the sensor hotter, we'll see the issue sooner there, and that would be useful data for the hardware team too.

Let's agree on how we'll validate the fix. Soyeon?

I propose three steps. First, error injection on ten devices, a hundred injections each, with the timeout fix. Pass criteria: zero system hangs, and the device remains responsive. Second, the original scenario: one-hour 4K sixty recordings at thirty degrees ambient, twenty runs. Pass criteria: zero hangs, and if the recording stops, the file must be playable. Third, our standard camera regression suite to make sure we didn't break anything else.

For the error injection step, please also enable the lock debugging options on two of the devices. If there's any other path that waits too long with the lock held, lock debugging will tell us directly, instead of us finding it the hard way in the field.

Will do. With the build tomorrow, I can finish the first two steps by Thursday and the regression suite by Monday.

Let me summarize the action items. Junho: restore the timeout and reset path in the recovery function, test build by tomorrow. Ming: send the error injection instructions and the recovery pattern from your newer driver today. Yuna: design the restructured recovery with Junho, target the October release. Soyeon: run the three-step validation, results by Monday. Kenji: share your reference link measurements at high temperature, so our hardware team has a baseline. And I'll contact the hardware team about the link measurements and the camera product owner about the thermal step-down.

I'll send the measurement report by Wednesday. If your hardware team wants to compare setups, I can also join a call with them.

And I'll add a unit test that simulates a drain that never completes, so if anyone drops the timeout again in a future refactoring, the test fails immediately.

Excellent. That last point is exactly how we stop this from coming back. Thank you, everyone. Our next call is Thursday, August 13th, at 10 a.m. Korea time, which is 9 a.m. in Shenzhen and 10 a.m. in Yokohama. We'll review the error injection and the first long recording results.

Thank you, Tom. I'll be there, and I'll bring the recovery timing data from our own lab for comparison.

See you Thursday. And I think this case just earned a place on the wiki page, too.