Weekly Issue Status Review after the Orion Lite Maintenance Release

Good afternoon, everyone. Let's get started; we have a lot to cover and only forty-five minutes. I see SQE, Kernel and BSP, and Platform Framework are all here. Minji, I know you came straight from the Orion storage meeting, so thank you for making it.

No problem. I'll give a quick update on that one at the end, since it affects the common kernel branch.

Chris from Kernel and BSP. Yuna is with me today because she owns most of the power and scheduler issues on this release.

Hi, everyone. Yuna Kwak, Kernel and BSP.

Brian from Platform Framework, and Elena from my team is joining as well. She's been leading the camera and system server investigations.

Hello. Happy to be here, although I'd be happier with fewer open issues.

Ha, wouldn't we all. Here's the agenda. First, Minji will give the overall numbers since release. Second, each team walks through their top issues and where they stand. Third, we decide what goes into the 1.3.1 hotfix and lock the schedule. Minji, over to you.

Thanks, Grace. Orion Lite 1.3.0 went out to the first market wave on July 28th, so we have one week of data. Right now we have forty-seven open issues linked to this release: three critical, twelve major, and thirty-two minor. That's up from thirty-one issues at the end of last week, so the trend is still going the wrong way, but the rate of new issues is slowing down.

How many of the forty-seven are actually new regressions in 1.3.0, as opposed to old issues that we just happened to notice?

Good question. We checked each one against 1.2.4. Twenty-nine are confirmed regressions, eleven also reproduce on 1.2.4, and seven we haven't been able to classify yet. All three critical issues are confirmed regressions, which is why I'm worried.

In terms of field data, the crash rate for system processes went from 0.21 to 0.34 per thousand device hours after the update. Most of that increase comes from one issue, the camera ANR, which Elena will cover. Battery complaints in the user feedback channel also doubled, which lines up with the standby drain issue Yuna is looking at.

Let's start with the biggest contributor, then. Elena, where are we on the camera ANR?

We've made good progress. The ANR happens when the camera app is launched cold within a few seconds of the device unlocking. The camera service waits on a lock in the system server that's held by the new face unlock feature during its cleanup. On most devices it's released quickly, but on devices with a lot of installed apps, cleanup takes over five seconds, and the camera app hits the ANR timeout.

Do we have a fix?

We have a fix in code review. It moves the cleanup off the critical path, onto a background thread, so it no longer holds the lock while it iterates over packages. We've tested it on fifty devices with no ANRs so far. I'd like two more days of soak testing before I call it done.

I want to challenge the classification, though. Your team marked this as major, but it's the single biggest driver of our crash rate increase, and it's visible to every user who opens the camera after unlocking. From the SQE side, I'd call it critical.

I see where you're coming from, Minji. Our reasoning was that the app recovers after the ANR dialog, and no data is lost. Under our severity guidelines, that's major. But I take the point about user visibility. I don't feel strongly about the label, as long as it doesn't change the priority, because we're already treating it as our top item.

Let's upgrade it to critical, then, mainly for reporting purposes, so management sees it at the top of the list. Brian, any other framework items in the top tier?

One more. Bluetooth audio stutters with a specific family of third-party earbuds. We've traced it to a codec negotiation change in the Bluetooth stack. The stack is from our connectivity vendor, so we've sent them logs, and they've acknowledged the issue. Their estimate for a fix is August 12th, which is later than I'd like.

Is there a workaround on our side, in case the vendor fix slips?

There is a fallback. We can add those earbuds to a compatibility list that forces the older codec. Audio quality is slightly lower, but there's no stuttering. It's a one-line configuration change, and we can remove it once the real fix lands.

Good to have that in our back pocket. Let's move to kernel. Chris, Yuna?

I'll start with standby battery drain, since it's the most visible. After the update, some devices lose around eight percent overnight instead of the usual three. Our wakelock analysis shows the location service waking the device every ninety seconds when the user has a specific fitness app installed. The app didn't change, but the framework changed how location requests are batched in 1.3.0.

Wait, so this is actually a framework change? I thought it was assigned to kernel because of the wakelocks.

It started with us because the symptom showed up in the kernel wakeup sources. But the root cause is the batching change in the location manager. I sent the analysis to your team on Friday, but I'm not sure anyone picked it up.

That's on me. It probably fell through the cracks between the two queues. Let's reassign it to framework today, with you as a reviewer, Yuna, since you know the wakeup side better than anyone.

Thank you both. This is exactly why we have this meeting. Chris, what else is on the kernel list?

The third critical issue is a fingerprint unlock latency regression. Median unlock time went from about three hundred milliseconds to five hundred and fifty. We bisected it to a scheduler tuning change that was meant to save power by keeping small tasks on the efficiency cores. The fingerprint processing thread now starts on a slow core and takes too long to migrate.

Is the fix to revert the tuning, or to make an exception for the fingerprint path?

We'd prefer a targeted fix: we boost the fingerprint thread when a touch event lands on the sensor area, so it starts on a performance core. Reverting the whole tuning would undo a four percent standby power improvement. The targeted fix is ready and tested on twenty devices. Unlock time is back to three hundred and ten milliseconds.

Twenty devices is a small sample for a scheduler change. Before it goes into a hotfix, I'd want at least our standard performance regression suite on a hundred devices, because scheduler tweaks tend to have side effects in places you don't expect.

That's reasonable. If SQE can take the build tomorrow, the suite takes about two days, so we'd have results by Thursday.

Before we talk about the hotfix, I want to come back to the seven issues that haven't been classified yet. Minji, what's blocking those? I don't like having unknowns sitting in the list for more than a few days, because sometimes the scariest problems hide there.

Most of them are blocked on reproduction. Four came from user feedback with very little detail, things like 'phone got hot and restarted', and we haven't been able to reproduce them in the lab. Two are waiting for logs from the carrier lab in Europe. And one is a random reboot that we've seen exactly once on an internal dogfood device.

For the random reboot, did the device leave a ramdump or at least a last kernel log? Even a single occurrence can be very informative if we have the right data. If it was a watchdog reset, the reason code alone narrows it down a lot.

Unfortunately, the dogfood device had ramdump collection disabled, because it's someone's daily phone. We only have the reset reason, which says it was a hardware watchdog. I've asked the owner to enable collection, but he's not thrilled about it.

A hardware watchdog on its own could be many things, but it's worth correlating with the heat reports from users. If the device was thermally throttled hard enough, some threads might have been starved long enough to miss the watchdog kick. I can check whether the thermal logs from those four feedback reports show anything unusual.

Please do. Even a negative result is useful, because it lets us close the loop. Minji, can we set a rule that if an unclassified issue has no new data after two weeks, we close it as insufficient information and reopen it if it comes back?

I'm fine with two weeks, as long as the reboot and heat reports are excluded from that rule until Yuna's analysis is done. If there's a thermal pattern, it could become our next critical issue, and I don't want to close it prematurely.

On the field side, one more signal to watch: the app store reviews for our camera app dropped from 4.3 to 4.1 stars in a week. Most of the negative reviews mention the freeze after unlocking. Once the ANR fix is out, that's a good metric to confirm we've actually solved it for users.

Okay. Let's talk about the hotfix. My proposal is to include the three critical fixes: the camera ANR, the location batching fix for battery drain, and the fingerprint boost. Plus the Bluetooth compatibility workaround if the vendor fix isn't ready. Everything else waits for 1.4.0 in September. Thoughts?

I'd like to add two of the major issues as well. They're small, low-risk fixes in the settings app that are already merged on the main branch. Leaving them out means users wait another month for no good reason.

I'd be careful with that. Every extra change in a hotfix is something we have to test, and the testing window is already tight. Unless they're really trivial, I'd rather keep the hotfix as small as possible. Can you send me the diffs, and I'll make the call by tomorrow morning?

That's fair. I'll send them right after the meeting.

Here's the schedule I'd like to propose. Content cutoff on Tuesday, August 11th. SQE takes the release candidate on Wednesday the 12th, runs the hotfix validation suite through Friday, and if everything passes, the OTA starts on Tuesday, August 18th, with a staged rollout: one percent, then ten, then everyone over a week.

For the staged rollout, I'd like us to agree on the halt criteria now rather than debating them on the day. My suggestion: if the system crash rate in the one percent group goes above 0.25 per thousand device hours, or if we see any new critical issue, we pause and review before expanding.

Agreed. I'll write those criteria into the rollout plan so the on-call release manager can act on them without waiting for a meeting.

That works for kernel, assuming the performance suite comes back clean on Thursday.

Framework can make the eleventh. The location batching fix is the only one that worries me, since we just picked it up today, but it's a contained change.

Minji, you mentioned an update on the Orion storage issue. Does it affect us?

Briefly: Orion 2.1.0 is on hold because of a kernel panic in the storage driver. The trigger looks like a shortened storage suspend timeout combined with new firmware on Orion. The timeout change is in the common kernel branch, so it's also in Orion Lite. So far, we have zero storage panics on Orion Lite in a hundred devices, because the storage part is different. But if Kernel Storage decides to revert or change that timeout, it will land in the common branch, and I'd like it in our hotfix too, for consistency.

Makes sense. I'll sync with Jiwoo's team and make sure that if a fix comes out of their experiments this week, we pick it up before the cutoff on the eleventh.

Good. I'll flag it in the hotfix plan as a conditional item.

Let me recap the action items. Elena: complete the soak test of the camera ANR fix by Thursday and send the two settings diffs to Minji today. Brian: take ownership of the location batching fix, with Yuna as reviewer, and target merging by Monday. Chris: deliver the fingerprint boost build to SQE tomorrow. Minji: run the performance suite, report by Thursday, and decide on the settings fixes by tomorrow morning. Brian also tracks the Bluetooth vendor fix and prepares the fallback. I'll publish the hotfix plan tonight.

One request: can we add a rule that issues moving between teams get reassigned within one working day? The battery issue sat for three days just because it was between queues.

Good catch, Yuna. I'll add that to the triage guidelines and bring it up at the release council on Friday. Okay, our next status review is Tuesday, August 11th, at 2 p.m. Korea time, which is also the content cutoff day, so please come with your fixes merged or a clear reason why not. Thank you, everyone.

Thanks, Grace. See you next Tuesday.