Release Readiness Review for the Fixed Orion Maintenance Build

Good afternoon, everyone. I think we're all here. Today's meeting has one clear purpose: decide whether Orion 2.1.1 is ready to replace the 2.1.0 build that we put on hold last week, and if so, when it goes out. Each team will report status, then we'll go through the open issues, and we'll finish with a go or no-go recommendation that I'll take to the release council tomorrow morning.

Minji, SQE. Tom is with me; he ran the stress campaigns on the candidate build.

Jiwoo and Hyunsoo from Kernel and BSP. We'll cover the storage fix, and Hyunsoo has the power numbers.

Seojin, Modem Software. I'll give an update on the Iberian reboot issue, since part of the long-term fix may affect this build.

Thanks. Let's start with the main reason this build exists. Jiwoo, where did we land on the storage panic?

We have a confirmed root cause and a verified fix. The four-group experiment we designed last Monday gave us very clean results. The control group, running the unchanged 2.1.0 build, had six panics on sixty devices. The group with the original five-hundred-millisecond timeout had zero. The group with the old storage firmware also had zero. And the debug group showed exactly the race we suspected: in nine separate events, a late completion arrived after the error handler had already aborted and released the request.

So both layers of the hypothesis were confirmed. What did we actually change in 2.1.1?

Two changes. First, the suspend timeout goes to one second, which matches the Qorvix reference driver and the storage specification. Second, the completion path now checks whether the request is still valid before touching it. If it isn't, we log a warning and ignore the completion instead of crashing. Laura from Qorvix reviewed both changes, and she confirmed the register dumps show the controller behaving within specification. So this was purely a software issue on our side.

And the stress results on the 2.1.1 candidate are clean. We ran the full seventy-two-hour profile on two hundred devices, and we had zero storage panics. We did see the validity check fire three times in the logs, which tells us the late completions still happen occasionally, but now they're harmless.

That's exactly the kind of evidence I like to see: not just the crash going away, but proof that the protection is doing its job.

On the battery side, since the shorter timeout was originally introduced to save standby power, I measured the impact of going to one second. Overnight standby drain went from 2.6 percent to 3.0 percent on our reference devices. That's a small regression compared to 2.1.0, but it's still better than 2.0.3, which was at 3.2 percent. I think it's acceptable for now, and I'll look for a smarter way to save that power later.

From SQE's perspective, that trade-off is clearly right. Nobody complains about a 0.4 percent difference overnight. Everyone complains about a crash.

Good. Minji, can you give us the overall picture of open issues on the 2.1.1 candidate?

Sure. We have twenty-three open issues found on 2.1.1. Zero are critical, two are major, and twenty-one are minor. Of the twenty-one minor ones, eighteen also exist on the current production build, 2.0.3, so they're not regressions. The other three are cosmetic issues in the settings app, and the framework team has fixes queued for 2.1.2.

And the two major issues?

The first one is the display driver watchdog bite that showed up alongside the storage panics last week. We've seen it twice more since then, so three times in roughly fourteen thousand device hours. It also reproduces on 2.0.3 at a similar rate, so it's not new. The display team believes it's related to a rare timing issue during panel power-off, and they're still collecting data.

The second major issue is the out-of-memory panic, which we've seen once on 2.1.1 and twice on 2.0.3, always after more than sixty hours of continuous stress with dozens of apps open. It's real, but it's hard to imagine a normal user hitting that pattern. Neither of the major issues is a regression, which is the key point for this decision.

Agreed. Our rule for maintenance releases is that we don't ship new critical or major regressions, and we don't make existing issues worse. It sounds like 2.1.1 meets both conditions.

Now the harder question. Yesterday's triage on the camera recording hang produced a timeout fix. Should it go into 2.1.1, or wait for the next build?

Here's where we are. Junho delivered the test build this morning. Soyeon started error injection testing an hour ago, and early results are very encouraging: forty injections so far, zero system hangs. But the full validation plan we agreed on finishes on Monday, August 17th, with the camera regression suite.

So if we include it, 2.1.1 can't be finalized until the seventeenth at the earliest. If we don't include it, 2.1.1 is ready today.

My instinct is not to hold the storage fix hostage to the camera fix. Every day 2.1.1 is delayed, devices on 2.0.3 keep running with the old storage driver. To be fair, 2.0.3 doesn't have the panic, but it also doesn't have the dozens of other fixes that were in 2.1.0, including two security fixes from July.

On the other hand, the camera hang is the more visible issue for users right now. It's been reported by a reviewer, and the video of the frozen concert recording has a few hundred thousand views. If we ship a maintenance release that doesn't address it, some people will ask why.

That's a fair concern, but I'd rather answer 'it's coming in the next update' than rush a fix into a build that has to go to forty carriers. Is there any reason the camera fix couldn't go into the September build, which has a content cutoff on August 25th?

No, none. Validation finishes on the seventeenth, which leaves more than a week of margin before the September cutoff. And September is already planned as a security-focused build, so a small, well-tested driver fix fits there nicely.

Okay, I'm persuaded. As long as we commit publicly to a timeframe, users can live with a few more weeks. Most people don't record forty-minute 4K videos every day.

Then let's agree on that. The camera fix goes into the September build. In the meantime, could we publish a short tip suggesting users record long videos in 4K thirty frames per second until the update arrives?

That's a sensible workaround. I'll suggest it to the support team, along with the data showing 4K thirty is unaffected even after two hours.

Seojin, how's the Iberian reboot issue, and does any of it belong in 2.1.1?

Good news first. The configuration flag went out last Friday, after twelve hours on five percent of the population, and the reboot rate on that carrier dropped from 0.9 percent of devices per day to 0.04 percent, which is essentially back to baseline. As we expected, the modem subsystem restarts are still happening at about 0.8 percent per day, but users now see a few seconds without signal instead of a full reboot.

That's a big improvement. And the firmware fix?

Qorvix delivered the patched firmware yesterday, on schedule. We're testing it in our lab, and Pedro is doing drive tests in Portugal this week. The fully validated version comes on August 14th. It's not ready for 2.1.1, and I wouldn't want it there anyway, because a modem firmware change needs carrier approval, which would slow down the whole build.

Agreed. So the firmware also goes to September, with the fast-track approval Lucia negotiated?

Yes. Lucia confirmed the carrier will review it in five working days. And since the configuration flag already removed the reboots, there's no urgency that would justify a separate emergency release.

I'm glad to hear that. The September build is getting busy, though: security patches, the interim GPU fix, the camera fix, and now modem firmware. I'd like a single owner for the integration plan, so we don't discover conflicts at the last minute.

Fair point. I'll own the September integration plan and publish a content list by Friday, with a target merge date for each item.

Let's talk about the rollout for 2.1.1. Since 2.1.0 never went out, we're essentially doing the original rollout with a fixed build. I propose we submit to the carriers that require approval tomorrow, and start the open-market rollout on Tuesday, August 18th: one percent for forty-eight hours, then ten percent, then everyone by August 25th.

I'd like the halt criteria to include the storage-specific signal, not just the overall crash rate. If we see any storage panic signature in the one percent group, we stop, even if the overall rate looks fine.

I'd add the validity check warning as a metric too. If it fires much more often in the field than in the lab, it doesn't mean users are affected, but it would tell us the timing in the real world is different from what we tested, and I'd want to know that early.

Good, both of those go into the rollout plan. Hyunsoo, anything on power we should watch?

Just the standby drain metric from telemetry. If it goes up by more than half a percent overnight compared to 2.0.3 devices, I'd want to look before we expand. I don't expect it, but it's cheap to watch.

Before we wrap up, I'd like to spend five minutes on process, because the council will certainly ask how a panic like this reached a release candidate in the first place. What would have caught it earlier?

Honestly, the timeout change should have been tested together with the new storage firmware. They were merged a week apart, by different teams, and each was tested on its own. Nobody owned the combination. If we had a rule that any change to storage power management triggers a run of the storage stress profile with the latest firmware, we would have seen it two weeks before the release candidate.

From the lab side, we could make that automatic. We already tag changes by subsystem. If a change touches the storage driver, we can schedule a smaller, twelve-hour storage stress run on fifty devices overnight. It wouldn't catch everything, but at a five percent panic rate, it would almost certainly have caught this one.

I like that, and I'd extend the same idea to firmware updates from vendors. A new firmware binary is a change too, even if no line of our code moves. Right now, firmware updates go through a much lighter process than code changes, and that's a gap.

Agreed. I'll write up both proposals in the root cause summary, so the council sees concrete prevention steps, not just the fix.

Perfect. The council always responds better to 'here's what we changed' than to 'here's what went wrong'.

Let me summarize the recommendation I'll take to the council. Orion 2.1.1 is a go. It includes the storage fix and everything that was in 2.1.0. The camera hang fix and the modem firmware go into the September build. Carrier submissions tomorrow, open-market rollout starting August 18th with the halt criteria we discussed. Does anyone disagree?

No disagreement. SQE signs off on 2.1.1.

Thank you. Actions: Minji, send the formal sign-off and the test summary to the council by 9 a.m. tomorrow, and propose the 4K thirty tip to support. Tom, set up the rollout dashboard with the storage signature and the validity check metric. Jiwoo, write a short root cause summary for the release notes archive. Hyunsoo, add the standby drain comparison to the dashboard. Seojin, keep the firmware validation on track for September. And I'll prepare the September content list by Friday.

I'll have the summary ready by Thursday. I'd also like to present the case at the Friday tech share later this month, since it's a good example of how a small timing change can expose a latent bug.

Great idea, Jiwoo. Thank you, everyone. This is a much better place than where we were a week ago. Our next review is Tuesday, August 18th, at 2 p.m. Korea time, right after the one percent rollout starts, so we can look at the first day of data together.

Sounds good. I'll have the dashboard live before the rollout starts, and I'll share a link in the channel on Monday so everyone can check that the metrics make sense before real data starts flowing in.

And I'll make sure the stress lab keeps a small group running 2.1.1 through the rollout, so if anything strange shows up in the field, we can try to reproduce it right away.

Thanks, Grace. See you next Tuesday, and I'll bring the first results from the drive tests in Portugal.