Planning the September Security Patch and Quarterly Platform Release Adoption

Hello, everyone. Thanks for joining. Before we start, a reminder that today's discussion includes vulnerability details that are under embargo until the September bulletin is published, so please don't forward the slides or the notes outside this group. I'm Grace from Release Management. I own the monthly maintenance release train for Orion and Orion Lite.

Jiho Nam, Product Security. I lead our patch response, which means I track every monthly bulletin and make sure each vulnerability is fixed in our builds on time.

Sungmin Yoo, also Product Security. I focus on vendor components, so anything in the chipset drivers or firmware comes through me.

Kevin Albright, partner engineering at Google. Some of you met me last week on the Atlas call. Today, I'm mainly here for the quarterly platform release question.

Olivia Grant, also Google, on the security partnerships side. I manage the advance notification program, so I'm the one who sends you the bulletin preview each month.

Raj Mehta, product security at Qorvix, joining from San Diego. I coordinate the fixes for vulnerabilities in our components that appear in Google's bulletins.

Thank you. We have two topics today. First, the September security bulletin, and specifically one critical vendor issue that could put our patch date at risk. Second, whether and when we adopt Google's quarterly platform release on Orion. Jiho, could you give us the overview of the September bulletin?

Sure. The preview we received from Olivia's team lists thirty-eight vulnerabilities for September. Thirty-one are in the Android platform itself, and Google has already provided patches for all of them. They're straightforward to apply, and my team has already merged twenty-six into our maintenance branch. The remaining seven are vendor issues: two in kernel components and five in Qorvix components.

And which one is the problem?

One of the Qorvix issues is rated critical. It's in the GPU driver, and it allows a malicious app to read and write kernel memory. In other words, it's a privilege escalation from a normal app to full control of the device. It's the most serious item in the whole bulletin, and right now, we don't have a fix for it.

Raj, what's the status on your side?

Let me be upfront about the timeline. We confirmed the vulnerability three weeks ago. The obvious fix, a bounds check in the memory mapping path, turned out to break a performance optimization that several games rely on. So we had to redesign the fix. The new version is in internal testing now, and our current estimate for delivery to customers is September 3rd.

That's a problem for us. Our September maintenance build has a content cutoff of August 25th, so that SQE can complete validation and the carriers can approve it in time for the first week of September. A September 3rd delivery misses that by more than a week.

I should add the policy context here. The bulletin goes public on September 7th. After that, the vulnerability details are known, and the risk of exploitation goes up quickly, especially for a GPU issue like this one. We expect partners to ship critical fixes as soon as possible after publication. For devices in our security update program, the expectation is that the patch level reported by the device is accurate, meaning we can't claim the September level without including this fix.

Right, and that's an important point. If we ship the September build without the GPU fix, we'd have to declare the August patch level, even though we include thirty-seven of the thirty-eight September fixes. That looks bad in every security report, and some enterprise customers have contractual requirements on the patch level.

I understand the pressure. One option I can offer is an interim fix. We can deliver the original simple bounds check by August 20th. It closes the vulnerability completely, but it costs about eight percent in frame rate for the affected games. The full fix without the performance penalty would follow on September 3rd.

Eight percent is noticeable. Which games are we talking about?

It mainly affects games that use a specific texture streaming technique. On our measurements, three of the top twenty games on your store are affected. Most users wouldn't notice in normal use, but enthusiasts who measure frame rates probably would.

From a security standpoint, I'd take the interim fix without hesitation. A temporary frame rate drop in three games is a much smaller problem than a known, public privilege escalation on millions of devices. We can replace it with the full fix in the October build.

I agree with the direction, but I want to make sure we're not trading one problem for another. If gamers notice a performance drop right after a security update, the story online becomes 'security update slows down games', and that can discourage people from updating at all. That's also a security risk, in a way.

That's a fair concern. What we've seen work for other partners is being transparent in the release notes without disclosing the vulnerability details: something like, this update includes security improvements to the graphics driver, and some games may see a minor performance change, which will be addressed in the next update. Users tend to accept that when it's stated up front.

Since we're talking about release notes, let me remind everyone of the embargo rules, because we had an incident with another partner last quarter. Until September 7th, you can ship the fix, but your release notes and any public communication must not describe the vulnerability. Shipping the patch early is fine. Describing it early is not.

Understood. Our standard practice is that maintenance release notes before the bulletin date only mention the patch level and generic wording like 'security improvements'. After the bulletin is public, we publish our own security page with the full list of fixes.

Does that also apply to the commit messages in our open-source kernel releases? We publish kernel source for every build.

Yes, and that's exactly where the incident happened last quarter. A partner published kernel source with a commit message that described the bug in detail, two weeks before the bulletin. Please make sure security fixes in your public source use neutral commit messages until the embargo lifts.

Good reminder. I'll check that our source release script strips or rewrites the messages for any commit tagged as security. If it doesn't, I'll get it fixed before the September build.

Let's move to the second topic. Kevin, Google's quarterly platform release came out in June, and we haven't adopted it on Orion yet. What's your view?

Our recommendation is to adopt it. Besides new features, it includes a large number of stability fixes and some security hardening that doesn't appear in the monthly bulletins, because it's not tied to a specific vulnerability. Partners who skip quarterly releases tend to accumulate a gap that makes the next major upgrade harder.

I don't doubt the value. My concern is risk. A quarterly release is roughly a thousand changed files in the framework. Putting that into a maintenance release that also carries a critical security fix and an interim GPU workaround is a lot of change at once. If something breaks, it'll be hard to tell what caused it.

I agree with Grace on not combining them in September. But from the security side, I don't want to skip the quarterly release entirely. Some of the hardening in it, especially around app permission checks, would close attack paths that we currently mitigate only partially.

Then what about adopting it in October? That would separate it from the September security build, and it would still put you on the latest quarterly base well before the next major release in December.

October works better for me, but there's a staffing issue. The framework team is already split between Atlas and the OS upgrade program, which kicked off yesterday. Merging a quarterly release in September means taking people away from one of those.

Couldn't the OS upgrade program reuse that work? If the upgrade merge starts from a base that already includes the quarterly release, there would be fewer conflicts later. It's not entirely extra work.

That's generally true. Much of the quarterly release is already part of the next major version, so partners who take it first usually find the major merge somewhat easier.

Does the quarterly release require any vendor component updates? If Qorvix has to deliver new drivers for it, that's another dependency on a team that's already busy with the GPU fix.

It doesn't. Quarterly releases keep the vendor interface frozen, by design. Your existing vendor components work unchanged, which is one of the reasons they're relatively safe to adopt.

I'm glad to hear that, honestly. My team has enough on its plate this month.

That's a good argument. I'll check with Brian and Elena whether the upgrade team can own the quarterly merge as their first step. If they agree, October is the plan. If not, I'll come back with an alternative rather than silently skipping it.

Before we finalize, there's one more stakeholder group we haven't mentioned: our enterprise customers. Several large companies that deploy Orion to their employees have security policies that block devices more than one patch level behind. If our September build reaches some carriers late, their devices could start getting blocked from corporate email in mid-October.

That's a good point. Which carriers are usually the slowest to approve maintenance releases?

Historically, two carriers in North America and one in Japan take three to four weeks. Everyone else is under two weeks. If we can get the September build to those three carriers by August 28th, we should be fine. If we submit in mid-September, their customers could fall behind.

For what it's worth, carriers are generally willing to prioritize a security-only maintenance release. If you tell them the build contains a critical fix and nothing else significant, many of them have a shorter review path. That's another reason not to bundle the quarterly release into September.

That's very useful to know. I'll ask our carrier team to request the expedited path for the September build with those three carriers, and to send them the submission on August 26th, the day after cutoff.

One more thing for the enterprise side. Some of those customers ask us directly, under a confidentiality agreement, whether the critical vulnerabilities affect our devices. Can we tell them before the bulletin is public, or does the embargo apply to them as well?

The embargo applies to them too, unless they're members of the advance notification program themselves. What you can say is that a security update is coming and that you recommend installing it as soon as it's available. That's usually enough for their planning.

Back to September for a moment. If the interim GPU fix arrives on August 20th, how much time does SQE need to validate it? GPU changes can have side effects outside of games, for example in camera previews or video playback.

I checked with Minji before this call. With the fix in hand on the 20th, SQE can run the graphics regression suite and the top hundred apps by the 24th, which still makes the cutoff on the 25th. It's tight, but there's no slack to give away.

I'll make sure the interim fix comes with our own test report, including the graphics conformance results, so SQE doesn't have to repeat tests we've already run. And if anything slips on our side, you'll hear from me by the 18th, not on the 20th.

That's very helpful, thank you. Early warning is all I ask.

Let me summarize. For September: we take Qorvix's interim GPU fix on August 20th, SQE validates it by the 24th, and the September build declares the September patch level. Release notes mention a minor graphics performance change without describing the vulnerability. The full GPU fix goes into the October build. For the quarterly release: target October, pending agreement from the upgrade team. Any objections?

None from Security. Action items on my side: I'll finish merging the remaining platform fixes by August 18th, and I'll draft the September security page so it's ready to publish on the seventh.

I'll verify the source release script for neutral commit messages by Friday, and I'll track the other four Qorvix fixes, which should be on time.

And I'll send the final version of the bulletin to your team on September 1st, as usual, so you can cross-check the list against your build.

Thank you, everyone. Let's meet again on Wednesday, August 19th, at 9 a.m. Korea time, which is 5 p.m. on the eighteenth in California, to confirm the interim fix is ready and review the quarterly release decision. I'll send the invitation with the embargo notice attached.

Works for me. Thanks, Grace. And if the upgrade team has questions about the quarterly merge, feel free to loop me in before the nineteenth. I can connect them with the engineers on our side who handled the most common conflicts for other partners.

Thank you. I'll see you all on the nineteenth, hopefully with good news. In the meantime, I'll send Sungmin a short weekly status on all five of our September fixes, so nobody has to chase us for updates.