Agreeing on a Monthly Kernel Update Cadence with Google and Qorvix

Good morning, everyone, and good evening to Nathan in Seattle. Thank you for joining on a Sunday. Last week, we found a USB crash on Orion that had already been fixed upstream three weeks earlier. We simply hadn't picked it up, because we only merge kernel updates once a quarter. Today, I'd like us to agree on a faster cadence.

Chris, Kernel and BSP. I cover Orion Lite and Atlas Lite, so whatever we decide here affects my products too.

Tom, SQE. I'm here to make sure that 'faster' doesn't mean 'less tested'.

Nathan Brooks, Android kernel team at Google. I'm happy to see this conversation happening. A monthly cadence is what we recommend to all partners.

Laura Chen, Qorvix customer engineering. Most of my time is on storage and display these days, but I'm covering the kernel relationship for Orion this month.

Let me describe our current process, so we all start from the same picture. Orion and Orion Lite run the generic kernel image for the 6.1 branch. Google publishes a new certified build of that image every month, containing all the upstream stable fixes and security patches. Today, we take one of those builds once a quarter, rebuild our vendor modules against it, and run our full kernel regression, which takes about three weeks.

One small correction, if I may. You shouldn't need to rebuild your vendor modules for each monthly image. The kernel module interface is frozen for the life of the branch. Modules built against the first 6.1 image should load and work on every later 6.1 image without being rebuilt. That's the whole point of the stable interface.

In theory, yes. In practice, we rebuild because two of our modules still use symbols that were added to the list after launch. They load fine, but our build system treats each image update as a full rebuild. That's a habit from the pre-GKI days, more than a real requirement.

And the three-week regression is the real bottleneck. It's the same suite we run for a full maintenance release: functional tests, performance, battery, and a long stress campaign. Running that every month for every model would take most of the kernel test capacity we have.

Let me explain what happens on our side, because it may change how much you need to repeat. Each monthly image goes through our own regression on a large device fleet, including reference hardware from several chipset vendors. It also runs on early partner devices for two weeks before it's published. So the core kernel changes are already well tested by the time you receive them.

I believe that, but your fleet doesn't include our devices, our modules, or our workloads. The storage panic this month only happened with our specific firmware and our specific timeout. No amount of testing on reference hardware would have caught it.

That's a fair point, and I agree. The question isn't whether you test, but what you test. Our testing covers the core kernel well. Your testing should focus on the interaction between the core kernel and your modules, under your workloads. That's a much smaller and more targeted suite than a full release regression.

That's actually close to what we proposed at the GKI tech share two weeks ago: a kernel-focused suite that runs in about twelve hours, with stress profiles for storage, USB, display, and power, on fifty devices per model. We've built a first version of it.

Has it run yet? I'd like to know whether it would have caught the two kernel issues we had this month.

We ran it retroactively last week, on the builds from before each fix. It caught the USB crash in nine hours, through the plug and unplug rig. It caught the storage panic in eleven hours. So, yes, both would have been caught before release.

That's very convincing. Twelve hours a month per model is something we can actually afford.

From Qorvix's side, I want to raise one concern. Our modules are tested against specific kernel images. If you update the core kernel every month, and something breaks in one of our drivers, we need to know which image introduced the problem. Otherwise, we'll spend days arguing about whether it's our bug or a kernel change.

That's exactly why monthly is better than quarterly for that problem, not worse. With a monthly update, if something breaks, the suspect is one month of changes, maybe a few hundred commits. With a quarterly update, it's three months, over a thousand commits, and bisecting becomes much harder.

That's a good argument. In that case, I'd ask for one thing: send us the kernel-focused suite results for each monthly update, even when everything passes. If our test team can see the trend, we can spot problems in our drivers before they become failures.

That's easy. We generate a report anyway. I'll add Qorvix to the distribution list.

One more vendor question. Qorvix also delivers its own driver fixes, separate from the core kernel. Today, those come whenever they're ready. Could we align them to the same monthly rhythm, so we test everything together once a month instead of in random batches?

For routine fixes, yes. We can deliver a monthly driver bundle about a week before your kernel update window. Critical security fixes would still come immediately, of course, as they do now.

What happens if a monthly image causes a problem that our suite doesn't catch, and it reaches users? With quarterly updates, at least we'd been running the image for weeks before shipping it.

Two safety nets. First, the monthly image doesn't ship on its own; it rides along with our normal monthly maintenance release, which still goes through the full staged rollout and halt criteria. Second, because the core kernel is a separate image, we can roll back just the kernel to the previous month's version in the next update, without touching anything else.

And from our side, if a regression is found in a monthly image, we publish a fixed image quickly, usually within a week. Partners who report problems early get them fixed for everyone. Your field data would actually help us improve the image for all partners.

That's a nice side benefit. Last week, Nathan's crash signature list saved us a week. If we're on recent images, the reverse can happen too: our reports can save other partners time.

I'd like one more safety net on our side. For the first three monthly updates, let's start the staged rollout at half a percent and hold it for seventy-two hours, just like we agreed for security-related firmware. Once we've seen a few clean months, we can go back to the normal rollout speed.

Agreed. It costs us a few days per release, and it gives everyone more confidence while the process is new. I'll add it to the pilot plan.

From Qorvix's side, if a rollout is paused because of something in one of our drivers, please call me directly. We'd rather start debugging on the same day than wait for a ticket to be triaged.

Let me propose a concrete plan. In September, we run a dry run: we take the latest monthly image, run the kernel-focused suite on Orion and Orion Lite, and see how long it really takes end to end, without shipping it. In October, the first real monthly image goes into the Orion October maintenance release. If October goes well, Orion Lite follows in November. After three months, we review the results.

I support that. The dry run will tell us whether twelve hours is realistic, including flashing, setup, and analyzing the results. My guess is that the first one will take closer to two days, and we'll get faster with practice.

What would make us stop the pilot? I'd like us to define that up front, rather than arguing about it later.

Good point. I'd propose: if a monthly image causes a field regression that our suite should have caught, we pause and improve the suite before continuing. If two monthly images in a row cause field regressions, we return to quarterly updates and rethink the approach.

That sounds very reasonable. I'd be surprised if you hit either condition, but it's good to have them written down.

Let's talk about timing within the month. Our maintenance releases have a content cutoff around the twenty-fifth. When does Google publish each monthly image, and does that leave enough time for our suite?

Monthly images are usually published in the second week of the month. So if you pick up the image around the tenth, you have about two weeks before your cutoff. That should be plenty for a twelve-hour suite, even with a rerun or two.

Two weeks works for us. My only concern is collisions with launches. In September, the stress lab is mostly booked for Orion Fold. If the dry run and a launch need the lab in the same week, the launch will win, and the kernel suite will slip.

Then let's reserve a permanent slot: fifty devices per model, every month, from the tenth to the fifteenth, protected from launch requests. If kernel updates are going to be routine, the lab capacity has to be routine too.

I can do that, but I'll need Grace's support at the release council, because launch teams will push back. I'll present it together with the retroactive results, since they show clearly what the suite would have prevented this month.

If the slot is fixed, it also helps us. We can plan our monthly driver bundle around it and know exactly when our code will be tested. Predictability is worth a lot to a vendor with many customers.

For Atlas and Atlas Lite, which launch on 6.12, should we follow the same monthly model from launch, or wait until they're stable?

I'd strongly recommend starting from launch. The first months after launch are when the most fixes land upstream for a new branch, because that's when the whole ecosystem is finding bugs at the same time. Partners who wait often end up six months behind on exactly the fixes that matter most.

That makes sense. And because Atlas will be built with the module build changes from day one, the monthly update should be cheaper there than on Orion.

I'll make sure the kernel-focused suite is part of the Atlas test plan from the first proto build, so by launch, we'll have six months of baseline data to compare each update against.

Qorvix will do the same for Atlas Lite. I'll make sure our Q7 team plans their monthly bundle from the first drop, not from launch.

How will we show management that this was worth the effort? Monthly updates mean more work for kernel and SQE every month.

I'd track two numbers. First, the age of the kernel fixes we ship: today, the average fix reaches our users about four months after it's published upstream. With monthly updates, it should be under six weeks. Second, the number of field kernel crashes that match a signature already fixed upstream. Last quarter, that was three of our top ten kernel crashes.

Three of the top ten crashes being already fixed upstream is a strong argument by itself. Those are crashes our users suffered for no good reason.

If it helps, I can share anonymized data on how field crash rates changed for partners who moved from quarterly to monthly. It's been a clear improvement for most of them.

Let me summarize. We'll move Orion and Orion Lite to monthly kernel image updates. September is a dry run. October is the first real update for Orion, and November for Orion Lite. Stop conditions as agreed. Actions: Tom finalizes the kernel-focused suite and runs the dry run. I'll clean up the two modules that use late symbols and change the build so we stop rebuilding modules for every image. Laura arranges the monthly Qorvix driver bundle. Nathan shares the partner data. Chris prepares Orion Lite for November.

I'll confirm the monthly bundle schedule with our release team this week, and send it to Jiwoo and Tom in writing.

And I'll make sure Atlas Lite starts on this model from day one, so we never build the quarterly habit there.

Thank you, everyone. I think this is one of the most valuable process changes we've made this year. We'll review the dry run results at the next vendor sync, on Sunday, September 13th, at 10 a.m. Korea time.

Thanks for having me. I'm looking forward to seeing the dry run numbers. If anything in the dry run looks strange, send it to me before the thirteenth, and I'll check whether other partners have seen the same thing.

I'll book the stress lab for the dry run before anyone else claims it.

And I'll write a one-page summary of today's decisions for the release council, including the stop conditions and the protected lab slot. If the council understands the reasoning before Tom asks for the slot, the conversation will be much easier.

Thank you all. This is the kind of process change that makes a vendor's life easier, not harder, which I don't get to say very often.