Debating Vendor Modules and Upstream-First under the Generic Kernel Image

Welcome back to the Friday tech share. Last week we talked about lockups, and judging by the wiki page views, people found it useful. Today's topic is more of a debate than a lecture. Sora will present an inventory of our out-of-tree kernel patches, and then we'll argue about what to do with them under the Generic Kernel Image. I've invited people from outside the kernel team on purpose, because this decision affects everyone.

Thanks, Daniel. For anyone who's new to the topic, I'll start with a two-minute explanation of what the Generic Kernel Image is and why it matters, and then show the numbers.

I'm definitely one of those people. From the framework side, the kernel is mostly a black box I complain about.

Same here. I'm mainly interested in what this means for testing and for the Atlas schedule.

And I'm here because GKI has big implications for how quickly we can ship kernel security fixes, which is my favorite topic.

Here's the short version. In the old model, every device maker took the kernel from the chipset vendor, added hundreds of their own changes, and built a kernel that was unique to each device. That gave us total freedom, but it meant that every security fix had to be merged and tested separately for every device. With GKI, there's one kernel binary, built by Google from a common source tree, and it's the same for every device on that kernel version.

Everything that's specific to our hardware or our product has to live outside that core binary, in loadable vendor modules. And modules can only use a defined set of kernel functions and data structures, called the symbol list. If a module needs something that isn't on the list, we have to ask Google to add it, and they'll only agree if there's a good reason.

So in practice, you lose the ability to change the core kernel directly. Any change you used to make inside the kernel now has to either become a module, go into the common tree, or disappear.

Exactly. There's one more option, which is where most of the debate comes from: vendor hooks. Google has added hundreds of small hook points in the core kernel where a vendor module can attach its own code, for example to adjust a scheduling decision. They're a pragmatic escape hatch, but they're also controversial, because they let vendors keep doing things that arguably should be solved properly upstream.

And from a security perspective, the big win is that Google can fix a vulnerability in the core kernel once, and every GKI device can pick it up quickly. In the old model, our median time to ship a kernel security fix on Orion was about forty days. Partners who are fully on GKI are doing it in under two weeks.

Sora, show them the numbers. That's where it gets real.

We're carrying three hundred and six out-of-tree patches from Orion. I went through every one of them over the last two weeks and sorted them into four buckets. First bucket: sixty-two patches are already covered by the 6.12 common tree, either because we upstreamed them or because upstream solved the same problem differently. Those we simply drop. That's the easy part.

Second bucket: one hundred and forty-one patches are driver code that can move into vendor modules with modest effort. These are things like our display panel drivers, sensor drivers, and charging logic. Third bucket: fifty-four patches change core kernel behavior, mostly in the scheduler, memory management, and power, and the only way to keep them is through vendor hooks, or by getting the change accepted upstream. Fourth bucket: forty-nine patches that, honestly, I can't justify keeping. They're workarounds for old bugs, debugging aids, or features nobody uses anymore.

Forty-nine patches nobody needs? How did we end up carrying those for years?

Because removing a patch feels risky, and keeping it feels free. Each time we moved to a new kernel, it was easier to rebase everything than to ask whether each patch still mattered. GKI is forcing us to ask the question we should have been asking all along.

Before you drop them, though, I'd want to know if any of the framework relies on their behavior. We've been bitten before by a kernel 'cleanup' that removed a sysfs file one of our services was reading.

That's a very fair point, and it's exactly why I want a review from each team before we drop anything. I'll publish the list of forty-nine with a short description of each, and give everyone two weeks to object.

Now the interesting part: the fifty-four core patches. My position is pragmatic. For Atlas, we should move as many of them as possible onto vendor hooks. We have a launch date in March, and upstreaming a scheduler change can easily take six months or more, with no guarantee of success. Vendor hooks let us keep our tuning and ship on time.

I'd push back on that. Every vendor hook is a place where our module code runs inside the most sensitive parts of the kernel. From a security standpoint, that's attack surface we own forever. And if Google changes or removes a hook in a future kernel, we're back to square one. Upstream-first may be slower, but it's the only approach that actually reduces our long-term burden.

In principle, I agree with you. In practice, 'upstream-first' sometimes means 'ship late'. I've seen teams submit a patch upstream, get into a six-month discussion about the right design, and end up with something that doesn't solve their original problem. I can't bet the Atlas launch on that.

Can I offer a framework perspective? We had the same debate about platform changes years ago. What worked for us was a two-track approach: use the escape hatch to ship, but open the upstream discussion at the same time, with a deadline to retire the local change. If we never set a deadline, the temporary solution becomes permanent.

I like that. And honestly, not all fifty-four patches are equal. About twenty of them are small power or scheduling tweaks that upstream would probably accept quickly, maybe within a release or two. Those are worth upstreaming now. The other thirty-four are larger, more product-specific changes where vendor hooks are the realistic option.

Then I'd propose a rule for the hooks: every vendor hook we use needs a named owner, a security review of the code that runs inside it, and an expiry date when we revisit whether it can go upstream. If nobody's willing to own it, it shouldn't exist.

I can live with that, as long as the security review doesn't become a bottleneck. If every hook waits three weeks for a review, we're back to shipping late.

Fair. Most hooks are small. If your team sends them in batches with a short design note, I can commit to a one-week turnaround.

Let me bring this back to testing. If the core kernel comes from Google and gets updated more frequently, does SQE need to retest everything every time the core kernel changes? Because that would be a lot of testing.

That's one of the trade-offs. The core kernel changes are tested heavily by Google and by other partners using the same binary, so they're less risky than our own changes used to be. But our modules interact with the core kernel, so we still need a solid regression suite. My suggestion is a smaller kernel-focused suite that runs on every core kernel update, and the full suite only for our own module changes and monthly releases.

That's reasonable, but I'd want the kernel-focused suite to include our stress profiles, not just functional tests. Most of the kernel bugs we find, like the storage panic this month, only show up under sustained load.

Agreed. A twelve-hour stress run on fifty devices would be a good minimum. Tom already proposed something similar for storage changes after the Orion incident, so we can reuse that setup.

One question from me: will framework engineers still be able to debug kernel issues the same way? If the core kernel is a Google binary, do we still get symbols and source for crash analysis?

Yes. Google publishes the source and the debug symbols for every GKI build, so ramdump analysis works exactly the same way. The only difference is that if the bug is in the core kernel, the fix goes to Google rather than into our tree.

Sora, you mentioned that modules can only use functions on the symbol list. What happens when one of our modules needs something that isn't on the list, and we're two weeks from a milestone?

That's the scenario that worries me most. Google updates the symbol list on a regular schedule, and requests are reviewed in batches. If we miss a window, we may have to wait several weeks. The way to avoid that is to identify every symbol our modules need early, ideally right after we convert the driver patches, and submit them all at once.

Kevin from Google also warned us about this on the Arvenna call. Partners who accumulate private symbol exports during bring-up end up spending weeks cleaning them up before certification. So we should build a check into our CI that fails any module using a symbol outside the official list, from day one.

From a security angle, I'd add one more thing. Some symbols give modules very powerful access, for example to modify page tables or credentials. When our modules request those, I'd like to know, because a vulnerability in a module with that kind of access is much more dangerous than one in a simple sensor driver.

That's easy to add. I can flag sensitive symbols in the CI report, and any new request for one of them would need Jiho's approval before we submit it to Google.

This sounds a lot like how we manage hidden platform APIs on the framework side. We keep an allow list, every exception needs a justification, and the list gets reviewed every release. It's tedious, but it's saved us from some very painful upgrades.

That's a good model to borrow. Let's treat the symbol list the same way, with a review at every kernel update.

If we're serious about upstreaming twenty patches, someone needs time to do it. Upstream work is not something engineers can do in the gaps between bring-up crises.

You're right, and that's a resource question I'll have to take to management. My proposal would be one engineer at fifty percent, dedicated to upstream work, starting in September. It's not much, but it's more than we've ever had.

I'd volunteer for that, actually. After two weeks with this inventory, I know these patches better than anyone, and I'd rather fix the root problem than rebase the same patches again in two years.

Just be careful not to take Sora away from Atlas integration. She's also the person who makes sure the daily build boots.

Noted. Let's start after first boot on Atlas, so the integration work is stable before Sora splits her time.

Let me try to capture what we agreed. One: the sixty-two patches already covered upstream are dropped. Two: the hundred and forty-one driver patches become vendor modules. Three: the forty-nine unjustified patches are published for a two-week objection period, then dropped. Four: of the fifty-four core patches, about twenty go upstream, and the rest use vendor hooks, each with an owner, a security review, and an expiry date. Five: a kernel-focused regression suite with stress runs for every core kernel update.

That works for me. It's a much better outcome than I expected when I walked in.

Same. And I'll have my team check the forty-nine patches for hidden dependencies before the objection period ends.

Action items. Sora: publish the full inventory with the four buckets by Tuesday, including the list of forty-nine for objections. Rachel: framework dependency check on the forty-nine by August 28th. Jiho: draft the vendor hook review checklist, so reviews are consistent. Ethan: define the kernel-focused regression suite with Tom. And I'll write the proposal for the upstream engineer and take it to management next week.

I'll also add a column to the inventory showing which patches touch interfaces that user space reads, like sysfs or procfs files, so Rachel's team can focus on those first.

That will save us a lot of time. Thanks, Sora. I'll also ask Tom to reserve the stress lab for the first kernel-focused run, so we have a baseline before the module conversion starts.

Thank you all. I thought this would be a lot more heated, but I think we found a sensible middle path. Next Friday, August 21st, at 4 p.m. Korea time, the topic is boot time optimization. Given what happened with the Pulse Watch boot loop, it should be a lively discussion.

I'll be there. This was the first kernel session where I understood everything, so thank you, Sora.

Thank you. I'll take that as a compliment. And if anyone wants to go through their team's patches in more detail before the objection period ends, just book thirty minutes with me next week. I'd much rather answer questions now than discover a hidden dependency after we've dropped something.