Tech Share on Boot Time Optimization without Breaking Hidden Dependencies

Good afternoon, everyone, and welcome to the third Friday tech share. Attendance keeps growing, which I take as a good sign, or maybe people just like the snacks. Today's topic is boot time. Dongwook led the boot time work on Pulse Watch, and as most of you know, that work had an unexpected side effect two weeks before launch. He's agreed to share both the techniques and the lessons, which takes some courage.

Thanks, Daniel. I'd rather people learn from my mistake than repeat it. I'll spend about twenty minutes on how we measured and reduced boot time, then ten minutes on what went wrong, and then I'd love a discussion about how we approach this on Atlas.

Mina from Framework. I worked on the other half of the Pulse Watch boot, from the system server onward, so I'll probably jump in a lot.

Nari from SQE. I found the boot loop, so I'm here to make sure we talk about testing.

Seojin from Modem Software. The modem was the other half of that race condition, so I want to understand how boot ordering decisions are made.

Let's start with measurement, because you can't optimize what you can't see. The first mistake people make is measuring boot time as one number, from power button to home screen. That number is useful for marketing, but useless for engineering. We split boot into five phases: the bootloader, the kernel up to the first user space process, the init phase that starts native services, the framework phase from the system server to the launcher, and finally the time until the device is actually usable.

Can you explain the difference between the last two? I always assumed that when the home screen appears, the device is ready.

That's a common assumption, and it's exactly where some teams cheat. You can make the home screen appear earlier by delaying a lot of work until after it's drawn. But if the user taps an app and it takes five seconds to open because the system is still busy in the background, the device isn't really ready. We measure the point where the CPU load drops below a threshold and a test app opens in normal time.

For tools, we used a kernel boot trace with timestamps for every driver probe, plus a system trace that covers the whole boot. The key is to collect them on every nightly build, not just when someone is investigating. On Pulse Watch, we had a dashboard showing each phase over time, so if a phase got slower by half a second, we knew which day and which change caused it.

How noisy were the measurements? Boot time can vary quite a bit from run to run.

Very noisy at first, about plus or minus two seconds. We reduced it by always booting ten times on three devices and taking the median, with the devices at the same temperature and battery level. After that, the variation was under three hundred milliseconds, which is good enough to see real regressions.

Now the techniques. In the kernel phase, the biggest win came from asynchronous driver probing. By default, many drivers probe one after another, and if one driver waits two hundred milliseconds for its hardware to respond, everything behind it waits too. We marked about forty drivers as safe to probe in parallel, and the kernel phase went from three point one seconds to one point eight.

How did you decide which drivers were safe to probe in parallel? That seems like exactly the kind of decision that could break hidden dependencies.

You're absolutely right, and I'll come back to that. For the kernel, we were fairly careful: a driver was only marked asynchronous if all its dependencies were declared explicitly in the device tree, so the kernel itself could enforce the ordering. That part actually went fine. The problem came later, in user space.

In the init phase, we did three things. We moved loading of non-essential kernel modules, like the ones for sensors that aren't needed until the user starts a workout, to after boot complete. We started the services that don't depend on each other in parallel instead of in a fixed sequence. And we removed six services that were left over from the previous watch and did nothing on the new hardware. Together, that saved about two seconds.

Six services doing nothing. That sounds a lot like the forty-nine kernel patches Sora found last week. We carry a lot of history without noticing.

How did you confirm those six services really did nothing? Removing a service seems even riskier than removing a wait.

Good question, and to be fair, we were more careful there. For each service, we checked which hardware it talked to and which other processes connected to it during a full day of use. If the hardware didn't exist on the watch and nothing connected to it, we removed it. We also kept the code for one release, disabled rather than deleted, so we could bring it back quickly if needed.

On the framework side, the largest saving came from reducing what the system server does before the launcher appears. We profiled every service's startup and found that about a third of the time was spent in services that the user can't interact with until much later, like backup, sync, and some health monitoring. We changed them to start after boot complete, with a short delay so they don't compete with the user's first actions.

Doesn't delaying services risk the problem you mentioned earlier, where the home screen looks ready but the device is still busy?

It does, and that's why we measured the usable point, not just the home screen. We tuned the delays so that background services start in small groups, and we checked that the test app still opened in normal time throughout. It's a balance: if everything starts at once, the first app launch is slow; if everything waits too long, sync and notifications feel delayed.

We also reduced the number of classes preloaded by the app process template. Preloading makes app launches faster later, but it costs boot time up front. On a watch with few apps, preloading thousands of classes wasn't worth it. On a phone, the trade-off is different, so please don't copy that change blindly to Atlas.

Now for the part I promised. The final change that got us below twenty seconds was starting the system server six seconds earlier by removing an explicit wait in init. That wait had been there for years, and the comment above it just said 'wait for modem'. Nobody remembered why. Our measurements showed nothing broke when we removed it, and the system server came up much faster.

And the reason it existed was the eSIM restore path, which only runs after a factory reset with an existing profile. So it was invisible in normal boot testing.

Exactly. The wait was an implicit dependency, an ordering guarantee that someone added years ago instead of fixing the provisioning service. When we removed it, we broke a contract that wasn't written down anywhere except in a three-word comment. Our boot tests didn't catch it because they always booted clean devices.

That's the testing lesson for me. Boot testing needs to cover the unusual boots, not just the normal one: first boot after a factory reset, boot after an OTA, boot with a full storage, boot with a SIM or eSIM profile, and boot after a crash. Those are the paths where old, forgotten waits usually matter.

I'd add a code review rule: if you remove a wait, a sleep, or an ordering constraint, you have to explain why it existed, not just show that nothing obviously broke. If you can't explain it, find out, or ask the original author before removing it.

I want to raise a slightly provocative question. How much does boot time actually matter? Most users reboot their phones once a week or less. We spent weeks of engineering time, and took real risk, to save six seconds that a user experiences maybe fifty times a year.

Fair point, but I'd argue boot time matters more than its frequency suggests. Boot is what happens after every update, and a slow boot after an update makes users feel the update made their phone worse. It's also on the critical path for recovering from crashes. If the device reboots unexpectedly, every second of boot is a second the user is staring at a logo, already annoyed.

I'd also mention emergency calls. If a phone reboots and the user needs to make an emergency call, the time until the modem is registered on the network really matters. That's a case where boot time is a safety topic, not just a convenience.

That's a strong argument. So maybe the right target isn't the total boot time, but specific milestones: time to emergency call capability, time to first unlock, and time until the device is usable. Those reflect what users actually need.

I like that framing a lot. It also changes how we optimize. If emergency calls are a milestone, then modem initialization is on the critical path and should start as early as possible, which is the opposite of what usually happens.

Let's bring this to Atlas. Marketing hasn't set a boot time claim yet, which is a rare opportunity. I'd propose we define milestone targets ourselves before anyone else does: emergency call capability within twelve seconds, first unlock within eighteen, and usable within twenty-two. Are those realistic?

For a new chipset on a new kernel, they're ambitious but not crazy. I'd suggest we treat them as budgets per phase rather than one target. If the bootloader gets two seconds, the kernel three, init four, and so on, each team knows its share and can't spend another team's time.

And SQE will add the unusual boot scenarios to the nightly suite from the first Atlas boot, not two weeks before launch. Factory reset with an eSIM, boot after an OTA, boot with full storage, and boot after a forced crash.

From the modem side, I'll document every ordering dependency between the modem and the framework explicitly, so nobody has to guess from a three-word comment again.

A practical question before we wrap up. When a boot regression appears on the dashboard, who owns it? On Pulse Watch, I sometimes saw a phase get slower and nobody picked it up for days, because it wasn't clear whether it was a kernel issue or a framework issue.

That's exactly why per-phase budgets help. If the kernel phase goes over its budget, kernel owns it, full stop. Even if the root cause turns out to be somewhere else, the owner of the phase is responsible for finding the right team. It removes the debate about who should look first.

I'd add one rule: a regression of more than three hundred milliseconds in any phase should automatically block the change that caused it from being promoted to the release branch. Otherwise, small regressions pile up, and by the time someone notices, there are twenty changes to untangle.

That's strict, but I like it. Our measurement noise is under three hundred milliseconds now, so the threshold wouldn't fire randomly. Let's try it on Atlas and see whether it causes more pain than it saves.

One more question from me. Dongwook, if you could go back, would you still have removed the wait for the modem?

Honestly, yes, but in a different order. I'd first find out why it existed, fix the provisioning service so it handles a slow modem properly, and only then remove the wait. The end result would have been the same boot time, without the boot loop. The mistake wasn't the optimization; it was skipping the investigation.

That's probably the most useful sentence from today. I'm going to put it on the wiki page, right at the top, so it's the first thing people read before they start optimizing anything.

Let me capture the action items. Dongwook: write up the boot measurement method and the phase dashboard for the wiki, and draft the per-phase budgets for Atlas. Mina: propose the milestone definitions for first unlock and usable. Seojin: document the modem ordering dependencies. Nari: add the unusual boot scenarios to the Atlas nightly suite. And I'll propose the review rule about removing waits and ordering constraints to all kernel and framework reviewers.

Thank you all for turning my worst week into something useful. I'll also share the raw boot traces from Pulse Watch on the wiki, before and after each optimization, so anyone working on Atlas can see exactly what changed in each phase and how we measured it.

That would be great for SQE too. Having a known good trace to compare against makes it much easier to explain a regression when we file it.

And I'll bring the modem dependency document to next week's session, in case anyone wants to review it before it's final.

That's what these sessions are for. Next Friday, August 28th, Jiwoo will present a blameless postmortem of the Orion storage panic. Same time, 4 p.m. Korea time. See you all then.