Diagnosing System Server Freezes Caused by Binder Thread Exhaustion

Good morning, everyone, and thanks for joining on a Saturday. I'm Soyeon from SQE. We have a system freeze on Orion Lite that's been climbing in the field for two weeks, and we finally have good dumps. I've invited Developer Relations as well, because a third-party app seems to be involved.

Elena, Platform Framework. System server hangs are my daily bread, so I've already looked at the dumps.

Brian, Framework lead. I'm here because whatever we decide will probably touch the package manager, which is my team's code.

Yuna, Kernel and BSP. Soyeon mentioned slow storage, so I'm here for the file system side.

Lisa Han, Developer Relations. I manage our relationships with the major app developers, so if we need one of them to change something, I'm the one who makes the call.

Here's what users report. The phone freezes completely for about a minute, usually in the early morning, and then it shows the boot animation and restarts. Users who are awake often notice it while their alarm is about to ring, which is how it became visible on social media. A few people said their alarm didn't go off.

A missed alarm is a serious problem for users, even if the freeze is short. What does the field data show about timing?

About seventy percent of the events happen between 2 a.m. and 6 a.m. local time, which is when the app store runs automatic updates while the phone is charging. The rate is about 0.4 percent of Orion Lite devices per week, and it's roughly three times higher on devices with less than ten percent free storage.

Do we know whether it happens on Orion as well, or only Orion Lite?

Orion has it too, but about eight times less often. Orion Lite has slower storage and less memory, which seems to matter.

Elena, can you walk us through the dumps?

Sure. I used the order Yuna described at the tech share three weeks ago. The reset reason is the system server watchdog. The main thread of the system server is blocked waiting for a binder reply. When I looked at the system server's binder thread pool, all thirty-one threads were busy. Not one was free to handle new requests, so every incoming call, including the one from the main thread, just waited.

What were the thirty-one threads doing?

Twenty-seven of them were waiting on the same lock inside the package manager. Three were handling unrelated requests normally. And one thread was holding the lock while writing package information to storage. That write had been in progress for over forty seconds.

Forty seconds for a write? That's absurd. The package information file is usually a few megabytes.

On a nearly full device, it's not absurd at all, unfortunately. When the file system has very little free space, it has to run garbage collection before it can write new data, moving existing blocks around to free up room. On slower storage, like Orion Lite's, that can take tens of seconds, and the writing thread is blocked the whole time.

That explains the storage correlation. But there's one more piece. The twenty-seven waiting threads were almost all serving the same app: a popular third-party launcher. It was calling the package manager repeatedly to rebuild its app list.

Which launcher is it? I may already have a contact there.

It's one of the top three launchers on our store, with several million installs on Orion Lite. Every time any app is installed or updated, the system sends a broadcast. This launcher reacts by querying the full list of apps, not once, but once for each of several categories, about twenty calls in total. During the night, when dozens of apps update in a row, it generates hundreds of calls in a few minutes.

Normally, those calls are fast, so it doesn't matter. But if they arrive while the package manager lock is held for forty seconds, every one of them grabs a binder thread and waits. Within seconds, the pool is exhausted.

So the launcher isn't doing anything forbidden; it's just inefficient. That matters for how I approach the developer. If I tell them their app is crashing our phones, they'll be defensive. If I tell them their app is making excessive calls and show them a better pattern, they'll usually fix it.

Exactly. And to be fair to them, our documentation doesn't warn against this pattern. The better approach is to listen for the specific package that changed and update only that entry, instead of rebuilding everything.

So we have three layers: a nearly full device makes writes slow, the package manager holds a lock during a slow write, and an app floods the system with calls during that window. Which ones do we fix?

The one we fully control, and the one that matters most, is the lock. The package manager shouldn't hold its main lock while writing to storage. The usual pattern is to take a snapshot of the data under the lock, release the lock, and then write the snapshot to disk. Then a slow write only delays the write, not every query.

I checked, and the newer Android version already does exactly that. It was changed upstream about a year ago. Orion Lite's framework is based on the older version, so it doesn't have the change. Backporting it is possible, but the package manager is one of the most sensitive parts of the system. A mistake there could corrupt the package database.

From the storage side, there's something we can do independently. We can make the file system start background garbage collection earlier when free space is low, during idle time, so that space is already prepared when a write arrives. It doesn't eliminate slow writes, but in our tests, it cuts the worst case from forty seconds to under ten.

Under ten seconds would be enough to avoid the watchdog in almost every case. The watchdog fires at sixty seconds, and the binder pool would recover once the lock is released.

And on the app side, I can contact the launcher developer next week. If they ship a fix within a month, that removes the flood for millions of users, on every device, not just ours.

Let's be concrete about risk and timing. Orion Lite's next release, 1.4.0, has its content cutoff on September 3rd, after Grace moved it. What can realistically make that?

The file system tuning is a configuration change plus a small kernel patch that's already upstream. I can have it ready on Monday, and the storage stress profile can validate it by Wednesday.

The package manager backport is a different story. Even if I start today, I wouldn't want it in a release without at least two weeks of testing, including upgrade testing with real user data. That means 1.5.0 in October at the earliest.

Is there a smaller framework change we could make for 1.4.0? For example, limiting how many binder threads a single app can occupy at once. If the launcher could use at most eight threads, the other twenty-three would stay free for the system.

That's an interesting idea, but I'd be careful. A per-app limit changes the behavior for every app, including well-behaved ones that legitimately make parallel calls. We could easily cause slowdowns elsewhere. I'd rather not introduce that kind of global change in a maintenance release.

Then the plan is: file system tuning in 1.4.0 as the immediate mitigation, the package manager fix in 1.5.0 as the real fix, and Lisa working with the launcher developer in parallel. Does anyone disagree?

No disagreement. I'd only add that the alarm issue deserves its own look.

Even if the freeze becomes rare, a soft reboot at the wrong moment can still swallow an alarm. That's the part users are angriest about. Shouldn't the alarm survive a system server restart?

It should, and normally it does. Alarms are stored persistently and rescheduled at boot. But if the restart happens within a minute of the alarm time, the alarm is rescheduled for a time that has already passed, and the current logic treats it as missed rather than firing it late.

That sounds like a separate bug that's worth fixing regardless of the freeze. An alarm that's two minutes late is far better than no alarm at all.

Agreed. If an alarm's time passed less than ten minutes ago when the system restarts, it should fire immediately. That's a small, contained change in the alarm service, and I think it can make 1.4.0.

Let's check our exposure beyond Orion Lite. Orion has the same framework version, so it has the same lock problem. It's just less visible because its storage is faster. Should the file system tuning go into Orion's next build as well?

I'd say yes, but in October, not as an exception. Orion's September build is already locked and submitted, and the rate on Orion is low enough that a month's wait is reasonable. I'll make sure the change goes into the common kernel branch, so it reaches both products.

For Atlas and Atlas Lite, the package manager already has the newer locking from the upstream framework, so the main risk is gone. But Atlas Lite will ship with only four gigabytes of memory in some markets, and its storage is a cost-reduced part. I'd want the same overnight update scenario in its test plan from the start.

I'll pass that to Nina for the Atlas Lite test plan. It fits nicely with the memory pressure tests she's already defining.

And the launcher developer's fix, if they make one, helps every device at once, including the new ones. That's another reason to approach them constructively.

This issue has probably existed for a long time, getting worse as devices filled up. Why did it take two weeks of field reports for us to notice?

Two reasons. First, our test devices are almost never full. We reset them often, so they rarely have less than half of their storage free. Second, the freezes happen at night, when nobody is watching, and a soft reboot doesn't show up as a crash in our main dashboard. It shows up as a watchdog event, which we only review weekly.

Then we should keep a group of test devices deliberately filled to ninety percent or more, and leave them that way. Real users live with nearly full phones for months. Our lab devices should too.

And watchdog restarts should be on the main dashboard, next to crashes, with the same alert thresholds. From the user's point of view, a minute-long freeze followed by a restart is worse than most crashes.

Agreed on both. I'll set up a group of twenty full devices in the stress lab next week, and ask Kyle to add watchdog restarts to the main field dashboard.

That's a cheap change with a big payoff. Thank you. Let's review the first week of data from the full devices at our next check-in.

For validation, I'd like to reproduce the original scenario in the lab, not just test each fix separately. We can fill test devices to ninety-five percent storage, install the same launcher, and trigger fifty app updates in a row while the device is charging with the screen off.

Please measure the longest package manager write in each run as well. That tells us directly whether the file system tuning is doing its job, even if the freeze doesn't happen.

And for the alarm fix, a simple test: schedule an alarm, force a system server restart thirty seconds before it, and check that it fires within a minute of the original time.

Let me summarize. Yuna: file system garbage collection tuning, ready Monday, validated by Wednesday. Elena: the alarm catch-up fix for 1.4.0, and start the package manager backport for 1.5.0. Lisa: contact the launcher developer next week with the data and a recommended pattern. Brian: add a warning about this pattern to our developer documentation. And SQE will build the full reproduction scenario and validate everything together.

I'll send the developer a sample of the call pattern from the dumps, anonymized, so they can see exactly what their app is doing. I'll also offer them early access to our test devices with nearly full storage, so they can reproduce the problem themselves and verify their fix before it ships.

That's a great idea. Developers fix things much faster when they can see the problem on their own desk.

I'll share the file system tuning patch with the upstream maintainers as well, along with our measurements. If they find it useful, other device makers with slower storage will benefit too.

Thank you, everyone. Our next check-in is Wednesday, September 2nd, at 10 a.m. Korea time, the day before the 1.4.0 cutoff. By then, I'd like to see the reproduction results with Yuna's tuning, the alarm test, and a first reply from the launcher developer, if we have one.

Thanks, Soyeon. This was a good example of the triage order from the tech share paying off. Three weeks ago, a dump like this would have bounced between teams for days.