Escalating a System Server Memory Leak before the Orion Fold Factory Build

Good morning, everyone. I'm Sarah, and as most of you know, I've moved from the Pulse Watch launch straight onto Orion Fold. We're thirty days from launch and nine days from the factory build cutoff on August 28th. I called this meeting because SQE found a memory leak over the weekend that could be a launch blocker, and I want a clear plan by the end of this hour.

Hyemi, SQE lead for Orion Fold. Before we get to the leak, a quick piece of good news: the hard lockup fix from last week is holding up. We've run twenty-two thousand folds on the robots with the fix, and we've had zero freezes.

That's a relief. The access timeout dumps pointed exactly at the interrupt handler, so we were fairly confident, but it's good to see it confirmed at scale.

Brian from Platform Framework, and Taemin from my team, who's been looking at the heap dumps since Sunday.

Hi. Taemin Ko, Framework. I work on window management and multi-display support.

Woojin, Display Software. I suspect part of this leak may end up in our area, so I'm glad to be here early.

Hyemi, what did you find?

Because the robots now run for days without freezing, we were able to see a much slower problem for the first time. The system server's memory grows by about one point two megabytes every time the device is folded or unfolded, and it never comes back down. After roughly two thousand folds, the system server reaches its memory limit, the low memory killer terminates it, and the device does a soft reboot: the screen goes black, the boot animation plays, and everything restarts in about twenty seconds.

Two thousand folds sounds like a lot. How long would that take for a real user?

Based on our usage studies from the previous foldable, the median user folds about eighty times a day, and heavy users fold up to two hundred times. So a median user would hit it after about twenty-five days without a reboot, and a heavy user after ten. Most people restart their phones less often than that, so in practice, many users would see a random restart every few weeks.

And a soft reboot loses whatever the user was doing, which reviewers will notice in their long-term reviews. It's not as dramatic as a freeze, but it's a launch blocker in my view.

Taemin, what do the heap dumps show?

There are actually two separate leaks, and they add up to the one point two megabytes. The first one is in managed memory, the Java heap. It's about two hundred kilobytes per fold. Every fold creates a new configuration listener for the taskbar feature on the main screen, and the old listener is never unregistered. So we accumulate thousands of listeners, and each one holds a reference to a full display configuration object.

That's a textbook leak, and the good news is that the fix is simple. Taemin already has a patch that unregisters the listener when the main display turns off. It's been tested on two robots since yesterday, and the managed heap stays flat.

And the second leak?

The second one is in native memory, and it's the bigger one: about one megabyte per fold. The heap dump tools don't show native allocations in as much detail, but what we can see is that the graphics buffer count grows with each fold. It looks like the surfaces used to draw the cover screen aren't being fully released when the cover display turns off.

That's where it crosses into our area. When the cover display turns off, the display driver releases its own buffers, but the composition layer for that display is supposed to be destroyed by the framework. If the framework keeps a reference to the layer, the buffers attached to it stay allocated, even though nothing is ever drawn to them again.

Is it possible the managed leak and the native leak are actually the same bug? If the old listeners hold references to display configuration objects, and those objects hold references to the layers, then fixing the listener leak might also release the buffers.

That was my first hope too, but no. With my patch, the managed heap stays flat, but the native memory still grows by about one megabyte per fold. So they're independent. The layer must be held by something else.

How do we find out what's holding it?

The compositor keeps a list of every layer with its owner and reference count. If we dump that list after, say, a hundred folds, we should see a hundred cover screen layers that should have been destroyed, and the dump tells us which process created them. That narrows it down to one owner in a few minutes.

I can run that today. One robot, a clean boot, a hundred folds, then the layer dump. I'll have it to you by early afternoon.

One suspicion I have: the always-on display on the cover screen. When the device is unfolded, the cover screen turns off, but the always-on clock has its own layer that might be recreated each time instead of reused. That feature was added late, in July.

That's a good lead. Late features are where leaks usually hide, because they get the least soak time. If it's the always-on clock, that's my team's code, and the fix would likely be to reuse a single layer.

I can test that theory quickly too. If I turn off the always-on display on the cover screen and run a hundred folds, and the native memory stays flat, we'll know. I'll run both tests in parallel on two robots, so we get both answers at the same time instead of one after the other.

Let's talk about time. The factory cutoff is August 28th. If we find the native leak today, can both fixes be in the factory build?

For me to sign off on a leak fix, I need a long soak: at least five thousand folds on several robots, to prove the memory stays flat well past the two-thousand-fold point. At our current robot speed, that's about five days. So if a fix is ready by Friday the twenty-first, the soak finishes on the twenty-sixth, which leaves two days of margin.

And if the native fix isn't ready by Friday?

Then I see two options. Option one: ship the factory build with only the managed leak fixed, and put the native fix in the day-one update. A brand-new device can't reach two thousand folds in its first week, so as long as users take the day-one update, they'd never hit the problem. Option two: a mitigation in the factory build that destroys and recreates the whole cover screen composition when the display turns off. It's heavy-handed, but it releases everything.

I'd be cautious about option two. Recreating the whole composition on every fold adds latency to the unfold animation, maybe fifty to eighty milliseconds. Users would feel that, and the fold animation is one of the first things reviewers judge. I'd rather not trade a slow leak for a visibly slower fold.

Understood. So option one is our fallback. Given what we learned on Pulse Watch, I'm comfortable with a day-one update as part of the plan, as long as it's planned from the start rather than added in a panic.

One condition from SQE: if we go with option one, the day-one update must be available before the first customer units ship, and we need the same companion check we used on Pulse Watch, so setup encourages the update.

Since the day-one update is now part of the plan either way, let's talk about logistics. On Pulse Watch, we learned that the update has to be ready before customer units arrive, and that some markets skip Wi-Fi during setup. What's different for Orion Fold?

The biggest difference is carriers. Orion Fold is sold mainly through carriers, and several of them require approval for any update, including a day-one update. The carrier certification build is the factory build, so if the native fix goes into a day-one update, it needs a separate, fast approval from each carrier before launch day.

How long do carriers usually need for a day-one update?

Ryan from my team says most carriers can review a small delta update in five to seven working days, if we tell them in advance that it's coming. The two US carriers that are usually slowest want at least ten. So if we submit the day-one update by September 3rd, we should have approvals before the eighteenth.

That gives us a hard deadline for the native fix even in the fallback case. If we can't fix it by Friday for the factory build, we still need it done, soaked, and submitted by September 3rd. That's two extra weeks, which should be plenty, but it isn't open-ended.

One more consideration. Review units for journalists usually ship about two weeks before launch, so around September 4th. Those units will run the factory build. If the native fix isn't in it, reviewers who use the device for a long-term review could hit the soft reboot after a few weeks, unless they install the update.

Good point. After Nova Tab, I'm not going to rely on reviewers following instructions. If we end up in the fallback case, I'll ask the review program team to flash the day-one update onto review units before they ship. It's manual work, but for a few hundred units, it's worth it.

That makes me even more motivated to have the fix ready by Friday. It would save everyone a lot of work.

How did we miss this until three weeks before launch? I'm not looking for blame, but I need to explain it to the launch director.

Honestly, the hard lockup hid it. Until last week, the robots froze every few hundred folds, so no device ever ran long enough to reach two thousand folds. Fixing one bug uncovered the next. It's a common pattern late in a program.

We could also catch this kind of leak much earlier with an automated memory check. If the nightly test suite folded the device a hundred times and compared system server memory before and after, a growth of one megabyte per fold would be impossible to miss. That test takes less than thirty minutes.

I'd extend it beyond folding. Any repeated user action, like rotating the screen, switching apps, or connecting and disconnecting a charger, should have a simple before-and-after memory comparison. Most leaks are small per action and only become visible after hundreds of repetitions.

Good. Let's add that to the retrospective list, and if it's really only thirty minutes, let's add the folding version to the nightly suite this week.

From the display side, I'd add the graphics buffer count to that check, not just total memory. Buffers are allocated in a separate pool, and on some builds they don't show up in the system server's memory numbers at all. If we only watch the system server, we could miss a buffer leak entirely.

Good point. I'll include both numbers in the nightly report, with a simple threshold: if either grows by more than five percent over a hundred folds, the test fails and opens a ticket automatically.

Let me summarize the plan. The managed leak fix goes into the factory build now, and the soak starts today. For the native leak: Hyemi runs the layer dump and the always-on display test today. If we have a fix by Friday the twenty-first, it goes into the factory build after a five-day soak. If not, it goes into the day-one update, which we plan from today, with the setup check. Option two, recreating the composition, is off the table. Any objections?

No objection. My team will support the layer analysis today, and Jisu can pair with Taemin on the fix if it turns out to involve the compositor.

Thank you. Actions: Taemin merges the managed leak fix today. Hyemi runs the layer dump and the always-on test, with results by 2 p.m. Brian owns the native fix if it's in the always-on clock. Woojin and Jisu support the compositor side. Taemin adds the hundred-fold memory check to the nightly suite. And I'll brief the launch director this afternoon and start the day-one update plan as a precaution.

Thank you, everyone. We'll meet again today at 4 p.m. Korea time to review Hyemi's results, and then daily at 9 a.m. until the factory build is handed off on the twenty-eighth. I know daily meetings are a burden, but at this point in a launch, a day matters.

Understood. I'll post the layer dump in the launch channel the moment I have it.

And I'll start reading the always-on clock code now, so I'm ready if the test confirms it. If it turns out to be somewhere else, I'll at least have ruled out one suspect before the four o'clock meeting.

Thanks, everyone. Sarah, I'll join the launch director briefing if you want a framework voice in the room. Sometimes it helps to hear directly from the team that owns the fix.