Responding to File System Corruption after Sudden Power Loss

Good afternoon, everyone, and good morning to Stefan in Munich. Today's issue is rare, but it's one of the most serious kinds of problems a phone can have: users losing their data. Service centers have reported a growing number of Orion devices that can't boot after a sudden power loss and need to be reset. I want to understand why and what we can do, quickly.

Kyle, Field Quality. I've combined the service center reports with telemetry from before the failures.

Marcus Lee, Global Service Operations. I manage the repair process across our service centers, so I see these devices when they come in.

Yuna, Kernel and BSP. I work on the file system and power management, so this is right in the middle of my area.

Hyunsoo, also Kernel and BSP. I worked on the storage driver this summer, so I know the storage side well by now.

Stefan Brandt from Kestrel Memory. We supply the storage for Orion. When I hear 'power loss' and 'corruption', I want to be here.

Marcus, could you describe what's coming into the service centers?

The customer usually says the phone died, either because the battery ran out completely or because it froze and they forced it off. When they charge it and turn it on, it shows a message that the data is corrupted and offers a factory reset. Unless they have a recent backup, they lose their photos and messages. In August, we received about four hundred such devices worldwide, compared with about a hundred and twenty in May.

Three times higher in three months. Is there any pattern in where they come from?

Yes, and it surprised me. A disproportionate number come from cold regions in the southern hemisphere, where it's winter now: southern Australia, New Zealand, Chile, and Argentina. And many customers mention that the phone died outdoors, while the battery indicator still showed a few percent.

That matches the telemetry. For devices that later came in with corruption, the last reported battery level before the failure was between two and six percent in most cases, and the battery temperature was below five degrees Celsius. Normal devices that shut down from low battery report one percent and shut down cleanly.

So the phone thinks it has a few percent left, but in the cold, the battery can't actually deliver enough power, and the voltage collapses before the system has a chance to shut down properly.

But phones lose power suddenly all the time. Why does it corrupt the file system on these devices and not others?

Our file system is designed to survive sudden power loss. It writes data in a way that, if the power goes away at any moment, the last consistent checkpoint is still intact. But that guarantee depends on one assumption: when the storage device says a write is complete, the data is really on the flash, not just in a temporary cache.

And that's where I should explain something about the firmware we released in July. It enabled a larger write buffer to improve performance. Data in that buffer is volatile until it's flushed to flash. The file system sends an explicit flush command at each checkpoint, and our firmware honors it. So under normal conditions, the guarantee holds.

Under normal conditions. But a flush takes time, and with a larger buffer, it takes longer. I measured it this summer during the timeout investigation: a full flush can take up to six hundred milliseconds with the new firmware, compared with under two hundred before. If the voltage collapses in the middle of a checkpoint's flush, the checkpoint is half written.

I should add that our storage part has power-loss protection for its own internal data, so the storage itself never becomes unusable. But it can't protect data the host hasn't finished flushing. That's a system-level guarantee, not something the storage can provide alone.

So we have a chain again: cold weather makes the battery estimate wrong, the device shuts down abruptly instead of gracefully, and the larger write buffer makes the window for a half-written checkpoint much wider.

Exactly. And in the file system, a half-written checkpoint should still fall back to the previous one. I'm checking whether there's a bug in the recovery logic, because a single bad checkpoint shouldn't require a full reset.

What have you found so far?

Marcus's team sent me disk images from five affected devices, with customer consent. In all five, the newest checkpoint is damaged, but the previous checkpoint is intact. The file system check tool at boot detects the damage, but instead of rolling back to the previous checkpoint, it reports the volume as corrupted and gives up. That's a policy setting in our boot scripts, chosen years ago to be conservative.

Wait. So the data is actually there, and recoverable?

In all five images I looked at, yes. I ran the check tool manually in repair mode, and it rolled back to the previous checkpoint. The only data lost was whatever was written in the last thirty seconds before the power loss. Photos, messages, and apps were all intact.

That changes everything for us. Right now, our technicians reset these devices because that's what the screen recommends. If there's a repair mode that works, we could recover data for most of these customers instead of wiping it.

That's the most urgent action, then. Every day we keep resetting devices, we're deleting data that could have been saved.

What would my technicians need to run the repair mode safely?

The service tool already has a way to run commands on a device in recovery mode. I can write a script that first makes a raw backup of the data partition to the service center's computer, then runs the check tool in repair mode, then verifies the result. The backup step is important: if the repair fails for any reason, the original data is still preserved.

Encrypted data is a concern, though. The data partition is encrypted with keys tied to the user's lock screen. Can the repair work without the user's PIN?

Good question. The file system metadata that needs repair isn't encrypted, only the file contents are. So the repair doesn't need the user's PIN, and it never sees the user's files in readable form. That's important for privacy, and it means technicians can run it without asking for the customer's credentials.

Then I'd like to issue a service bulletin as soon as the script is tested: for any Orion device showing the corruption message, run the repair procedure before considering a reset. How quickly can the script be ready?

I can have it tested on twenty artificially corrupted devices by Monday. If you can hold any affected devices that come in until then, rather than resetting them, we can recover those customers' data too.

I'll send an instruction to all service centers today: hold these devices, offer the customer a loaner phone, and don't reset anything until Monday.

Now let's prevent it from happening in the first place. What can we change in the device software?

Three things. First, change the boot policy so the check tool automatically rolls back to the previous checkpoint instead of giving up. That alone would turn almost all of these cases into a normal boot with thirty seconds of lost data. Second, make the low-battery shutdown temperature-aware: in the cold, start the shutdown at five percent instead of one, because the battery can't be trusted below that.

Third, when the battery is critically low, we can ask the storage to flush more often, so there's less data sitting in the volatile buffer at any moment. It costs a little performance, but at five percent battery, nobody is running benchmarks.

We can help with the third one. Our firmware supports a mode that limits the size of the write buffer when the host requests it. Instead of flushing more often, you can simply tell the storage to use a smaller buffer when the battery is low. It's cleaner, and the flush time drops back to under two hundred milliseconds.

That's better than what I proposed. I didn't know that mode existed. Could you send me the command specification?

Of course, today. I'd also recommend using it during the shutdown sequence in general, not only at low battery. Any shutdown benefits from a smaller buffer.

Which of these can go into the October update, and which need more time?

The boot policy change is small, but it's sensitive: it changes what happens to user data after a crash. I want it tested very thoroughly, including on devices with real corruption from the field. The temperature-aware shutdown is simple and low risk. Both can make the October cutoff on the twenty-fifth if I start this week.

The buffer limit at shutdown needs Stefan's specification and storage stress testing, which is now automatic for any storage change. I'd target October as well, but if the stress results aren't clean in time, it can wait for November without much risk, because the other two changes already address most cases.

Should we consider an emergency update, given that this involves data loss? It's winter in the southern hemisphere now, and it will be winter in the north in a few months.

I thought about that. The rate is still low, about one in fifty thousand devices per month, and the service center repair procedure will protect most affected users' data starting next week. Southern winter ends in a few weeks, and the October update will be out well before northern winter. I think October is the right balance, as long as the service bulletin goes out on Monday.

Is this only Orion? The boot policy and the low-battery logic are shared across our devices.

Orion Lite shows the same pattern, but about five times less often, because its storage uses the older firmware without the large buffer. Nova Tab is too new to have data. Pulse Watch has a different storage design and no reports.

The boot policy fix applies to every device anyway, and it's the most important one. I'll make it a common change, so Orion Lite and Nova Tab get it in October too. And for Atlas, it should be the default from day one.

Let me ask the uncomfortable question. Why didn't our testing catch this before launch? We have power-cut rigs in the lab.

We do, and they ran for weeks. But every one of those tests ran at room temperature, with a lab power supply instead of a real battery. At twenty-five degrees the battery estimate is accurate, so the phone always shuts down cleanly before the voltage collapses. The failure needs two things at once: a cold battery and a heavy write in progress.

On our side, the firmware qualification also ran at room temperature for the flush-time measurement. The six hundred milliseconds only shows up when the buffer is full, and we didn't model a full buffer during a power cut. That's a gap in our test plan, and I'll own it.

I appreciate that. Here's what I'd propose. We move two of the power-cut rigs into the thermal chamber and run them at minus five degrees with real batteries. Each cycle fills the write buffer, drains the battery to the shutdown threshold, and checks whether the device boots cleanly afterwards.

How many cycles do we need before we trust the fix?

Before the fix, I'd expect to reproduce the corruption within a few hundred cycles. After the fix, I want at least five thousand clean cycles across both rigs. That's about four days of chamber time, so it fits before the October update freeze.

And going forward, cold-temperature power-cut testing becomes a standard entry in the release checklist, not a special project. Atlas will run it from its first stable build.

Before we wrap up, we need to decide what we tell customers. Most of these reports come from northern regions, and winter is only getting started there. I don't want service centers to be the first place people learn about this.

I agree, but we should be careful not to sound alarming. The rate is still low, and most users will never see it. A simple support article with practical advice is the right level for now.

Then the article says three things. Keep the phone charged above twenty percent in very cold weather, keep it in an inner pocket when you're outside, and if it doesn't start after a shutdown, bring it to a service center before trying a factory reset, because we can usually recover the data.

That last point matters most. In the reports we reviewed, a lot of the data loss happened because users reset the phone themselves. The repair script can't help after a reset.

I'll have the draft ready tomorrow and run it past legal. Once the software fix ships, we update the article to say the issue is resolved.

Let me summarize. Marcus: instruct service centers today to hold affected devices and offer loaners. Yuna: the repair script tested by Monday, then the boot policy and temperature-aware shutdown for October. Hyunsoo: the shutdown buffer limit using Stefan's specification. Stefan: send the specification today. Kyle: add battery temperature and level at shutdown to the corruption dashboard. And Marcus and I will issue the service bulletin on Monday.

Thank you. This is the best news I've had about this problem. Customers who thought their photos were gone will be very happy.

Thank you, everyone. We'll meet again on Monday, September 7th, at 3 p.m. Korea time, to review the repair script results before the bulletin goes out.