Pausing a Staged Rollout over Wi-Fi Disconnection Reports

Good morning, everyone. Thanks for joining so quickly. I'm Grace from Release Management. Two days ago, we started the staged rollout of the fixed Orion maintenance update to one percent of users. The storage metrics are perfectly clean, which is great news. But since yesterday afternoon, Field Quality has been seeing a spike in Wi-Fi complaints from that one percent group. I've paused the expansion to ten percent until we understand it.

Hannah, Field Quality. Kyle is with me, and he has the telemetry breakdown.

Minho Seo, Connectivity Software lead. My team owns the Wi-Fi framework and driver integration. Haeun is our Wi-Fi driver expert.

Hi, Haeun Kim. I've been going through the driver logs from affected devices since last night.

Victor Lin from Qorvix connectivity engineering, in Hsinchu. I support the Wi-Fi chip and firmware on Orion.

Kyle, what are we seeing exactly?

In the one percent group, user feedback mentioning Wi-Fi went up about six times compared to the same population before the update. Telemetry confirms it: Wi-Fi disconnect events per device per day went from about one to almost nine. Most users say the same thing: the phone keeps dropping off their home Wi-Fi every few minutes and reconnecting. It's not a crash, but it's extremely annoying, and some users switched to mobile data and burned through their data plans.

Is it everyone in the one percent group, or a subset?

A clear subset. About eighty percent of the extra disconnects come from devices connected to routers from one manufacturer, and specifically to networks configured in a mixed security mode, where the router accepts both older and newer security standards at the same time. That configuration is the default on that manufacturer's popular home routers in Korea and Japan.

So even though only one percent of users have the update, if we expanded, a large share of users in Korea and Japan would be affected.

Exactly. Based on our router telemetry, roughly fifteen percent of Orion users in those two countries connect to that router family at home. That's why pausing at one percent was the right call.

Haeun, what do the driver logs show?

The pattern is very consistent. The phone is connected normally, then the router starts a routine key refresh, which happens every few minutes on these routers. During that refresh, our device responds with a message the router doesn't accept, the handshake times out, and the router disconnects us. Then we reconnect from scratch, which takes about three seconds, and the cycle repeats at the next key refresh.

And this didn't happen on the previous build, which tells us something changed in that exchange. The maintenance update includes a new Wi-Fi firmware from Qorvix. Victor, can you tell us what changed in the firmware's handling of key refresh?

Yes. The new firmware includes a security hardening change. It strictly enforces a protection feature for management messages whenever the network advertises support for it. In mixed security mode, these routers advertise the feature as optional, but some of their firmware versions don't actually implement it correctly during key refresh. The old firmware was more lenient and simply ignored the inconsistency. The new firmware treats it as an error.

So technically, our device is now doing the correct thing, and the routers are the ones that don't follow the standard.

That's a fair summary from a standards perspective. But I understand that customers don't care who is technically correct. Their Wi-Fi worked yesterday, and today it doesn't.

Exactly. And I'd add that other phones work fine with these routers, which means other device makers handle this case more pragmatically. We have to as well.

How did this get through our testing? We tested this build for weeks.

Our Wi-Fi interoperability lab has about sixty router models, including two from this manufacturer. But both of ours run newer router firmware, where the bug in the router is already fixed. The affected customers mostly have routers that haven't been updated in years. In other words, we tested against the routers we wish customers had, not the ones they actually have.

We could fix that with telemetry. We know which router models and firmware versions our users connect to most. If the lab mirrored the top fifty combinations from real usage, including old firmware, we'd catch this kind of thing before release.

I like that. It means keeping some routers deliberately outdated, which feels strange, but that's what reality looks like.

On our side, I have to admit that our release testing for this firmware was also done against current router firmware. We assumed customers keep their routers updated, and that assumption was clearly wrong for this market. I'll raise it with our test team, because other customers of ours sell phones in Korea and Japan too.

That's appreciated. It also suggests a broader lesson for our release process: any firmware change that touches security behavior should get a staged rollout that starts smaller and stays longer at the first step, because interoperability problems often only appear in the field.

Agreed. I'll propose that rule at the next release council: security-related firmware changes start at half a percent and stay there for at least seventy-two hours.

Let's talk about options. I'm conscious that the one percent group is suffering right now, and that the rest of the update, including the storage fix, is waiting.

From our side, there are two possible firmware changes. One: revert the strict enforcement entirely. That's quick, but it also removes a security improvement that protects against a known class of attacks. Two: keep strict enforcement in general, but fall back to the lenient behavior when the network advertises the feature as optional and the handshake fails in this specific way. That's safer, but it needs about a week to implement and test.

There's also something we can do on the driver side without new firmware. The driver tells the firmware which security features to use when it connects. For networks in mixed mode, we can tell the firmware not to require the protection feature. It has the same effect as the fallback for these routers, and it's a configuration change in our driver, so it could be ready in two days.

The downside is that it applies to all mixed-mode networks, not just the buggy routers. Devices on correctly working mixed-mode routers would lose the extra protection too. But that's exactly how the previous build behaved, so we wouldn't be any worse off than before the update.

That's an important point. We'd be returning to known, previously shipped behavior, not inventing something new. I like the shape of that as an immediate step, with Victor's smarter firmware fallback as the long-term solution.

What about the one percent group right now? Can we help them faster than a new build?

Yes, actually. The driver setting I described can also be pushed through our device configuration service, the same way Seojin's team changed the modem restart policy in Spain. If we push it to the one percent group today, they should see relief within hours, without waiting for a new build.

That's the best news I've heard today. Let's do that first, on the same cautious pattern: a small slice of the one percent group, watch for a few hours, then all of them.

Before we move on, I want to make sure we're comfortable with the security side of the configuration change. We're deliberately turning off a protection on some networks. Should Product Security be part of this decision?

Yes, I think they should at least review it. I'll send Jiho a short summary today. My expectation is that he'll be comfortable, because we're only restoring the behavior we've shipped for the last two years on these networks, and the protection still applies on networks that require it rather than offering it as optional.

For context, the attack this feature protects against requires the attacker to be physically close to the victim's network and to actively interfere with it. It's a real risk, but not something that affects most home users on a daily basis. That's why many vendors are cautious about enforcing it strictly on mixed-mode networks.

That's helpful context. But I'd like the long-term solution to restore the protection wherever the router actually supports it correctly. Victor, does your smarter fallback do that?

Yes. With the firmware fallback, the device tries the protected exchange first, and only if the router fails in this specific way does it fall back for that network. Routers that implement it correctly keep the protection. Once that firmware ships, we can remove the driver configuration entirely.

I'll add an expiry note to the configuration, so it's clear it's temporary and tied to Victor's firmware. Otherwise, in a year, nobody will remember why it's there.

Is any other product shipping the same Wi-Fi firmware? I don't want to find the same problem in another rollout next week.

The same firmware branch is used on Orion Fold and on Nova Tab S, which launches tomorrow. Orion Lite uses an older Wi-Fi chip with a different firmware line, so it isn't affected.

Nova Tab launches tomorrow. Has anyone checked it?

Not yet, but the same driver configuration applies. Nova Tab's factory build is already final, so the configuration service is the only practical way to protect launch users. I'll ask Irene's team to approve pushing it to Nova Tab devices from day one. Tablets are mostly used at home on Wi-Fi, so the exposure there is even higher than on phones.

I'll set up the same Wi-Fi disconnect metric for Nova Tab, so we can see within a day whether the configuration is working there too.

Now, the rollout. Once the configuration fix is confirmed in the one percent group, can we resume the expansion with the same build plus the configuration?

I think so, as long as the configuration is applied before the update is installed, not after. Otherwise, new users would have a few hours of broken Wi-Fi before the configuration reaches them. The configuration service can target devices by build number, so we can set it up to apply to every device that receives the update.

I'd want twenty-four hours of clean data from the one percent group after the configuration is applied before we go to ten percent. Wi-Fi problems tend to show up in the evening, when people are home, so one evening of data is the minimum.

Agreed. If tonight's data is clean, we resume to ten percent on Saturday morning, and continue with the original halt criteria plus a new one: Wi-Fi disconnects per device per day must stay within twenty percent of the pre-update baseline.

For the users already affected, I'd like to post a short note in our community forum. Something like: we're aware some users are experiencing Wi-Fi disconnections after the latest update, a fix is rolling out now, and no action is required. We shouldn't name the router manufacturer publicly.

Agreed on not naming them. But I'd like our partnership team to contact the router manufacturer privately. They may want to push a firmware update to their routers, which would help every device, not just ours.

We've worked with that manufacturer before on interoperability. I'm happy to share our analysis with them, if your team agrees.

Let me summarize. Haeun: prepare the driver configuration today, push it to a small slice of the one percent group by early afternoon, then to all of them by this evening. Minho: set up the configuration to apply automatically to every device that receives the update. Kyle: monitor Wi-Fi disconnects tonight and report tomorrow morning. Victor: start the smarter firmware fallback, target one week, and share the analysis with the router manufacturer through our partnership team. Hannah: post the community note once the configuration is live. And I'll decide on resuming to ten percent on Saturday morning based on Kyle's data.

I'll also add three of the affected router models with their old firmware to the lab this week, so we can verify both the configuration fix and Victor's firmware against real customer setups.

And I'll send Haeun the list of the top fifty router and firmware combinations from telemetry by Monday.

Thank you, everyone. This is exactly why we roll out in stages. One percent of users had a bad two days, which is unfortunate, but it's far better than fifteen percent of Korea and Japan having a bad week. Let's meet again on Saturday, August 22nd, at 9 a.m. Korea time, to make the ten percent decision.

Thank you. I'll have a firmware plan for you by then, including a test build that Haeun can try against the old routers in your lab. If the fallback works on those, we'll have much more confidence before it goes anywhere near customers.

And I'll make sure Jiho's review of the configuration is done before the evening push, so we're not waiting on approvals at the last minute.

Thanks, Grace. I'll draft the community note now so it's ready when the configuration goes live, and I'll share it with the Korea and Japan support teams as well, so their answers match what we post publicly.