Responding to Field Reports of Random Reboots on an Iberian Carrier Network

Good afternoon, and good morning to our colleagues in Madrid and Lisbon. Thank you for joining. I'm Hannah, field quality manager for Orion. We have a field issue that's growing fast in one region, and I want to make sure every team that can help is in the same room. Let's do quick introductions.

Kyle, Field Quality. I manage the telemetry dashboards and the crash analytics pipeline, so I'll be presenting most of the numbers today.

Seojin Lim, Modem Software. My team integrates the modem firmware and owns the interface between the modem and the application processor.

Wei Zhang from Qorvix modem engineering, joining from Shanghai. I'm the technical contact for the modem protocol stack on your Orion devices.

Lucia Fernandez, carrier engineering for Southern Europe, based in Madrid. I'm the main technical link to the carrier where we're seeing this problem.

Pedro Alves, same team, in Lisbon. I handle the Portuguese side of the same carrier group, and I've been collecting logs from our test devices on their network this week.

Thanks, everyone. Kyle, could you start us off with what the field data shows?

Sure. Starting on July 30th, the unexpected reboot rate for Orion devices on this carrier's network in Spain and Portugal jumped from our normal baseline of about 0.03 percent of devices per day to 0.9 percent per day. That's a thirty-fold increase. On every other carrier in Europe, the rate is flat. And the same Orion software on the same carrier's network in other countries is also flat. So this is very specific to one carrier group in one region.

And did anything change on our side around July 30th? An OTA, a configuration push?

Nothing. The last Orion OTA in that region was on July 9th, and the reboot rate stayed flat for three weeks after it. We also haven't pushed any server-side configuration changes since mid-July. That's what makes us think the trigger is on the network side.

How is this showing up for customers? I've seen some noise on social media, but I'd like the full picture.

Customer contacts about random restarts in Spain are up about four times compared to last month. Most users say the phone restarts while they're on the move, often on the train or in the car. A few mention it happening during calls, which is obviously the worst moment. There are two threads on a popular Spanish tech forum with several hundred replies, and one tech news site picked it up yesterday.

Seojin, what do the crash logs tell us?

The reboots are all the same type: a modem subsystem crash that escalates into a full device restart. Normally, when the modem firmware crashes, the application processor restarts only the modem, and the user sees a few seconds without signal. But on Orion, for this carrier's software configuration, the policy is set to trigger a full system restart with a ramdump instead. That policy was requested during the carrier's acceptance testing last year, so they could collect complete dumps. It was supposed to be reverted before launch, and it wasn't.

So there are really two problems here: the modem is crashing, and when it crashes, we take down the whole phone instead of just restarting the modem.

Exactly. And the second problem is entirely on us. That's the frustrating part, because it turned a minor, recoverable glitch into a very visible reboot.

On the first problem, I've analyzed twelve of the modem dumps Pedro collected. All twelve show the same assertion in the handover module. It happens during a handover from a 5G cell to an LTE cell, when the network sends a measurement configuration with a combination of parameters our firmware didn't expect. Specifically, the configuration includes a timer value that's outside the range in our parameter table, and the firmware asserts instead of clamping the value.

That fits what the carrier told me. On July 29th, they rolled out a network optimization across their 5G sites in Spain and Portugal, which changed handover parameters to reduce dropped calls in high-speed scenarios, like trains and highways. They say it's fully compliant with the standard, and other device makers haven't reported problems.

I believe them that it's compliant. The value is unusual, but it's valid according to the specification. The bug is in our firmware. We should handle any valid value gracefully, and we didn't. I'll be straightforward about that.

That also explains the pattern in my test drives. On the Lisbon to Porto train, I reproduced the reboot four times in three hours, always within a few seconds of the phone switching from 5G to LTE. In the city, where handovers happen at lower speed, I couldn't reproduce it at all in two days.

So we have a clear root cause, which is great. Now let's talk about how to get relief to customers as fast as possible. I see three levers: ask the carrier to roll back the parameter change, fix our restart policy, and fix the modem firmware. Let's go through them in order of speed.

I'll be realistic about the first option. I already asked informally yesterday, and they were not enthusiastic. The change reduced dropped calls on their high-speed routes by almost twenty percent, and they don't want to roll back an improvement for all their customers because of a bug in one device. I understand their position. We might get a partial rollback on specific train routes as a goodwill gesture, but I wouldn't count on it.

Fair enough. We shouldn't ask them to fix our bug. Seojin, how fast can we change the restart policy?

This is the good news. The policy is controlled by a configuration flag that we can update through our device configuration service, without a full OTA. If we flip it, a modem crash restarts only the modem. The user loses signal for about five seconds instead of losing the whole phone for a minute. Technically, we could push it to the affected devices within twenty-four hours.

I want to raise one concern about that. If we flip the flag, the reboots disappear from our telemetry, but the modem crashes don't. We'd lose visibility of the real problem, and a five-second signal loss during a call still drops the call. I don't want us to declare victory because the dashboard turns green.

That's a very good point. We can keep visibility by reporting modem subsystem restarts as a separate telemetry event. The pipeline already supports it; we just haven't been watching that metric for this carrier. And you're right that it's a mitigation, not a fix.

Then the real fix is the firmware. Wei, what's the timeline?

The code change itself is small: we clamp the timer value to the supported range and log a warning instead of asserting. My team can deliver a patched firmware to Seojin's team by Monday, August 10th. But firmware changes in the handover module need regression testing on our side across the major network vendors' equipment, which normally takes about a week. So a fully validated firmware would be ready around August 14th.

Then on our side, the firmware has to go into an Orion OTA, which means our own regression plus carrier approval. Lucia, how long does this carrier usually take to approve a maintenance OTA?

Normally about ten working days. But given the customer impact, and the fact that it's a modem fix for their network, I think I can get them to fast-track it to five. They want this fixed as much as we do, because their store staff are getting the complaints too.

Before we move on, I want to understand our exposure beyond Orion. Wei, is the same handover module used in other chipsets or firmware branches that we ship?

Yes, unfortunately. The same module is used in the firmware for Orion Lite and for the Orion Fold, which hasn't launched yet. The parameter table is identical, so in theory, both are vulnerable to the same configuration. I'd recommend we include them in the fix rather than wait to see crashes.

That's interesting, because I don't see any increase for Orion Lite on that carrier. Is it possible that Orion Lite doesn't support the 5G bands where the new parameters were deployed?

That would be my guess. Orion Lite only supports the lower 5G bands, and the carrier told me the optimization was rolled out on their mid-band sites first, especially along the main rail corridors. If they extend it to the lower bands later this year, Orion Lite users would be exposed too.

Then we have a window, but not a long one. Seojin, can the Orion Lite and Orion Fold firmware pick up the same fix without a separate validation cycle?

For Orion Fold, yes, because it's still in development and the fix can ride along with the next firmware drop before launch. For Orion Lite, it would go into the September maintenance release. It's a lower priority there, since nobody is affected today, but I'd still like it in before the carrier expands the rollout.

I'll ask the carrier's radio team whether they have a date for the low-band rollout. If they plan it for October, the September release gives us enough margin. If it's earlier, we'll need to rethink.

Let's talk about communication. Customers are already discussing this publicly. I don't want to stay silent until the firmware ships, but I also don't want to promise a date we might miss. Thoughts?

From what I've seen on the forums, people mostly want to know that we're aware and working on it. A short acknowledgment from our official community account would go a long way. Something like: we're aware of unexpected restarts on some networks, a fix is in progress, and an improvement is rolling out this week.

I'd support that, as long as we don't name the carrier. Publicly blaming the network would damage the relationship, and technically, the bug is ours.

Agreed, please don't name them. I'll also share the draft statement with the carrier before it goes out, as a courtesy. They appreciate being told before they read it on a forum.

Good. I'll draft the statement with our communications team today and send it to Lucia for the courtesy review. We'll publish it once the configuration flag starts rolling out, so we can honestly say an improvement is on its way.

Before we close, I'd like to ask how the debug restart policy ended up in a production build. If it happened for this carrier, it could have happened for others.

I checked that last night. It was added to the carrier-specific configuration file during acceptance testing, with a comment saying 'remove before launch'. There was no automated check for debug settings in carrier configurations, only in the main build configuration. I've already scanned all the other carrier configurations for Orion and Orion Lite, and this is the only one with that setting.

That's a relief. Let's make sure it can't happen again. Can we extend the build check to cover carrier configurations, so a debug-only setting fails the release build?

Yes. It's a straightforward change to the release checks. I'll have it in place by the end of next week, and I'll present it at the release council so other programs adopt it too.

Let me summarize the actions. Seojin: push the restart policy flag to affected devices in Spain and Portugal within twenty-four hours, and enable separate telemetry for modem subsystem restarts. Wei: deliver the patched firmware by Monday, August 10th, and the fully validated version by Friday, August 14th. Lucia: ask the carrier to fast-track OTA approval, and explore a partial rollback on high-speed train routes as a goodwill gesture. Pedro: verify the configuration flag on the Lisbon to Porto route tomorrow. Kyle: build a dashboard for modem restarts on this carrier. And I'll handle the customer statement.

I'll take the 8 a.m. train tomorrow and report back by the afternoon. I'll bring two devices with the flag and one without, so we can compare them on exactly the same route and at the same time.

One addition from my side. Before we push the flag to everyone in the region, I'd like to roll it out to five percent of affected devices first and watch the numbers for twelve hours. It's a low-risk change, but changing crash handling behavior on a live population is something I never like to do all at once.

Agreed, that's a sensible precaution. Twelve hours still keeps us well within the twenty-four-hour target.

Understood. I'll also share the list of other parameter combinations we plan to test, in case Lucia's team knows of other unusual configurations in the region.

Thank you, everyone. I appreciate the honesty from all sides today; it made this a lot faster. Let's meet again on Monday, August 10th, at 4 p.m. Korea time, which is 9 a.m. in Madrid and 8 a.m. in Lisbon, to review the effect of the configuration flag and the patched firmware delivery.

Sounds good. I'll have an answer from the carrier on the approval timeline by then.

And I'll share the first post-flag numbers in the channel on Saturday morning, so we're not waiting until Monday to see whether it worked.