A bad server payload from Firebase Analytics made every iOS app using the SDK crash on launch, for two hours on Monday: Reveneau

Why it mattersA third-party SDK that fetches its own config from a vendor's server can crash an already-released app without any code change on the app team's side, so the team's rollout controls and canaries never see the failure and the only defence is the vendor's own.

An iOS app that shipped with Firebase Analytics on Monday could crash on launch without anyone at the app team pushing a change. The Firebase iOS SDK fetches an experiment configuration from Google's servers on startup, and starting at 17:41 US Pacific time on Monday the reply to that fetch contained a nil dictionary key. Every already-released build that decoded it hit an NSInvalidArgumentException and died before the first screen.

The primary account is issue #16728 on Google's own firebase/firebase-ios-sdk repository, opened by a developer whose production app began recording 56 crashes on 56 users in the first 18 minutes across four already-released builds. The trace names -[__NSDictionaryM setObject:forKeyedSubscript:]: key cannot be nil, reached from APMETaskManager handleFetchingExperimentsResponse:data:error: while the SDK was parsing the response to POST https://app-analytics-services.com/sdk-exp. The report identifies Firebase iOS SDK 12.14.0 and Xcode 26.5, installed through Swift Package Manager.

What the numbers looked like

The crash spread as apps started fetching the bad payload. CamperMate on iOS logged 87 crashes on 87 users in the first hour, on three released builds, through React Native Firebase. Another team quoted "about 20k crashes so far," and one wrote that a single service passed 110,000 crash events before the fix landed. None of these teams shipped an update on Monday and none of them had a Firebase A/B test configured. The stack trace was the same in every report: the SDK's experiment worker queue was inside the response handler when the app died.

What Google did

A Firebase engineer, @ncooke3, wrote a pinned summary on the same issue explaining the timeline. Google Analytics for Firebase iOS+ started returning "an incorrectly formatted payload" at 17:41 US Pacific time on 28 September. A fix was fully rolled out to the servers at 19:52 US Pacific the same day, two hours and eleven minutes later, and no SDK update was required on the app team's side. The summary also warns that cached responses on individual devices can keep crashing for up to four hours after the rollout, so remaining crashes were expected to clear on their own by 23:52 US Pacific.

Why an app team could not stop it

The failure sits inside a client library that fetches its own configuration from a vendor's server. The app's rollout controls, feature flags and canary release do not sit between that fetch and the parse, which is why the first crashes appeared on four builds at once for one team and on three for another. Pinning a Firebase SDK version does nothing here, because the server chooses what to send and every version that parses the response is exposed. The only defence is the vendor's own change management on the config it serves.

The issue was labelled priority: p0 on Google's side and is still open at the time of writing, with app teams asking for a client-side guard so that a future malformed payload cannot end a session before the app has a chance to log it.

Source

Primary: Issue #16728, firebase/firebase-ios-sdk on GitHub
Discussion: Hacker News thread on the incident

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.