← The playbook

Quality

App stability is an engineering problem, not a bug queue

Most teams treat stability as a queue. Crashes come in, someone triages the top of the list, a fix ships, and next month the list looks about the same. I've found it works much better to treat stability as an engineering problem with its own process: measure what users actually feel, find the few causes behind the many reports, fix each cause once, and make sure it can't come back. That's how the app I worked on got to and stayed at 99.99% crash-free users. Here's the loop.

Start from the number users feel

A crash list sorted by event count is sorted by the wrong thing. One user stuck in a crash loop can generate thousands of events, while a crash that hits a few thousand people once each barely registers. So I rank by users affected, and I look at two numbers before anything else:

  • Crash-free users over a rolling window. It's the number leadership understands and the one that tracks how the app feels.
  • User-perceived crash and ANR rates in Android vitals. Google Play uses these to decide how visible your app is, and it flags an app as having bad behavior above 1.09% for crashes and 0.47% for ANRs. Those thresholds are a floor, not a goal.

Then I slice by device, OS version and app version. A crash that only exists on one chipset or one Android release is a very different problem from one that's everywhere, and the slice usually tells you which.

Same week, two ways to sortevents lie, users don't
By event count
NullPointerException in Sync48k events
3 users in a crash loop
IllegalStateException in Fragment2.1k events
By users affected
ANR: disk read on main thread9.4k users
IllegalStateException in Fragment1.8k users
NullPointerException in Sync3 users
Illustrative numbers. The top of the list changes completely once you count people instead of events.

Group by cause, not by stack trace

Crash reporters group by stack trace. That's a reasonable default and a poor model of reality. Ten reports with ten different top frames often come from one bad assumption: a Fragment touched after onDestroyView, a callback that outlives its screen, a list mutated from two threads. Fix the ten traces one at a time and you'll be fixing the eleventh next sprint.

So before writing any fix I spend time grouping. I read the breadcrumbs and the last few screens before each crash, and I ask one question of every report: what did the code assume that wasn't true here? When three or four reports give the same answer, that's one cause with one fix.

Ten traces, one assumption"the view is still there"
ProfileFragmentNPE on binding
SearchFragmentIllegalState
Adapter callbackview detached
Dialog dismissnot attached
↓ same root cause ↓
UI work scheduled from a coroutine that outlived the viewFix once: collect with repeatOnLifecycle, scope work to viewLifecycleOwner, add a lint rule
Grouping is the step most teams skip, and it's where most of the leverage is.

The ANRs you can't reproduce

The hard ones are the ANRs that never happen on my own phone. They live on slower devices, under memory pressure, with a full disk or a busy binder. Guessing doesn't work here. Evidence does, and modern Android gives you a lot of it.

Since Android 11, ApplicationExitInfo tells you why your process died last time, including the main thread's stack at the moment of an ANR. I read it on the next launch and send it with the breadcrumbs, which gives me real traces from real devices instead of a guess:

KExitReasonReporter.ktcore/diagnostics/src/main/kotlin
class ExitReasonReporter(    private val context: Context,    private val reporter: DiagnosticsReporter,) {    fun reportPreviousExits() {        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.R) return        val am = context.getSystemService(ActivityManager::class.java)        am.getHistoricalProcessExitReasons(context.packageName, 0, 5)            .filter { it.reason == ApplicationExitInfo.REASON_ANR }            .forEach { info ->                // For ANRs, the trace holds every thread's stack at the moment it froze                val trace = info.traceInputStream?.bufferedReader()?.use { it.readText() }                reporter.anr(                    timestamp = info.timestamp,                    importance = info.importance,                    trace = trace,                )            }    }}

Once you can read the traces, the usual suspects show up quickly:

  • Disk on the main thread. SharedPreferences commits, a database opened eagerly, a file read during startup.
  • Lock contention. The main thread waiting on a lock a background thread holds while it does I/O.
  • Slow binder calls. Asking a system service for something on the main thread, on a device where that service is busy.
  • Work started at the wrong point in the lifecycle. Heavy work in onCreate or in a BroadcastReceiver, which has a much shorter budget than people assume.

In debug builds, StrictMode turns most of these into immediate, loud failures, so they're caught before they ship:

KDebugApp.ktapp/src/debug/kotlin
class DebugApp : App() {    override fun onCreate() {        StrictMode.setThreadPolicy(            StrictMode.ThreadPolicy.Builder()                .detectDiskReads()                .detectDiskWrites()                .detectNetwork()                .penaltyLog()                .penaltyDeathOnNetwork()                .build()        )        super.onCreate()    }}

Fix the class, then lock the door

When I find a cause, I fix the whole class of bug, not the one report. If one screen did disk I/O on the main thread, I look for every place that pattern exists. Then I add something that fails the build if it comes back: a custom lint check, an architecture test, or a unit test that replays the exact sequence that broke.

That last step is the one that changes the trend line. A fix without a check is a fix that holds until the next refactor. A fix with a check is permanent, and it teaches the next engineer, or the next AI assistant, what not to do.

KLifecycleCollectionTest.ktcore/ui/src/test/kotlin
class LifecycleCollectionTest {    @get:Rule    val mainDispatcherRule = MainDispatcherRule() // swaps Dispatchers.Main for a test dispatcher    @Test    fun `updates stop reaching the screen once it is stopped`() = runTest {        val owner = TestLifecycleOwner(Lifecycle.State.STARTED)        val states = MutableSharedFlow<ProfileState>()        val rendered = mutableListOf<ProfileState>()        backgroundScope.launch {            owner.repeatOnLifecycle(Lifecycle.State.STARTED) {                states.collect { rendered += it }            }        }        runCurrent()        states.emit(ProfileState.Loading)        owner.handleLifecycleEvent(Lifecycle.Event.ON_STOP) // the app goes to the background        runCurrent()        states.emit(ProfileState.Loaded(sampleProfile))        runCurrent()        assertThat(rendered).containsExactly(ProfileState.Loading)    }}

Ship the fix in stages

Stability fixes are code changes, and code changes cause regressions. So every release goes out in stages with crash and ANR rates watched at each step, and with a clear rule written down in advance for when to halt. If a regression gets through, it stops at a small percentage of users instead of reaching all of them.

Staged rollouthalt if crash-free users or ANR rate moves past the line
1%smoke test
10%watch a day
50%compare cohorts
100%keep watching
a regression found at 1% is a fix; at 100% it's an incident
Percentages are an example. The point is that the halt rule exists before the release does.

What I keep on the dashboard

  • Crash-free users and sessions, per app version, so a new release is compared with the one it replaced.
  • User-perceived ANR rate, sliced by device tier. Low-end devices are where ANRs live.
  • Top causes by users affected, after grouping, with an owner next to each.
  • Exit reasons from ApplicationExitInfo, including low-memory kills, which never show up as crashes but feel exactly like one to the user.

None of this is glamorous. It's a habit: measure what users feel, find the cause behind the reports, fix it once, and leave a check behind so it stays fixed.