Inside LiteRT: a case study in on-device pose tracking
CoreCues is a side project: an app that watches you exercise through the front camera and corrects your form out loud, with a coach called Arc. I gave it three rules. The intelligence runs on the phone, with no server behind it. Android and iPhone share one core. And I should be able to say what every megabyte in the download is for.
The third rule is the one that taught me the most. It forced me to stop treating the ML runtime as a black box and learn what LiteRT actually does when it runs a model. This is that story, and the numbers that came out of it.
Version one, and the first real set
The starting point was the easiest stack available: Google's MediaPipe Pose Landmarker for tracking, a rep counter built on a single knee angle, and a set of unit tests that all passed. Then I put the phone on the floor and did a set of squats. It counted about twice as many reps as I did, and told me to go deeper on reps where I was already well below parallel.
The cause was one landmark. For a few frames at the bottom of a squat, the knee point would slip, and a slipping knee looks exactly like standing up. Tests built on clean, synthetic angles never see that. The fix was to stop trusting any single signal: a rep now counts only when the knee angle and the hips' travel agree, and a fault is only spoken once it repeats.
// A rep only counts when two independent signals agree.// One landmark slipping for a few frames can fool the knee angle,// but it can't also move the hips.val kneeAtDepth = smoothedKneeAngle < DEPTH_DEGREESval hipsAtDepth = hipDrop > MIN_HIP_TRAVEL * thighLengthif (phase == Phase.DESCENDING && kneeAtDepth && hipsAtDepth) { phase = Phase.BOTTOM}That set changed how I think about pose models. A landmark is a noisy estimate with a confidence score, not a fact. Everything downstream has to be built on that assumption.
What was actually in the app
Once the counting worked, I unzipped the release build. It was 29.0 MB. One file, libmediapipe_tasks_jni.so, was 10.5 MB. The two pose model bundles were another 15.2 MB. Close to 90% of my app was machine learning plumbing I hadn't written and didn't understand.
That 10.5 MB library is two things welded together: the pose-specific work that happens around the models, and a private copy of the runtime that executes them. Every app built on MediaPipe carries its own copy of that runtime. To do better, I had to understand both halves.
How LiteRT runs a model
LiteRT is the current name for TensorFlow Lite. Underneath the friendly APIs, running a model comes down to a few steps, and each one turned out to matter for a camera app:
- The model is a graph, not code. A
.tflitefile is a flatbuffer describing operations and their weights. The runtime walks that graph. Nothing in it is specific to Android or iOS, which is why the same file runs on both. - An accelerator compiles it for the hardware. On the CPU, LiteRT runs the operations through XNNPACK. On the GPU, it compiles the graph into shaders. That compile is expensive and happens on the first run, so a model that hasn't been warmed up will stall the first live frame.
- Tensors are just buffers with a shape. The pose landmark model wants a 256 × 256 RGB image as floats. If you allocate that buffer per frame, garbage collection and heat catch up with you in a long workout. You create the input and output buffers once and write into them.
- Not every operation runs everywhere. If the GPU can't run part of a graph, work falls back to the CPU. A silent fallback is a silent slowdown, so you want to know where each model actually landed.
Two models, and the glue between them
The other half of MediaPipe's library was the part I had never thought about. Pose tracking isn't one model. It's two, with real work between them:
MediaPipe does all of that in C++ and you never see it. That's a good default. It also meant the runtime and the glue were sold together.
Moving to LiteRT in Google Play services
LiteRT in Google Play services moves the runtime out of your app. It lives in Play services, shared by every app on the phone, and your APK carries only the models and a thin client. It also adds GPU inference through the newer CompiledModel API.
My models were already .tflite files, so it looked like a dependency swap. It wasn't. MediaPipe always uses the runtime compiled into its own library, with no setting to point it at Play services. To use the shared runtime, I had to stop using MediaPipe and own the glue myself. That became :core-pose, about 800 lines of Kotlin, which I checked frame by frame against MediaPipe's own output with a small Python reference before trusting it.
The LiteRT side is short. This is most of it:
val environment = LiteRt.initialize( context, LiteRtInitializationOptions.builder().setEnableGpu(true).build(),).await()val model = CompiledModel.create( context.assets, "pose_landmarks_full.tflite", CompiledModel.Options(Accelerator.GPU), environment,)val inputs = model.createInputBuffers()val outputs = model.createOutputBuffers()// The first run compiles the GPU shaders. Do it now, not on a live frame.inputs[0].writeFloat(FloatArray(256 * 256 * 3))model.run(inputs, outputs)The docs show passing GPU and CPU together so LiteRT falls back on its own. I try them one at a time instead, so a stats chip on the camera screen can say which accelerator each model actually landed on. During testing, that's exactly what you want to see.
Proving it on one phone
I didn't want to find out later that my version was worse. So before deleting anything, I split the build into two product flavors, one per pose backend:
flavorDimensions += "pose"productFlavors { create("litert") { dimension = "pose" applicationIdSuffix = ".litert" versionNameSuffix = "-litert" isDefault = true } create("mediapipe") { dimension = "pose" }}dependencies { "mediapipeImplementation"(project(":android:pose-mediapipe")) "litertImplementation"(project(":android:pose-litert"))}The applicationIdSuffix is what made this useful. Both builds install side by side, so I could do the same set with each, on the same phone, in the same light. Nothing else in the app knows there are two backends. CI builds both release APKs on every push and prints a size breakdown for each.
Where the 12.5 MB came from
- About three quarters is the runtime. The MediaPipe library is gone. What replaces it is three small client libraries, 1.4 MB in total, that talk to the runtime in Play services.
- 3 MB came from understanding the model files. MediaPipe ships each model tier as a sealed bundle, and the full and lite bundles each contain the same person detector with identical weights. LiteRT loads plain
.tflitefiles, so a small Gradle task unzips the bundles at build time and ships the detector once. - The last 0.4 MB was MediaPipe's Java classes and the protobuf code they depend on, which R8 could finally remove.
The frame path got leaner too. MediaPipe needed a Bitmap and a rotated copy for every frame. Now the camera bytes are copied once into a buffer that's reused for the whole workout.
What it cost
Size is the easy win to report, so here is the other side.
- I own the pipeline. The 800 lines of glue are mine to maintain, test and keep in step with new model versions.
- Startup is asynchronous. The first initialization can take a while if Play services has to fetch the runtime, and compiling for the GPU adds more. The app says so on screen while it happens.
- It needs Google Play services. On phones without it, the fallback is the standalone LiteRT library, which puts a runtime back in the APK.
- The speed comparison isn't clean. The MediaPipe flavor ran on the CPU and the LiteRT flavor runs on the GPU, with a different pipeline around it. I can attribute every megabyte. I can't do the same for every millisecond.
- Privacy copy had to change. Play services reports runtime usage and performance metrics to Google. That's documented, and I rewrote the privacy policy to say so plainly.
iOS got the pipeline for free
Because the glue was now plain Kotlin, it moved into the shared Kotlin Multiplatform core. The iPhone version runs the same :core-pose pipeline and the same .tflite files on LiteRT for iOS, accelerated with Metal. There's no Play services on an iPhone, so there the runtime ships inside the app. Only the camera and the voice are platform code. The thresholds, the state machine and the rep counter are identical on both phones.
Tests from real workouts
The double-counting bug taught me that synthetic tests aren't enough. During development the app could record landmarks, never video, and those recordings became fixtures. CI replays them on every push and fails if a rep count or a form fault changes:
@Testfun `recorded squat sets keep their rep counts`() { for (session in recordedSessions("squat")) { val result = FormEngine().replay(session.frames) assertEquals(session.expectedReps, result.reps, session.name) }}That lets me tune a smoothing constant or a threshold and know in seconds whether I broke anything, without doing another set of squats in front of a camera.
What I took away
- Know what's in your download. Unzipping the APK was the most useful ten minutes of the project.
- The model is the small part. Pre-processing, decoding, cropping and tracking are where pose tracking lives, and they're ordinary code you can own and test.
- Warm up, reuse, and check the accelerator. Compile before the first live frame, allocate buffers once, and make fallbacks visible.
- Measure both sides before you delete one. Two flavors on one phone turned an argument into a table.
Links
- LiteRT in Google Play services: overview, limitations and the data the runtime reports
- CompiledModel API for Play services: setup, GPU and migration
- LiteRT overview: platforms and accelerators, including Metal on iOS
- Smaller apps and faster GPU inference with LiteRT in Google Play services: the announcement