OnDevice AI Meter: Test methodology
Version 3 of the public methodology (updated 2026-11-15 for suite version 4, app 1.0.4). The reference values are still those of suite version 3 (app 0.3.9, 2026-11-12).
OnDevice AI Meter measures a phone's on-device AI performance so that results can be checked and repeated. This document describes what is measured, how, and what the results do and do not show.
Overview
One run times fixed AI models on every processor the phone offers, checks that each processor's answers are correct, and turns the best correct times into one AI Capability Score and a level.
- Tests, in groups: Recognition AI = photo recognition, object detection and sound recognition. Generative AI = image style, speech to text and a chat model (LLM). Extra = the efficiency test (not scored).
- Optional parts: the chat model and speech to text run only if you downloaded them; the efficiency test can be switched off.
- Processors: CPU, GPU and NPU, each measured separately. A processor counts only if its answers match the CPU's.
- Also measured, not scored: the 2-minute efficiency test (how much CPU work an accelerator takes off the CPU) and the phone's state (battery, temperature, charging).
- Not scored: Google-managed AI features such as Gemini Nano. Google controls their versions, so they are shown beside the score, not in it.
- The tests run on the phone, offline. Test results stay on it. Released versions send anonymous usage events and crash reports (Firebase) and show ads (Google AdMob) outside the tests; see the privacy policy.
Tests and models
Each test runs one to three models chosen in advance, using open licences and LiteRT-compatible files. Every phone runs the same set, so scores stay comparable.
| Test | Models (float / INT8 unless noted) | Licence | Source |
|---|---|---|---|
| Image classification | MobileNet V2 1.0 224 | Apache 2.0 | TensorFlow hosted models |
| Image classification | EfficientNet-Lite0 | Apache 2.0 | TensorFlow on Kaggle Models |
| Object detection | EfficientDet-Lite0 | Apache 2.0 | Google MediaPipe models |
| Chat model (LLM) | Qwen3 1.7B, INT4 (optional download) | Apache 2.0 | litert-community on Hugging Face |
| Sound recognition | YAMNet (front end on a PC; network float / INT8) | Apache 2.0 | Google MediaPipe models |
| Speech to text | Whisper tiny, float / INT8 (optional download) | Apache 2.0 | litert-community on Hugging Face |
| Image style | Magenta arbitrary image stylization, float16 / mixed INT8 | Apache 2.0 | TensorFlow Lite files from Magenta |
Exact files are pinned by SHA-256 in the app's source repository. The chat model is never bundled with the app.
Model choice: reliability first
Models are chosen for how reliably they finish and give correct answers on every reference phone, not for how impressive they are.
- A new candidate model first runs as a trial: it is measured and reported, but not scored, until it finishes with matching answers on every reference device.
- A model that keeps failing on some phones or processors is replaced rather than kept with exceptions. A failure on one processor alone (for example a compile time-out) falls back to the CPU and is fixed or dropped before the suite version is frozen.
- Over time the suite narrows to the models with the best completion record.
Which phones are covered
The app aims to cover as many phones as possible. A phone is supported when its chipset and Android version are supported by the app and it completes more than 60% of the tests. A test counts as finished when it gives a time and correct answers, on the CPU at least. Each run reports its own completion rate: the share of the tests that were run which finished, and whether that exceeds 60%. Tests you left out (not downloaded, efficiency test switched off) are not counted. A skip for low memory or a time-out counts as not finished. A run that stops early because the phone is too hot (see below) counts neither for nor against the phone.
- Minimum Android version: set by the system features the tests need, kept as low as possible to reach more phones.
- Phone problems are shown, not hidden. When one phone of an otherwise working model fails because of the phone itself (a faulty driver, firmware or an overheating unit), the app adds no special workaround for it. It states what failed, why, and what still counts.
- Phones that complete 60% or less are identified through the app's run reports and users' in-app problem reports. Before such a phone model is removed from the store's list of supported devices, a problem in the app is ruled out first.
Test inputs
All inputs ship with the app and are identical on every phone.
- Classification photos: 14 public-domain or CC0 photos from Wikimedia Commons, centre-cropped to 224×224 lossless PNG. They were chosen from 41 candidates because both MobileNet versions recognise them with a clear margin.
- Detection photos: 9 of those photos at 320×320, each showing a clear everyday object.
- Chat model: a 256-token prompt for speed runs, plus three quiz questions with one correct answer each: the days of the week in order (since app 1.0.8; before, the planets in order from the Sun, which the small chat model often got wrong on every phone), the capital of France, and 17 + 25.
- Why not random noise: on random pixels every class scores almost the same, so tiny rounding differences change the answer and correct processors look wrong. Real photos have a clear answer.
Photo sources and licences are listed in the app's source repository.
Processors
Each model runs separately on every processor the phone offers, through Google's LiteRT runtime (LiteRT-LM for the chat model), with fixed settings so every phone runs the same way.
| Processor | Route | Fixed settings | Skipped when |
|---|---|---|---|
| CPU | LiteRT (XNNPACK) | one thread per fast core (every core outside the slowest cluster, by top frequency), at least 4 and at most 8 (suite 4; 4 on every phone before) | never; it is the reference for answers |
| GPU | LiteRT GPU (OpenCL / OpenGL ES) | FP16 precision, default priority; if the answer check fails, the model runs once more at FP32 and counts only if it then matches ("Full precision (FP32)", since app 0.1.32) | the phone offers no GPU; INT8 models (INT8 on GPUs gave wrong answers); the chat model when the GPU failed a vision answer check |
| NPU, Snapdragon | LiteRT with the vendor's Qualcomm libraries | "Burst" mode for the main tests, "sustained high" for the NPU half of the efficiency test | the chip is not supported |
| NPU via NNAPI | Android's Neural Networks API through LiteRT's C API in a small native bridge | — | the phone has no vendor NNAPI driver (the framework's reference driver is CPU-only); the driver cannot compile the model |
| NPU, chat model | needs a model file compiled for each chip | — | not yet supported |
A processor is measured only if the phone reports it. LiteRT and NNAPI can quietly run work on the CPU instead. So any GPU, NPU or NNAPI result that is not at least 3% faster than the CPU is flagged "no speed-up" and never shown as a working accelerator. On Google Tensor phones, LiteRT's NPU route is skipped because Google limits that route to approved apps; the TPU is measured through NNAPI instead.
AI processor access differs by platform. Each chip maker decides how apps reach its NPU, and the app measures only what an ordinary app can reach: Snapdragon through Qualcomm's QNN libraries bundled with the app (most recent chips); Google Tensor through NNAPI (Google limits its direct LiteRT route to approved apps); Samsung Exynos through NNAPI where a vendor driver exists (about 2x the CPU on the Exynos 2400, no gain on the Exynos 1480, no driver on the 1330); MediaTek through NNAPI drivers that gave no gain on the phones tried (MediaTek's own route is not open to the app yet); entry-level chips usually have no NPU an app can reach. Large gaps between phones in the rankings therefore reflect both the chip and how open its NPU is to apps. See also the FAQ ("Why AI processors differ between phones") and in the app's result details.
Timing
The reported time is the median of many timed runs on one fixed photo, after a warm-up.
- Load: the model is loaded and compiled for the processor. Load time is recorded separately and not scored.
- Warm-up: at least 10 runs and at least 1 s (CPU), 2 s (GPU) or 0.5 s (NPU), because processor clocks ramp up in the first second.
- Timed rounds: 5 rounds, each at least 50 runs and at least 1 s long. Every run includes reading back one output, because GPUs and NPUs can return before their work is done. Only the smallest output is read, so copying large outputs is not timed.
- Result: the median of all timed runs, plus the 90th percentile.
- Spread: how much the round medians differ, as a share of their median, leaving out the single fastest and slowest round. The full range is kept too. The target is 5% or less.
- Steadiness: a result whose spread is over 10% is labelled unsteady on this phone. Unsteady is not a failure: the score uses the median, which holds up when single rounds jump, and the completion rate ignores steadiness. Only wrong answers or a test that does not finish count as failures. For the chat model, the spread of answer speed across its three runs is used. Since app 0.3.10 a set of rounds that comes out unsteady is timed once more with the model still loaded, and the steadier set counts; if the lane that decides a test's score is still unsteady, that test's score is shown as approximate (≈).
Why results vary. CPUs are usually steady. GPUs vary most on short tests, because phones lower GPU clocks in the short gaps between photos and must raise them again. Some GPU drivers are unreliable and can swing widely or give wrong answers. Heat matters in long runs; the efficiency test records the thermal state, but the score does not depend on it. The app does not tune itself per phone, so every phone is measured the same way.
Why runs are not simply made longer. Each round already lasts at least 1 s and 50 runs. Longer rounds average more photos but do not remove the gaps between photos, which is where GPU clocks drop. They also heat the phone, which brings throttling in, and make users wait longer. Running photos back to back without reading each answer would hide the gap, but it would measure throughput rather than the time to answer one photo.
Chat model: LiteRT-LM's own benchmark reads a 256-token prompt and writes 64 tokens, three times per processor. The medians of prompt speed and answer speed (tokens per second), time to first token and load time are reported.
Every line of a run is also written to a log file on the phone, and the full report is saved as JSON.
Answer checks
A fast processor with wrong answers counts for nothing. After timing, each processor answers every test input once (untimed), and its answers are compared with the CPU running the same model.
| Test | A processor's results match the CPU when… |
|---|---|
| Image classification | its top answer equals the CPU's on at least 90% of the 14 photos, and is always among the CPU's top 5 |
| Object detection | its best class equals the CPU's on at least 90% of the 9 photos, with the score within 0.15 |
| Chat model | it answers at least as many of the 3 quiz questions correctly as the CPU (greedy decoding, model "thinking" off) |
Wrong quiz answers. The quiz checks that a processor runs the model correctly, not how capable the model is. A small, 4-bit compressed model can get a question wrong on every processor. That is a limit of the model, so the CPU's own result is the yardstick and quiz answers are never scored on their own. A processor fails the check only when it gets fewer right than the CPU, which points to number-precision differences (GPUs often compute at half precision) or a driver problem.
The CPU itself is checked against reference answers computed on a PC with the same model files. This shows that photos are decoded identically on the phone.
Two kinds of check (named since app 0.3.10). Execution consistency: each GPU, NPU or NNAPI result is compared with the CPU running the same model, which shows that the processor ran the model correctly. Model output check: the CPU's answers are compared with reference answers from a PC (photos, objects, sounds, image style and speech), and the chat model answers a short quiz. Neither is a measure of how accurate a model is at its task: the score never rewards or penalises a model's accuracy, only the speed of processors whose answers are consistent.
A processor that fails shows "Results don't match" and is left out of the score.
Sound, speech and image style tests
Sound recognition. A sound classifier (YAMNet) listens to seven short public-domain clips (applause, speech, a car horn, a church bell, a crying baby, a siren and a thunderstorm) on every processor, with the same timing rules and answer check as the photo tests. It is scored and part of Recognition AI.
Speech to text. A small speech recognizer (Whisper tiny, downloaded on demand like the chat model) transcribes four short public-domain recordings on each processor. Two things are timed: the encoder, and how fast the text is written (word pieces per second). Each processor's transcripts must match the CPU's within 10 % (counted in word pieces), and a graphics processor that misses at half precision is retried at full precision. It is scored and part of Generative AI. While a processor writes its transcripts, the screen shows the words (at most about five redraws a second), and if you turn the speaker on, the next processor pass plays each recording and the test waits for it to finish before the next one (only the model's own work is timed, so results don't change). The sound test plays its clips the same way. The encoder is timed with the screen still.
Image style. A style transfer model (Magenta arbitrary image stylization, Apache 2.0) repaints four public-domain photos in the style of two public-domain paintings (The Starry Night by Van Gogh, The Great Wave by Hokusai), eight pictures in all. The style of each painting is prepared on a PC; the phone times the repainting network at 384x384 pixels with the same warm-up and rounds as the photo tests. Answers are compared by structural similarity (SSIM, 1 = identical, on grey copies): each processor must be at least 0.95 against the CPU on every picture, and the CPU at least 0.90 against the PC's reference. A graphics processor that misses at half precision is retried at full precision, and an accelerator that is not 3% faster than the CPU is flagged "no speed-up". It was a trial from app 0.1.68 and is scored since suite version 3 (app 0.3.9), part of Generative AI, after it finished with matching pictures on every reference phone.
The speech score combines the encoder time and the text-writing speed.
Efficiency test and pre-run checks
Efficiency test. On by default (a checkbox on the start screen; it can be unticked) and not scored. Last in every run. It asks how much CPU work an accelerator takes off the CPU. MobileNet V2 runs at a fixed 30 photos per second, like a camera, for 1 minute on the CPU and then 1 minute on the fastest accelerator whose answers matched the CPU and that was faster than the CPU. That is the AI processor (NPU, through Qualcomm libraries or through NNAPI with INT8) or the graphics processor (GPU, float model). Both halves use the same model and the same work, so they compare like for like. The CPU half runs first, so the accelerator half starts on a slightly warmer phone; this favours the CPU, never the accelerator. Per half, the app measures its own CPU time per photo with Android's public Process.getElapsedCpuTime, minus the app's idle CPU time measured just before (5 s, screen held still). Work done on the accelerator itself is not CPU time, which is the point. It also measures the time per photo in the 2nd and in the last 10-second window ("slowed down" if the last is more than 10% slower), and records Android's live thermal headroom and the battery temperature at the start and end of each half. CPU work saved = 1 - (accelerator CPU time per photo / CPU-lane CPU time per photo). Example: "Efficiency: the AI processor needs 92% less CPU work than the CPU for the same photos. It kept its speed under the steady load." If no accelerator gave matching answers, the result says that nothing takes work off the CPU. The test reports no energy or power figures and uses no non-public or vendor-specific APIs. It is shown beside the score, never inside it.
Pre-run checks. Before a run the app records battery level, battery temperature, thermal state and power source, and warns about:
- battery under 20% while not charging (some phones slow down)
- battery over 40 °C (let the phone cool first)
- the phone already limiting its own speed because of heat
- charging (recorded, not a failure; results are compared only within the same power state)
Heat safety. Before a run the app shows a warning, until you tick "Don't show this again". It also says that running the tests is your choice and that the author can't be responsible for damage to your phone. During a run it checks between measurements (never inside a timed loop) how hot the phone is. If Android reports a "severe" heat state, or the battery is above 45 °C, the run pauses between tests for up to 3 minutes to let the phone cool, then goes on; tests that start while the phone is still that hot are marked "ran hot", and the run keeps its level. If Android reports a "critical" heat state or worse, or the battery is above 48 °C, the run stops by itself and keeps the results so far. Such a run counts neither for nor against the phone's support, and its score (from the tests that finished) gets no level. You can also stop a run yourself with the Stop button. Phones also protect themselves by slowing down. (Since app 1.0.5; before, the run stopped at "severe" or 45 °C.)
On phones with less than 8 GB of memory, the chat model also needs free memory of at least 1.2× its file size, or it is skipped (larger phones always try, since Android counts cached apps as used memory), and each processor gets at most 15 minutes for it.
Scoring and levels
Every result shows the numeric AI Capability Score, for comparing, and one of five levels, for a quick read.
- Reference phone: a reference phone (a recent flagship) on battery scores 1000 in every test, so 3000 per half and 6000 in total, on suite version 3. Every reference value is the median of the same three full runs on battery with the released app (2026-11-12), so each test is measured as you run it. Suite version 2 (app 0.1.66-0.3.8) took them from separate part runs on a cooler phone, so its scores are not comparable.
- Test score: for time-based tests, reference time ÷ your time × 1000 (twice as fast = 2000). For the chat model, your speed ÷ reference speed × 1000, averaging prompt and answer speed. Speech to text combines the encoder time and the text-writing speed (geometric mean), from the matching processor with the best combined result. Each test uses its best processor whose answers matched.
- Two halves: the Recognition score is the sum of the three recognition test scores (photos, detection, sound). The Generative score is the sum of image style, speech to text and chat.
- AI Capability Score: the sum of the two halves (since app 0.3.18). Before, each half and the total were geometric means and the reference phone scored 1000; the app recomputes those earlier results as sums from their stored test scores, so your test history stays comparable. Before 0.1.66 the tests were weighed differently, so those scores are not comparable.
In short, where each name stands for that test's score:
- Recognition = photos + detection + sound
- Generative = image style + speech to text + chat
- AI Capability Score = Recognition + Generative
Edge cases:
- If a test cannot run (not enough memory, time-out), that test gets the floor score of 1 and the level is capped at the lowest two.
- Downloads are optional. If you did not download the chat model or speech to text, that test does not run and the result is marked "partial": a score from the tests that ran (shown only when both halves have at least one test), but no level. A level needs every scored test. Tests you left out are not counted in the completion rate.
- A processor marked "Results don't match" never counts. If no processor matches, that test gets the floor score of 1.
- The efficiency test and Google-managed AI features are shown beside the score, not in it.
Levels: Basic (under 1200), Capable (1200-2399), Advanced (2400-4799), Flagship (4800-9599) and Cutting-edge (9600 or more): each level starts at twice the score of the one below, and the reference phone (6000) is Flagship. A capped result is at most Capable; partial results (a scored test left out) get no level.
Fairness rules and versioning
- Same inputs everywhere: models, photos and prompts are fixed and pinned by SHA-256. Scores are compared only within the same suite version. Since app 0.3.11 every report also records one SHA-256 fingerprint over all bundled models and test inputs, computed on the phone from the installed app, and whether it equals the suite's pinned value; the downloaded chat and speech models are checked against their own pinned SHA-256 before use.
- Nothing else runs during a test: no ads load or refresh and no network work happens until the run ends.
- Same power state: results are compared only with runs in the same power state (on battery or charging). Since app 0.3.10 a score measured while charging (at the start or end of the run) is marked "Charging" next to the score, because the reference phone is measured on battery.
- What the score measures: the AI speed that ordinary apps can reach on the phone, not the chip's peak. Some AI processors are closed to apps or reached only through Android's older NNAPI route, so a phone can score below what its chip could do with the maker's own tools. A processor that apps cannot use is reported as not available, never estimated. NNAPI is deprecated: if a phone update removes its driver, that phone's AI processor drops out of the score, and the app shows this rather than hiding it.
- Honest fallbacks: a processor the phone does not offer is skipped, never scored. Suspected CPU fallbacks are flagged.
- Model downloads: the optional chat and speech models are downloaded once, ahead of the run, and checked against its pinned SHA-256 before use.
- New suite version: any change to models, inputs, settings, runtime versions (LiteRT, LiteRT-LM, Qualcomm QNN) or the formula raises the suite version, and the reference phone is measured again. A formula change that earlier results can be recomputed with, from their stored test scores, keeps the suite (the summed score of app 0.3.18). Suite 3 runs on LiteRT 2.3.2, LiteRT-LM 0.18.0 and QNN 2.51.0; each score records its suite and runtimes.
- Suite 4 (app 1.0.4): the CPU uses one thread per fast core, 4 to 8, instead of 4 on every phone, so phones with more fast cores are not held back. Each report records the cores, the fast cores and the threads used, and the run screen shows them ("CPU · 6 cores used"). The reference phone has 4 fast cores, so it runs exactly as in suite 3 and its reference values stand. Suite 3 and suite 4 scores are not compared.