← Simplicity.IO

OnDevice AI Meter: Test methodology

Version 3 of the public methodology (updated 2026-11-15 for suite version 4, app 1.0.4). The reference values are still those of suite version 3 (app 0.3.9, 2026-11-12).

OnDevice AI Meter measures a phone's on-device AI performance so that results can be checked and repeated. This document describes what is measured, how, and what the results do and do not show.

Overview

One run times fixed AI models on every processor the phone offers, checks that each processor's answers are correct, and turns the best correct times into one AI Capability Score and a level.

Tests and models

Each test runs one to three models chosen in advance, using open licences and LiteRT-compatible files. Every phone runs the same set, so scores stay comparable.

TestModels (float / INT8 unless noted)LicenceSource
Image classificationMobileNet V2 1.0 224Apache 2.0TensorFlow hosted models
Image classificationEfficientNet-Lite0Apache 2.0TensorFlow on Kaggle Models
Object detectionEfficientDet-Lite0Apache 2.0Google MediaPipe models
Chat model (LLM)Qwen3 1.7B, INT4 (optional download)Apache 2.0litert-community on Hugging Face
Sound recognitionYAMNet (front end on a PC; network float / INT8)Apache 2.0Google MediaPipe models
Speech to textWhisper tiny, float / INT8 (optional download)Apache 2.0litert-community on Hugging Face
Image styleMagenta arbitrary image stylization, float16 / mixed INT8Apache 2.0TensorFlow Lite files from Magenta

Exact files are pinned by SHA-256 in the app's source repository. The chat model is never bundled with the app.

Model choice: reliability first

Models are chosen for how reliably they finish and give correct answers on every reference phone, not for how impressive they are.

Which phones are covered

The app aims to cover as many phones as possible. A phone is supported when its chipset and Android version are supported by the app and it completes more than 60% of the tests. A test counts as finished when it gives a time and correct answers, on the CPU at least. Each run reports its own completion rate: the share of the tests that were run which finished, and whether that exceeds 60%. Tests you left out (not downloaded, efficiency test switched off) are not counted. A skip for low memory or a time-out counts as not finished. A run that stops early because the phone is too hot (see below) counts neither for nor against the phone.

Test inputs

All inputs ship with the app and are identical on every phone.

Photo sources and licences are listed in the app's source repository.

Processors

Each model runs separately on every processor the phone offers, through Google's LiteRT runtime (LiteRT-LM for the chat model), with fixed settings so every phone runs the same way.

ProcessorRouteFixed settingsSkipped when
CPULiteRT (XNNPACK)one thread per fast core (every core outside the slowest cluster, by top frequency), at least 4 and at most 8 (suite 4; 4 on every phone before)never; it is the reference for answers
GPULiteRT GPU (OpenCL / OpenGL ES)FP16 precision, default priority; if the answer check fails, the model runs once more at FP32 and counts only if it then matches ("Full precision (FP32)", since app 0.1.32)the phone offers no GPU; INT8 models (INT8 on GPUs gave wrong answers); the chat model when the GPU failed a vision answer check
NPU, SnapdragonLiteRT with the vendor's Qualcomm libraries"Burst" mode for the main tests, "sustained high" for the NPU half of the efficiency testthe chip is not supported
NPU via NNAPIAndroid's Neural Networks API through LiteRT's C API in a small native bridge—the phone has no vendor NNAPI driver (the framework's reference driver is CPU-only); the driver cannot compile the model
NPU, chat modelneeds a model file compiled for each chip—not yet supported

A processor is measured only if the phone reports it. LiteRT and NNAPI can quietly run work on the CPU instead. So any GPU, NPU or NNAPI result that is not at least 3% faster than the CPU is flagged "no speed-up" and never shown as a working accelerator. On Google Tensor phones, LiteRT's NPU route is skipped because Google limits that route to approved apps; the TPU is measured through NNAPI instead.

AI processor access differs by platform. Each chip maker decides how apps reach its NPU, and the app measures only what an ordinary app can reach: Snapdragon through Qualcomm's QNN libraries bundled with the app (most recent chips); Google Tensor through NNAPI (Google limits its direct LiteRT route to approved apps); Samsung Exynos through NNAPI where a vendor driver exists (about 2x the CPU on the Exynos 2400, no gain on the Exynos 1480, no driver on the 1330); MediaTek through NNAPI drivers that gave no gain on the phones tried (MediaTek's own route is not open to the app yet); entry-level chips usually have no NPU an app can reach. Large gaps between phones in the rankings therefore reflect both the chip and how open its NPU is to apps. See also the FAQ ("Why AI processors differ between phones") and in the app's result details.

Timing

The reported time is the median of many timed runs on one fixed photo, after a warm-up.

  1. Load: the model is loaded and compiled for the processor. Load time is recorded separately and not scored.
  2. Warm-up: at least 10 runs and at least 1 s (CPU), 2 s (GPU) or 0.5 s (NPU), because processor clocks ramp up in the first second.
  3. Timed rounds: 5 rounds, each at least 50 runs and at least 1 s long. Every run includes reading back one output, because GPUs and NPUs can return before their work is done. Only the smallest output is read, so copying large outputs is not timed.
  4. Result: the median of all timed runs, plus the 90th percentile.
  5. Spread: how much the round medians differ, as a share of their median, leaving out the single fastest and slowest round. The full range is kept too. The target is 5% or less.
  6. Steadiness: a result whose spread is over 10% is labelled unsteady on this phone. Unsteady is not a failure: the score uses the median, which holds up when single rounds jump, and the completion rate ignores steadiness. Only wrong answers or a test that does not finish count as failures. For the chat model, the spread of answer speed across its three runs is used. Since app 0.3.10 a set of rounds that comes out unsteady is timed once more with the model still loaded, and the steadier set counts; if the lane that decides a test's score is still unsteady, that test's score is shown as approximate (≈).

Why results vary. CPUs are usually steady. GPUs vary most on short tests, because phones lower GPU clocks in the short gaps between photos and must raise them again. Some GPU drivers are unreliable and can swing widely or give wrong answers. Heat matters in long runs; the efficiency test records the thermal state, but the score does not depend on it. The app does not tune itself per phone, so every phone is measured the same way.

Why runs are not simply made longer. Each round already lasts at least 1 s and 50 runs. Longer rounds average more photos but do not remove the gaps between photos, which is where GPU clocks drop. They also heat the phone, which brings throttling in, and make users wait longer. Running photos back to back without reading each answer would hide the gap, but it would measure throughput rather than the time to answer one photo.

Chat model: LiteRT-LM's own benchmark reads a 256-token prompt and writes 64 tokens, three times per processor. The medians of prompt speed and answer speed (tokens per second), time to first token and load time are reported.

Every line of a run is also written to a log file on the phone, and the full report is saved as JSON.

Answer checks

A fast processor with wrong answers counts for nothing. After timing, each processor answers every test input once (untimed), and its answers are compared with the CPU running the same model.

TestA processor's results match the CPU when…
Image classificationits top answer equals the CPU's on at least 90% of the 14 photos, and is always among the CPU's top 5
Object detectionits best class equals the CPU's on at least 90% of the 9 photos, with the score within 0.15
Chat modelit answers at least as many of the 3 quiz questions correctly as the CPU (greedy decoding, model "thinking" off)

Wrong quiz answers. The quiz checks that a processor runs the model correctly, not how capable the model is. A small, 4-bit compressed model can get a question wrong on every processor. That is a limit of the model, so the CPU's own result is the yardstick and quiz answers are never scored on their own. A processor fails the check only when it gets fewer right than the CPU, which points to number-precision differences (GPUs often compute at half precision) or a driver problem.

The CPU itself is checked against reference answers computed on a PC with the same model files. This shows that photos are decoded identically on the phone.

Two kinds of check (named since app 0.3.10). Execution consistency: each GPU, NPU or NNAPI result is compared with the CPU running the same model, which shows that the processor ran the model correctly. Model output check: the CPU's answers are compared with reference answers from a PC (photos, objects, sounds, image style and speech), and the chat model answers a short quiz. Neither is a measure of how accurate a model is at its task: the score never rewards or penalises a model's accuracy, only the speed of processors whose answers are consistent.

A processor that fails shows "Results don't match" and is left out of the score.

Sound, speech and image style tests

Sound recognition. A sound classifier (YAMNet) listens to seven short public-domain clips (applause, speech, a car horn, a church bell, a crying baby, a siren and a thunderstorm) on every processor, with the same timing rules and answer check as the photo tests. It is scored and part of Recognition AI.

Speech to text. A small speech recognizer (Whisper tiny, downloaded on demand like the chat model) transcribes four short public-domain recordings on each processor. Two things are timed: the encoder, and how fast the text is written (word pieces per second). Each processor's transcripts must match the CPU's within 10 % (counted in word pieces), and a graphics processor that misses at half precision is retried at full precision. It is scored and part of Generative AI. While a processor writes its transcripts, the screen shows the words (at most about five redraws a second), and if you turn the speaker on, the next processor pass plays each recording and the test waits for it to finish before the next one (only the model's own work is timed, so results don't change). The sound test plays its clips the same way. The encoder is timed with the screen still.

Image style. A style transfer model (Magenta arbitrary image stylization, Apache 2.0) repaints four public-domain photos in the style of two public-domain paintings (The Starry Night by Van Gogh, The Great Wave by Hokusai), eight pictures in all. The style of each painting is prepared on a PC; the phone times the repainting network at 384x384 pixels with the same warm-up and rounds as the photo tests. Answers are compared by structural similarity (SSIM, 1 = identical, on grey copies): each processor must be at least 0.95 against the CPU on every picture, and the CPU at least 0.90 against the PC's reference. A graphics processor that misses at half precision is retried at full precision, and an accelerator that is not 3% faster than the CPU is flagged "no speed-up". It was a trial from app 0.1.68 and is scored since suite version 3 (app 0.3.9), part of Generative AI, after it finished with matching pictures on every reference phone.

The speech score combines the encoder time and the text-writing speed.

Efficiency test and pre-run checks

Efficiency test. On by default (a checkbox on the start screen; it can be unticked) and not scored. Last in every run. It asks how much CPU work an accelerator takes off the CPU. MobileNet V2 runs at a fixed 30 photos per second, like a camera, for 1 minute on the CPU and then 1 minute on the fastest accelerator whose answers matched the CPU and that was faster than the CPU. That is the AI processor (NPU, through Qualcomm libraries or through NNAPI with INT8) or the graphics processor (GPU, float model). Both halves use the same model and the same work, so they compare like for like. The CPU half runs first, so the accelerator half starts on a slightly warmer phone; this favours the CPU, never the accelerator. Per half, the app measures its own CPU time per photo with Android's public Process.getElapsedCpuTime, minus the app's idle CPU time measured just before (5 s, screen held still). Work done on the accelerator itself is not CPU time, which is the point. It also measures the time per photo in the 2nd and in the last 10-second window ("slowed down" if the last is more than 10% slower), and records Android's live thermal headroom and the battery temperature at the start and end of each half. CPU work saved = 1 - (accelerator CPU time per photo / CPU-lane CPU time per photo). Example: "Efficiency: the AI processor needs 92% less CPU work than the CPU for the same photos. It kept its speed under the steady load." If no accelerator gave matching answers, the result says that nothing takes work off the CPU. The test reports no energy or power figures and uses no non-public or vendor-specific APIs. It is shown beside the score, never inside it.

Pre-run checks. Before a run the app records battery level, battery temperature, thermal state and power source, and warns about:

Heat safety. Before a run the app shows a warning, until you tick "Don't show this again". It also says that running the tests is your choice and that the author can't be responsible for damage to your phone. During a run it checks between measurements (never inside a timed loop) how hot the phone is. If Android reports a "severe" heat state, or the battery is above 45 °C, the run pauses between tests for up to 3 minutes to let the phone cool, then goes on; tests that start while the phone is still that hot are marked "ran hot", and the run keeps its level. If Android reports a "critical" heat state or worse, or the battery is above 48 °C, the run stops by itself and keeps the results so far. Such a run counts neither for nor against the phone's support, and its score (from the tests that finished) gets no level. You can also stop a run yourself with the Stop button. Phones also protect themselves by slowing down. (Since app 1.0.5; before, the run stopped at "severe" or 45 °C.)

On phones with less than 8 GB of memory, the chat model also needs free memory of at least 1.2× its file size, or it is skipped (larger phones always try, since Android counts cached apps as used memory), and each processor gets at most 15 minutes for it.

Scoring and levels

Every result shows the numeric AI Capability Score, for comparing, and one of five levels, for a quick read.

  1. Reference phone: a reference phone (a recent flagship) on battery scores 1000 in every test, so 3000 per half and 6000 in total, on suite version 3. Every reference value is the median of the same three full runs on battery with the released app (2026-11-12), so each test is measured as you run it. Suite version 2 (app 0.1.66-0.3.8) took them from separate part runs on a cooler phone, so its scores are not comparable.
  2. Test score: for time-based tests, reference time ÷ your time × 1000 (twice as fast = 2000). For the chat model, your speed ÷ reference speed × 1000, averaging prompt and answer speed. Speech to text combines the encoder time and the text-writing speed (geometric mean), from the matching processor with the best combined result. Each test uses its best processor whose answers matched.
  3. Two halves: the Recognition score is the sum of the three recognition test scores (photos, detection, sound). The Generative score is the sum of image style, speech to text and chat.
  4. AI Capability Score: the sum of the two halves (since app 0.3.18). Before, each half and the total were geometric means and the reference phone scored 1000; the app recomputes those earlier results as sums from their stored test scores, so your test history stays comparable. Before 0.1.66 the tests were weighed differently, so those scores are not comparable.

In short, where each name stands for that test's score:

Edge cases:

Levels: Basic (under 1200), Capable (1200-2399), Advanced (2400-4799), Flagship (4800-9599) and Cutting-edge (9600 or more): each level starts at twice the score of the one below, and the reference phone (6000) is Flagship. A capped result is at most Capable; partial results (a scored test left out) get no level.

Fairness rules and versioning