Prerequisites
- A signed build using the final native and wrapper versions
- The exact effects, camera resolution, recording, and streaming configuration intended for release
- At least one lower-tier supported Android device, one current Android device, and one older supported iPhone
- A stable room-temperature starting condition and repeatable session duration
Define the test cases
Run each device through the same workload sequence:
- Preview with no active effect.
- One lightweight LUT or colour package.
- One face-tracked beauty or AR package.
- Background segmentation.
- Processed recording.
- The production streaming handoff.
Record the authored package identifier and version. Two packages in the same category can have materially different GPU and inference cost.
Capture the matrix
Download the device-matrix CSV template and create one row for every device, effect, and workload combination. Keep raw measurements separate from conclusions so later releases can be compared with the same fields.
| Field | Record |
|---|---|
| Device | Model, OS, chipset, RAM, ABI, battery health if known |
| Input | Camera, resolution, target FPS, front or back, lighting |
| Workload | Package ID, enabled beauty/background features, recording or provider |
| Frame rate | Average, low interval, visible stutter, and time to first processed frame |
| Memory | Baseline, steady state, peak, and value after cleanup |
| Thermal | Initial state, throttling transition, temperature or platform state, and time |
| Battery | Start/end percentage and fixed-duration session length |
| Lifecycle | Navigation loops, background/foreground, lock/unlock, rotation, camera switch |
| Outcome | Pass, conditional pass, fail, logs, and reproduction steps |
Run long enough to expose throttling
Warm the app with the target workload and continue for the same fixed duration on every device. A short cold-device run does not represent sustained live streaming. Do not compare devices that began at materially different thermal or battery states without recording that difference.
Set release thresholds
Define thresholds from the product experience rather than publishing one universal SDK number. At minimum specify acceptable frame rate, maximum startup delay, memory recovery after cleanup, recording success, camera lifecycle success rate, and whether thermal throttling is allowed during the target session length.
Nosmai targets 30 FPS under supported conditions, but does not claim one certified accuracy, battery, thermal, or memory value for every device and effect. Report measured values with their complete test configuration.
Verify cleanup and recovery
Repeat open, close, background, foreground, camera switch, recording stop, and provider teardown. Memory should stabilize rather than grow with every cycle, and only one camera owner should exist. Preserve logs for failed cycles after redacting credentials and private service details.
Common errors
| Mistake | Why it invalidates the result | Fix |
|---|---|---|
| Comparing different effects as one benchmark | Pass count and inference requirements vary by package | Run the same package version on every device |
| Testing a debug build | Tooling and assertions distort size and timing | Use the signed release configuration |
| Reporting only average FPS | Long stalls and thermal drops disappear in the mean | Include low intervals, session duration, and thermal transition |
