Effects iOS
Effects / Guides / Custom frame pipelines / iOS

Process external CVPixelBuffer frames on iOS with Nosmai Effects

Updated August 22, 2026

Send BGRA pixel buffers through the Nosmai off-screen pipeline and receive processed output for your own encoder or renderer.

Prerequisites

  • A working iOS quickstart
  • An app-owned BGRA CVPixelBuffer source with stable dimensions

When to use this guide

Use off-screen mode when the host supplies external frames. If Nosmai owns the camera and another component only needs processed output, use NosmaiCore.liveFrameStreamCallback instead.

Keep one camera/frame source. Do not start a Nosmai-owned capture session and an app-owned AVCaptureSession for the same camera. Flutter and React Native do not expose this native input method as a general-purpose Dart/JavaScript pixel-buffer API; implement custom capture/RTC routing in a native adapter.

Prepare the output callback

NosmaiSDK *sdk = [NosmaiSDK sharedInstance];

[sdk setCVPixelBufferCallback:^(CVPixelBufferRef output, double timestamp) {
    CVPixelBufferRetain(output);
    dispatch_async(encoderQueue, ^{
        [consumer consumePixelBuffer:output timestamp:timestamp];
        CVPixelBufferRelease(output);
    });
}];

Retain the output only when the consumer continues after the callback returns. encoderQueue and consumer are host-owned. Bound their queue to avoid retaining a buffer for every incoming camera frame when the encoder falls behind. The callback is a single output registration: do not overwrite another adapter's registration. If several components need the same frames, let one owner fan them out with explicit retention.

Initialize off-screen mode

BOOL ready = [sdk initializeOffscreenWithWidth:width height:height];
if (ready) {
    [sdk setProcessingMode:NosmaiProcessingModeOffscreen];
}

Input buffers must use kCVPixelFormatType_32BGRA and keep stable dimensions. Read the actual size and pixel format from the supplied buffer; a camera configured for bi-planar NV12 is not BGRA just because its size matches. Convert unsupported input in the host before submission and reinitialize off-screen dimensions when the source changes.

Processing is initialized after the normal SDK/licence setup. Check isExternalProcessingAvailable and initialization success before accepting frames from the source.

Submit a frame

BOOL accepted = [sdk processFrame:inputPixelBuffer mirror:NO];

You can pass a CMSampleBufferRef with processSampleBuffer:mirror:. For asynchronous submission, retain the input until processFrameAsync:mirror:completion: finishes.

CVPixelBufferRetain(inputPixelBuffer);
[sdk processFrameAsync:inputPixelBuffer mirror:NO
           completion:^(BOOL success, NSError *error) {
    CVPixelBufferRelease(inputPixelBuffer);
    if (!success) {
        NSLog(@"External frame failed: %@", error.localizedDescription);
    }
}];

Keep at most one submission in progress rather than enqueuing every camera frame. The completion describes submission processing; the output callback is where the processed image is delivered. Do not assume a one-to-one callback count during pipeline changes or use a successful return as a face-detection signal.

Orientation, timestamps, and threading

The submission methods expose a horizontal mirror flag, not a rotation-in- degrees argument. Supply buffers in the orientation expected by the custom pipeline and apply any required physical rotation before submission. Do not confuse a UIImage display-orientation tag with the orientation of its buffer pixels. Validate face-effect alignment on both camera positions.

Callback timestamps are in seconds. Keep a consistent monotonic timebase for the downstream recorder/RTC adapter; convert units explicitly when its API uses milliseconds, nanoseconds, or CMTime. These methods do not expose a custom per-frame timestamp argument, so do not assume arbitrary source PTS is passed through unchanged.

Use one serialized input owner and avoid changing effect/session state while teardown is racing a frame. The SDK manages its native rendering context; the host does not need to make an unrelated OpenGL context current before calling the pixel-buffer API. Keep callbacks short and dispatch UI work to the main queue. Do not recursively submit another frame or tear down the rendering pipeline from inside its output callback.

Receive SDK-owned camera output instead

When the ordinary Nosmai camera is already running, use the high-level live output property rather than starting off-screen input:

[NosmaiCore shared].liveFrameStreamCallback =
    ^(CVPixelBufferRef output, double timestampSeconds) {
        // Consume synchronously, or retain before passing to bounded async work.
    };

This callback runs on a background processing thread. Set it to nil when the consumer stops. The low-level setCVPixelBufferCallback: and the Core live output property are alternative ownership routes, not independent broadcast subscriptions; use one adapter owner rather than installing competing handlers.

Return to live camera mode

Stop the external source first, then clear its callback:

[sdk setCVPixelBufferCallback:nil];
[sdk setProcessingMode:NosmaiProcessingModeLive];

Wait for submitted frames and downstream retained buffers to finish before releasing source or encoder resources. Setting live mode does not open a camera by itself; use the normal camera lifecycle when returning to preview.

Verify it worked

Apply a visible effect and confirm the output buffer orientation, colors, timestamps, and ownership on a physical device. Use getProcessingMetrics during development to inspect processed and dropped frames.

Common errors

SymptomCauseFix
processFrame returns NOOff-screen mode is not ready or the pixel format is unsupportedInitialize the exact dimensions and submit kCVPixelFormatType_32BGRA
Output crashes after the callbackThe consumer kept a borrowed buffer without retaining itRetain before asynchronous use and release after consumption

Next steps

References

Nosmai

An AR and AI company. Founded on the idea that the next decade of software will be visual, and that the tools for it should be within reach of anyone building.

Newsletter

Product updates and release notes. No spam.

© 2026 nosmai, inc · all rights reserved