Prerequisites
- A working iOS quickstart
- An app-owned BGRA
CVPixelBuffersource with stable dimensions
When to use this guide
Use off-screen mode when the host supplies external frames. If Nosmai owns the camera and another component only needs processed output, use NosmaiCore.liveFrameStreamCallback instead.
Keep one camera/frame source. Do not start a Nosmai-owned capture session and an app-owned AVCaptureSession for the same camera. Flutter and React Native do not expose this native input method as a general-purpose Dart/JavaScript pixel-buffer API; implement custom capture/RTC routing in a native adapter.
Prepare the output callback
NosmaiSDK *sdk = [NosmaiSDK sharedInstance];
[sdk setCVPixelBufferCallback:^(CVPixelBufferRef output, double timestamp) {
CVPixelBufferRetain(output);
dispatch_async(encoderQueue, ^{
[consumer consumePixelBuffer:output timestamp:timestamp];
CVPixelBufferRelease(output);
});
}];
Retain the output only when the consumer continues after the callback returns. encoderQueue and consumer are host-owned. Bound their queue to avoid retaining a buffer for every incoming camera frame when the encoder falls behind. The callback is a single output registration: do not overwrite another adapter's registration. If several components need the same frames, let one owner fan them out with explicit retention.
Initialize off-screen mode
BOOL ready = [sdk initializeOffscreenWithWidth:width height:height];
if (ready) {
[sdk setProcessingMode:NosmaiProcessingModeOffscreen];
}
Input buffers must use kCVPixelFormatType_32BGRA and keep stable dimensions. Read the actual size and pixel format from the supplied buffer; a camera configured for bi-planar NV12 is not BGRA just because its size matches. Convert unsupported input in the host before submission and reinitialize off-screen dimensions when the source changes.
Processing is initialized after the normal SDK/licence setup. Check isExternalProcessingAvailable and initialization success before accepting frames from the source.
Submit a frame
BOOL accepted = [sdk processFrame:inputPixelBuffer mirror:NO];
You can pass a CMSampleBufferRef with processSampleBuffer:mirror:. For asynchronous submission, retain the input until processFrameAsync:mirror:completion: finishes.
CVPixelBufferRetain(inputPixelBuffer);
[sdk processFrameAsync:inputPixelBuffer mirror:NO
completion:^(BOOL success, NSError *error) {
CVPixelBufferRelease(inputPixelBuffer);
if (!success) {
NSLog(@"External frame failed: %@", error.localizedDescription);
}
}];
Keep at most one submission in progress rather than enqueuing every camera frame. The completion describes submission processing; the output callback is where the processed image is delivered. Do not assume a one-to-one callback count during pipeline changes or use a successful return as a face-detection signal.
Orientation, timestamps, and threading
The submission methods expose a horizontal mirror flag, not a rotation-in- degrees argument. Supply buffers in the orientation expected by the custom pipeline and apply any required physical rotation before submission. Do not confuse a UIImage display-orientation tag with the orientation of its buffer pixels. Validate face-effect alignment on both camera positions.
Callback timestamps are in seconds. Keep a consistent monotonic timebase for the downstream recorder/RTC adapter; convert units explicitly when its API uses milliseconds, nanoseconds, or CMTime. These methods do not expose a custom per-frame timestamp argument, so do not assume arbitrary source PTS is passed through unchanged.
Use one serialized input owner and avoid changing effect/session state while teardown is racing a frame. The SDK manages its native rendering context; the host does not need to make an unrelated OpenGL context current before calling the pixel-buffer API. Keep callbacks short and dispatch UI work to the main queue. Do not recursively submit another frame or tear down the rendering pipeline from inside its output callback.
Receive SDK-owned camera output instead
When the ordinary Nosmai camera is already running, use the high-level live output property rather than starting off-screen input:
[NosmaiCore shared].liveFrameStreamCallback =
^(CVPixelBufferRef output, double timestampSeconds) {
// Consume synchronously, or retain before passing to bounded async work.
};
This callback runs on a background processing thread. Set it to nil when the consumer stops. The low-level setCVPixelBufferCallback: and the Core live output property are alternative ownership routes, not independent broadcast subscriptions; use one adapter owner rather than installing competing handlers.
Return to live camera mode
Stop the external source first, then clear its callback:
[sdk setCVPixelBufferCallback:nil]; [sdk setProcessingMode:NosmaiProcessingModeLive];
Wait for submitted frames and downstream retained buffers to finish before releasing source or encoder resources. Setting live mode does not open a camera by itself; use the normal camera lifecycle when returning to preview.
Verify it worked
Apply a visible effect and confirm the output buffer orientation, colors, timestamps, and ownership on a physical device. Use getProcessingMetrics during development to inspect processed and dropped frames.
Common errors
| Symptom | Cause | Fix |
|---|---|---|
processFrame returns NO | Off-screen mode is not ready or the pixel format is unsupported | Initialize the exact dimensions and submit kCVPixelFormatType_32BGRA |
| Output crashes after the callback | The consumer kept a borrowed buffer without retaining it | Retain before asynchronous use and release after consumption |