Prerequisites
- A working Android quickstart
- One app-owned camera or video source, with known dimensions and orientation
- One processing owner and, for GPU export, a compatible shared EGL context
When to use this guide
The native Android integration leaves Camera2 ownership to the host. Choose one input route; do not feed OES, YUV, and external I420 simultaneously into the same session.
| Route | Input contract | Use |
|---|---|---|
| Preview-backed OES | Camera2 writes to the SDK-provided SurfaceTexture | GPU camera input with processed texture export |
| Preview-backed YUV | ImageReader planes with explicit row and chroma pixel strides | Camera2 fallback or an existing YUV source |
| Standalone external I420 | Direct planar Y, U, and V buffers; chroma pixel stride is one | Synchronous CPU-plane input/output |
OES avoids the CPU-plane upload used by YUV. It is not a universal guarantee of a particular frame rate: qualify the actual device, driver, effect, and encoder together. External I420 is a separate path; do not assume it has identical face-effect behaviour to the preview-backed paths. Validate tracking and makeup, not only colour filters, on the chosen route.
Configure OES camera input
Register the consumer's shared EGL context before NosmaiSDK.initialize if processed textures will be consumed by another engine. Despite the method's name, setAgoraShareContext is also used by the native LiveKit/WebRTC examples:
NosmaiSDK.setAgoraShareContext(sharedEglContextHandle); NosmaiSDK.initialize(context, licenseKey);
Enable OES before starting processing. Wait for the SDK-provided surface, configure the actual camera buffer size, then start Camera2. This example is for a portrait setup with a front camera whose sensor orientation is 270:
final int inputWidth = 1280;
final int inputHeight = 720;
final boolean front = true;
final int sensorOrientationDegrees = 270;
final int rotationMode = 1;
previewView.enableOesInput(true);
NosmaiPreviewView.OnOesReadyListener oesReady = surfaceTexture -> {
previewView.post(() -> {
surfaceTexture.setDefaultBufferSize(inputWidth, inputHeight);
previewView.setCameraOrientation(front, sensorOrientationDegrees);
NosmaiSDK.setCameraFacing(front);
NosmaiSDK.setMirrorOverride(NosmaiSDK.MIRROR_AUTO);
previewView.setOesInputFrameInfo(inputWidth, inputHeight, rotationMode);
camera2Source.open(surfaceTexture);
});
};
previewView.addOnOesReadyListener(oesReady);
NosmaiSDK.startProcessing(previewView);
camera2Source is the app's serialized Camera2 owner. It creates the capture session targeting a Surface backed by that SurfaceTexture. Do not replace the SDK's frame-available listener, call updateTexImage() yourself, or reuse a surface from a retired preview.
Dimensions, rotation, and mirroring
setOesInputFrameInfo(width, height, rotationMode) takes the camera buffer dimensions before rotation. A 1280x720 buffer that produces portrait output still uses 1280, 720, not 720, 1280. Reconfigure after a camera or input-size change and reopen Camera2 against the current OES-ready surface.
The third argument is a rotation-mode ID, not raw sensor degrees. The basic Android rotation IDs are 0 (none), 1 (left), 2 (right), and 8 (180°); other IDs include flips. In the portrait setup used by the mobile examples, front sensor 270 uses mode 1, and rear sensor 90 uses mode 2. These examples are not a universal mapping for every display rotation or a source that has already rotated its pixels.
setCameraOrientation(isFront, sensorOrientationDegrees) is different: its second argument is the sensor orientation in degrees. It does not replace the input rotation metadata. Apply mirroring once, using a deliberate mirror policy; do not independently mirror the input, texture coordinates, and RTC frame unless the pipeline requires those separate transforms.
Keep the preview attached with a valid lifecycle and non-zero size. For diagnosis, first use a normally sized preview and verify face-effect alignment before hiding it. STREAMING_ONLY suppresses on-screen display; DUAL_OUTPUT also displays the preview. Selecting DUAL_OUTPUT is not a face-detection switch and does not repair incorrect input metadata.
Feed Camera2 YUV planes
For a YUV session, call enableOesInput(false) before startProcessing and wait for previewView.addOnPipelineReady. Deliver YUV_420_888 images from a dedicated ImageReader handler, not the UI thread:
Image image = imageReader.acquireLatestImage();
if (image != null) {
try {
if (previewView.isPipelineReady()) {
Image.Plane[] planes = image.getPlanes();
previewView.processYuvFrame(
planes[0].getBuffer(), planes[1].getBuffer(), planes[2].getBuffer(),
image.getWidth(), image.getHeight(),
planes[0].getRowStride(), planes[1].getRowStride(),
planes[2].getRowStride(), planes[1].getPixelStride(),
planes[2].getPixelStride(), rotationMode);
}
} finally {
image.close();
}
}
The final argument is the same rotation-mode ID convention used by the preview input, not the degree convention of the standalone I420 methods below. Do not assume U and V pixel stride is one, swap them, close the image before processing returns, or retain its planes for later work after closing it.
Use one frame-processing owner, keep only the newest waiting image, and do not race frame processing with teardown or input reconfiguration. Serialize control/session changes; these APIs are not a promise that arbitrary SDK calls from unrelated threads are safe. Reduce resolution or the enabled effect set if processing exceeds the frame budget.
requestRenderUpdate() requests display redraw; it does not submit another camera frame. OES is frame-driven. Do not add an unconditional extra redraw after every OES frame or assume a redraw is required to obtain an exported texture. Keep any YUV display scheduling separate from frame submission and measure it in the final host pipeline.
Export processed GPU textures
setTextureFrameCallback delivers (textureId, width, height, timestampNs, producerFence). The result is a processed GL_TEXTURE_2D, not the camera's external OES texture. The fifth argument is a fence handle, not a pool slot index. Never cast it to int for release.
Use the texture ID with waitStreamFrame, completeStreamFrame, and releaseStreamSlot. The consumer must have a current EGL context in the registered share group. This bounded-copy pattern allows one pending copy:
AtomicBoolean copyBusy = new AtomicBoolean(false);
NosmaiSDK.setRenderMode(NosmaiSDK.RenderMode.STREAMING_ONLY);
NosmaiSDK.setTextureFrameCallback((textureId, outputWidth, outputHeight, timestampNs, producerFence) -> {
if (!copyBusy.compareAndSet(false, true)) {
NosmaiSDK.releaseStreamSlot(textureId);
return;
}
boolean posted = consumerGlHandler.post(() -> {
try {
if (!NosmaiSDK.waitStreamFrame(textureId)) {
NosmaiSDK.releaseStreamSlot(textureId);
return;
}
try {
textureCopier.copyToOwnedFbo(textureId, outputWidth, outputHeight, timestampNs);
} finally {
NosmaiSDK.completeStreamFrame(textureId);
}
} finally {
copyBusy.set(false);
}
});
if (!posted) {
copyBusy.set(false);
NosmaiSDK.releaseStreamSlot(textureId);
}
});
consumerGlHandler and textureCopier belong to the host. The handler must make its shared EGL context current before running copies; the copier samples the borrowed texture into an app-owned framebuffer. Forward that owned result to the RTC SDK using the provider's own retention rules.
After a successful wait, completeStreamFrame(textureId) fences the consumer's GPU reads and returns the frame. For an unused/dropped frame, call releaseStreamSlot(textureId) instead. Complete or release exactly once; never do both, retain a borrowed texture afterwards, or block the producer callback waiting for the network. Do not depend on a fixed pool size.
setTexturePoolEnabled is an experimental allocation-pooling option, disabled by default; it is not required for this callback and does not remove frame ownership obligations.
To stop, prevent new deliveries, clear the callback, drain consumer work and return outstanding frames before stopping Nosmai or destroying either EGL context. Switch back to PREVIEW_ONLY only if the same preview will continue.
Prepare standalone I420 processing
Initialize Nosmai first, then configure the stable input dimensions:
boolean ready = NosmaiSDK.initializeExternalFramePipeline(width, height);
if (!ready) {
throw new IllegalStateException("External processing is unavailable");
}
NosmaiSDK.setExternalFrameMode(true);
NosmaiSDK.setExternalFrameThreadMode(NosmaiSDK.ThreadMode.GL_THREAD);
Reinitialize the pipeline if the input dimensions change.
Process I420 planes
NosmaiSDK.ProcessedI420Frame output =
NosmaiSDK.processExternalI420Sync(
yBuffer, uBuffer, vBuffer,
width, height,
yStride, uStride, vStride,
rotation, mirror
);
if (output != null) {
consumer.onFrame(output);
}
The three inputs must be direct ByteBuffer instances with correct strides. Consume or copy the returned native buffers before submitting the next frame. Here rotation is degrees (0, 90, 180, or 270), not the preview's rotation-mode ID. I420 has separate planar U and V samples with pixel stride one. Camera2 YUV_420_888, NV21, or NV12 data is not automatically planar I420; repack it first or use processYuvFrame with explicit pixel strides.
Process writable planes in place
boolean processed = NosmaiSDK.processExternalI420InPlace(
yBuffer, uBuffer, vBuffer,
width, height,
yStride, uStride, vStride,
rotation, mirror
);
Use this path only when the source planes are writable and large enough for their declared strides. The call blocks until processing finishes.
Stop safely
Stop the source before disabling the pipeline:
NosmaiSDK.clearAllEffects(); NosmaiSDK.setFrameCallback(null); NosmaiSDK.setExternalFrameMode(false);
For preview-backed input, close Camera2/ImageReader first, clear callbacks, drain exported textures, then call stopProcessing(previewView). Recreate surface-dependent state on the next session rather than reusing old handles.
Verify it worked
Use a visible effect and confirm output dimensions, orientation, timestamp order, and color on a physical arm64 device. Track dropped or late frames in the host consumer.
Common errors
| Symptom | Cause | Fix |
|---|---|---|
Processing returns null or false | Buffer type, strides, dimensions, or pipeline state are invalid | Use direct buffers and reinitialize after an input-size change |
| Colors or orientation are wrong | Plane order, stride, rotation, or mirror metadata is incorrect | Validate the source contract before submitting the frame |
Next steps
- Understand off-screen rendering
- Integrate Agora on Android
- Native WebRTC example
- Native LiveKit example