Effects Android
Effects / Guides / Custom frame pipelines / Android

Process external I420 frames on Android with Nosmai Effects

Updated August 22, 2026

Send direct I420 planes through the Nosmai external-frame pipeline and deliver the processed result to your own consumer.

Prerequisites

  • A working Android quickstart
  • One app-owned camera or video source, with known dimensions and orientation
  • One processing owner and, for GPU export, a compatible shared EGL context

When to use this guide

The native Android integration leaves Camera2 ownership to the host. Choose one input route; do not feed OES, YUV, and external I420 simultaneously into the same session.

RouteInput contractUse
Preview-backed OESCamera2 writes to the SDK-provided SurfaceTextureGPU camera input with processed texture export
Preview-backed YUVImageReader planes with explicit row and chroma pixel stridesCamera2 fallback or an existing YUV source
Standalone external I420Direct planar Y, U, and V buffers; chroma pixel stride is oneSynchronous CPU-plane input/output

OES avoids the CPU-plane upload used by YUV. It is not a universal guarantee of a particular frame rate: qualify the actual device, driver, effect, and encoder together. External I420 is a separate path; do not assume it has identical face-effect behaviour to the preview-backed paths. Validate tracking and makeup, not only colour filters, on the chosen route.

Configure OES camera input

Register the consumer's shared EGL context before NosmaiSDK.initialize if processed textures will be consumed by another engine. Despite the method's name, setAgoraShareContext is also used by the native LiveKit/WebRTC examples:

NosmaiSDK.setAgoraShareContext(sharedEglContextHandle);
NosmaiSDK.initialize(context, licenseKey);

Enable OES before starting processing. Wait for the SDK-provided surface, configure the actual camera buffer size, then start Camera2. This example is for a portrait setup with a front camera whose sensor orientation is 270:

final int inputWidth = 1280;
final int inputHeight = 720;
final boolean front = true;
final int sensorOrientationDegrees = 270;
final int rotationMode = 1;

previewView.enableOesInput(true);
NosmaiPreviewView.OnOesReadyListener oesReady = surfaceTexture -> {
    previewView.post(() -> {
        surfaceTexture.setDefaultBufferSize(inputWidth, inputHeight);
        previewView.setCameraOrientation(front, sensorOrientationDegrees);
        NosmaiSDK.setCameraFacing(front);
        NosmaiSDK.setMirrorOverride(NosmaiSDK.MIRROR_AUTO);
        previewView.setOesInputFrameInfo(inputWidth, inputHeight, rotationMode);
        camera2Source.open(surfaceTexture);
    });
};
previewView.addOnOesReadyListener(oesReady);
NosmaiSDK.startProcessing(previewView);

camera2Source is the app's serialized Camera2 owner. It creates the capture session targeting a Surface backed by that SurfaceTexture. Do not replace the SDK's frame-available listener, call updateTexImage() yourself, or reuse a surface from a retired preview.

Dimensions, rotation, and mirroring

setOesInputFrameInfo(width, height, rotationMode) takes the camera buffer dimensions before rotation. A 1280x720 buffer that produces portrait output still uses 1280, 720, not 720, 1280. Reconfigure after a camera or input-size change and reopen Camera2 against the current OES-ready surface.

The third argument is a rotation-mode ID, not raw sensor degrees. The basic Android rotation IDs are 0 (none), 1 (left), 2 (right), and 8 (180°); other IDs include flips. In the portrait setup used by the mobile examples, front sensor 270 uses mode 1, and rear sensor 90 uses mode 2. These examples are not a universal mapping for every display rotation or a source that has already rotated its pixels.

setCameraOrientation(isFront, sensorOrientationDegrees) is different: its second argument is the sensor orientation in degrees. It does not replace the input rotation metadata. Apply mirroring once, using a deliberate mirror policy; do not independently mirror the input, texture coordinates, and RTC frame unless the pipeline requires those separate transforms.

Keep the preview attached with a valid lifecycle and non-zero size. For diagnosis, first use a normally sized preview and verify face-effect alignment before hiding it. STREAMING_ONLY suppresses on-screen display; DUAL_OUTPUT also displays the preview. Selecting DUAL_OUTPUT is not a face-detection switch and does not repair incorrect input metadata.

Feed Camera2 YUV planes

For a YUV session, call enableOesInput(false) before startProcessing and wait for previewView.addOnPipelineReady. Deliver YUV_420_888 images from a dedicated ImageReader handler, not the UI thread:

Image image = imageReader.acquireLatestImage();
if (image != null) {
    try {
        if (previewView.isPipelineReady()) {
            Image.Plane[] planes = image.getPlanes();
            previewView.processYuvFrame(
                planes[0].getBuffer(), planes[1].getBuffer(), planes[2].getBuffer(),
                image.getWidth(), image.getHeight(),
                planes[0].getRowStride(), planes[1].getRowStride(),
                planes[2].getRowStride(), planes[1].getPixelStride(),
                planes[2].getPixelStride(), rotationMode);
        }
    } finally {
        image.close();
    }
}

The final argument is the same rotation-mode ID convention used by the preview input, not the degree convention of the standalone I420 methods below. Do not assume U and V pixel stride is one, swap them, close the image before processing returns, or retain its planes for later work after closing it.

Use one frame-processing owner, keep only the newest waiting image, and do not race frame processing with teardown or input reconfiguration. Serialize control/session changes; these APIs are not a promise that arbitrary SDK calls from unrelated threads are safe. Reduce resolution or the enabled effect set if processing exceeds the frame budget.

requestRenderUpdate() requests display redraw; it does not submit another camera frame. OES is frame-driven. Do not add an unconditional extra redraw after every OES frame or assume a redraw is required to obtain an exported texture. Keep any YUV display scheduling separate from frame submission and measure it in the final host pipeline.

Export processed GPU textures

setTextureFrameCallback delivers (textureId, width, height, timestampNs, producerFence). The result is a processed GL_TEXTURE_2D, not the camera's external OES texture. The fifth argument is a fence handle, not a pool slot index. Never cast it to int for release.

Use the texture ID with waitStreamFrame, completeStreamFrame, and releaseStreamSlot. The consumer must have a current EGL context in the registered share group. This bounded-copy pattern allows one pending copy:

AtomicBoolean copyBusy = new AtomicBoolean(false);
NosmaiSDK.setRenderMode(NosmaiSDK.RenderMode.STREAMING_ONLY);
NosmaiSDK.setTextureFrameCallback((textureId, outputWidth, outputHeight, timestampNs, producerFence) -> {
    if (!copyBusy.compareAndSet(false, true)) {
        NosmaiSDK.releaseStreamSlot(textureId);
        return;
    }
    boolean posted = consumerGlHandler.post(() -> {
        try {
            if (!NosmaiSDK.waitStreamFrame(textureId)) {
                NosmaiSDK.releaseStreamSlot(textureId);
                return;
            }
            try {
                textureCopier.copyToOwnedFbo(textureId, outputWidth, outputHeight, timestampNs);
            } finally {
                NosmaiSDK.completeStreamFrame(textureId);
            }
        } finally {
            copyBusy.set(false);
        }
    });
    if (!posted) {
        copyBusy.set(false);
        NosmaiSDK.releaseStreamSlot(textureId);
    }
});

consumerGlHandler and textureCopier belong to the host. The handler must make its shared EGL context current before running copies; the copier samples the borrowed texture into an app-owned framebuffer. Forward that owned result to the RTC SDK using the provider's own retention rules.

After a successful wait, completeStreamFrame(textureId) fences the consumer's GPU reads and returns the frame. For an unused/dropped frame, call releaseStreamSlot(textureId) instead. Complete or release exactly once; never do both, retain a borrowed texture afterwards, or block the producer callback waiting for the network. Do not depend on a fixed pool size.

setTexturePoolEnabled is an experimental allocation-pooling option, disabled by default; it is not required for this callback and does not remove frame ownership obligations.

To stop, prevent new deliveries, clear the callback, drain consumer work and return outstanding frames before stopping Nosmai or destroying either EGL context. Switch back to PREVIEW_ONLY only if the same preview will continue.

Prepare standalone I420 processing

Initialize Nosmai first, then configure the stable input dimensions:

boolean ready = NosmaiSDK.initializeExternalFramePipeline(width, height);
if (!ready) {
    throw new IllegalStateException("External processing is unavailable");
}

NosmaiSDK.setExternalFrameMode(true);
NosmaiSDK.setExternalFrameThreadMode(NosmaiSDK.ThreadMode.GL_THREAD);

Reinitialize the pipeline if the input dimensions change.

Process I420 planes

NosmaiSDK.ProcessedI420Frame output =
    NosmaiSDK.processExternalI420Sync(
        yBuffer, uBuffer, vBuffer,
        width, height,
        yStride, uStride, vStride,
        rotation, mirror
    );

if (output != null) {
    consumer.onFrame(output);
}

The three inputs must be direct ByteBuffer instances with correct strides. Consume or copy the returned native buffers before submitting the next frame. Here rotation is degrees (0, 90, 180, or 270), not the preview's rotation-mode ID. I420 has separate planar U and V samples with pixel stride one. Camera2 YUV_420_888, NV21, or NV12 data is not automatically planar I420; repack it first or use processYuvFrame with explicit pixel strides.

Process writable planes in place

boolean processed = NosmaiSDK.processExternalI420InPlace(
    yBuffer, uBuffer, vBuffer,
    width, height,
    yStride, uStride, vStride,
    rotation, mirror
);

Use this path only when the source planes are writable and large enough for their declared strides. The call blocks until processing finishes.

Stop safely

Stop the source before disabling the pipeline:

NosmaiSDK.clearAllEffects();
NosmaiSDK.setFrameCallback(null);
NosmaiSDK.setExternalFrameMode(false);

For preview-backed input, close Camera2/ImageReader first, clear callbacks, drain exported textures, then call stopProcessing(previewView). Recreate surface-dependent state on the next session rather than reusing old handles.

Verify it worked

Use a visible effect and confirm output dimensions, orientation, timestamp order, and color on a physical arm64 device. Track dropped or late frames in the host consumer.

Common errors

SymptomCauseFix
Processing returns null or falseBuffer type, strides, dimensions, or pipeline state are invalidUse direct buffers and reinitialize after an input-size change
Colors or orientation are wrongPlane order, stride, rotation, or mirror metadata is incorrectValidate the source contract before submitting the frame

Next steps

References

Nosmai

An AR and AI company. Founded on the idea that the next decade of software will be visual, and that the tools for it should be within reach of anyone building.

Newsletter

Product updates and release notes. No spam.

© 2026 nosmai, inc · all rights reserved