Sensors & capture, ISP blocks, video processing, codecs, display/output, transport & sync, bandwidth & memory, platforms, calibration, key specs, design hooks, and application recipes.
Sensors & Capture
Image Sensors & Interfaces (MIPI CSI-2/SLVS-EC/LVDS/Parallel)
Multi-lane, high data rates; global/rolling shutter; FSIN/GEN sync.
Raw Streams (Bayer/YUV/Mono)
8/10/12/14-bit; line/frame timing & virtual channels.
Image Signal Processing (ISP)
Demosaic & White Balance
Adaptive interpolation; gray-world/learned AWB.
Auto Exposure & Auto Focus
HDR histograms; contrast AF / phase-assist AF.
Denoise & Sharpen (2D/3DNR, EE)
Temporal 3DNR with motion compensation; edge enhancement.
HDR Fusion Modes
Staggered/line-interleaved/long-short exposures; ghost control.
Lens Shading & Dewarp
LSC LUT & geometric correction (fisheye/warp).
Color Pipeline (CCM/Gamma/Tone)
Calibration matrices & tone/gamma mapping curves.
ISP Stats & Metadata
Histograms/ROIs; on-die temp & in-frame timestamps.
Video Processing & Post
Scaling & Resampling
Polyphase/edge-adaptive; SR (SRCNN/ESRGAN-style).
Temporal Processing
Stabilization/deflicker; frame rate conversion (FRC).
Overlay & Composition
Multi-plane, alpha blend, chroma key.
Deinterlacing/Defog/Derain
Field-to-frame & visibility restoration.
Video Analytics (DNN/Classic CV)
Real-time detect/segment/track; NPU/FPGA offload.
Codecs
Encoders (H.264/H.265/AV1)
CBR/VBR, low latency, slices/tiles—quality vs bitrate.
Decoders
Multi-stream 4K/8K, error concealment.
VQ Metrics (PSNR/SSIM/VMAF)
Algorithm/parameter closed-loop tuning.
Display & Output
Display Interfaces
HDMI/DP/eDP/MIPI DSI/LVDS/BT.1120; multi-display & HDR.
Color & HDR
BT.709/2020; PQ/HLG & gamut mapping.
Display Controller & Blender
Overlay/rotate/scale; double/triple buffering; TE sync.
Transport & Sync
Transport over IP (RTP/RTSP/SRT/NDI)
End-to-end latency & jitter buffering.
Hardware Timestamping (PTP/IEEE 1588, ST 2059)
Multi-camera sync & delay calibration.
Broadcast I/O (3G/6G/12G-SDI, ASI, TS)
Routing/switching and return feeds.
Memory & Bandwidth
Bandwidth Estimation
≈ res × bit-depth × fps × channels; AFBC/tiling reduce BW.
DDR/LPDDR & AXI QoS
RW priorities, row-hit rate, jitter avoidance.
Visually-Lossless Compression
AFBC/TICO/JPEG-XS for low-latency links/displays.
Hardware Platforms
Vision SoC / MPU
ISP + VPU + NPU—capture→AI→display in one.
FPGA Image Pipelines
Deterministic latency; MIPI/SLVS-EC fan-in; line-rate ops.
Custom ISP/NPU on ASIC
Peak perf/W for fixed tasks—requires early algo freeze.
Calibration & Imaging Science
Camera Calibration
Intrinsics/extrinsics; multi-camera epipolar setups.
Distortion & Vignetting
LUTs & geometric models.
Color Management (ICC/3D LUT)
Gray-card/spectral workflows.
Shutter Sync
Mitigate motion artifacts in mixed global/rolling setups.
Key Specs & Selection
ISP Capabilities
Bit depth, MPix/s, HDR modes, 3DNR & demosaic quality.
Codec Capabilities
Res/FPS, B-frames & low latency, bitrate ranges, ROI.
Memory/Bandwidth
DDR rate/width & on-chip SRAM/tiling efficiency.
Latency
Capture→ISP→codec→transport→display; <100 ms/ <30 ms tiers.
Color
Bit depth, gamut (709/2020), ΔE & curve accuracy.
Power
mW/MPix; per-stream codec power; NPU TOPS/W.
Reliability
Thermal drift, hot noise, dark current, defect pixels/rows.
Design Hooks & Pitfalls
Clocks & Sync
Sensor clocks/triggers; PTP/TE/VSYNC alignment; unified timebase.
Bandwidth & QoS
Shared DDR across ISP/codec/display—AXI QoS to avoid stutter.
Buffers & Latency
VBV/jitter buffers; no B-frames & small GOP for low latency.
Noise & HDR
3DNR vs detail at high ISO; motion-segmented HDR ghost control.
Color Consistency
Multi-cam AWB/CCM alignment; display EOTF/gamut match.
EMI/Signal Integrity
MIPI/SLVS-EC length/impedance/returns; HDMI/DP ESD & EQ.
Thermals & Hot Noise
Sensor↔ISP thermal coupling; cooling & bad-pixel growth.
Containers & Wrapping
MP4/TS/Matroska choices affect compatibility & trick-play.
Algorithm Deployment
NPU/FPGA order vs ISP; INT8/FP16 quantization effects.
Watchdogs & Health
Drop detection, black-box logs, link & frame-health monitors.
Application Recipes
Machine Vision
Multi-CSI-2 + low-latency ISP + calibration + TSN/timestamps.
Video Conferencing/Education
Multi-cam/mic, no B-frames, echo cancel, auto-framing.
Security/Surveillance
Starlight + 3DNR/HDR + ROI coding + SRT/RTSP.
Automotive Perception
Multi-cam calib, dewarp/stitch (surround view), AEB/ADAS prep.
UAV/Action Cameras
EIS + low-latency link + high-bitrate H.265/AV1 + telemetry OSD.
Quick Cheats
Raw Bandwidth
≈ resolution × frame-rate × bit-depth × channels.
Bitrate Heuristics
1080p60 H.264 ≈ 6–10 Mbps; H.265 −30–50%; AV1 −20–30%.
End-to-End Latency
Capture/ISP/codec/transport/display—budget by stage.
AE/AF Ordering
Low-light/HDR: stabilize AE before AF to reduce breathing.
