123 Main Street, New York, NY 10001

Submit Your BOM

On-Camera / Edge Analytics for Security Cameras

On-Camera / Edge Analytics for Security Cameras

← Back to: Security & Surveillance

On-Camera / Edge Analytics means the camera itself runs the full inference pipeline and outputs trustworthy results—stable events/metadata and ROI or masked video—under real limits like thermal, bandwidth, and worst-case scenes. The core is an evidence-backed output contract (timestamps, model/version IDs, integrity checks) so every alarm can be reproduced, verified, and tied to the exact video segment.

H2-1. What “On-Camera / Edge Analytics” Really Means

Purpose: lock the scope and define a verifiable output contract (events, metadata, ROI streams) that proves analytics truly runs on the camera—not in VMS/NVR.

Card A — What it is / isn’t (acceptance rules)

  • It is inference + decision + packaging performed on the camera, producing outputs that can be tied back to specific frames/PTS (not just “alarm happened”).
  • It is bounded-latency behavior under load (motion spikes, bitrate spikes, thermal throttling), proven by p95 timing from pipeline nodes.
  • It is reproducible behavior: outputs carry model/pipeline identity so field changes are explainable and traceable.
  • It isn’t “forward the stream to NVR/VMS to infer later” (that is platform analytics, not on-camera analytics).
  • It isn’t an opaque alarm with no evidence fields (no track, no rule, no timestamps → not debuggable, not auditable).
Hard constraints Thermal / power caps Deterministic latency targets Bounded memory & queues Fault recovery (safe degrade)

Card B — Output contract (minimum fields)

  • Identity: device_id, fw_ver, pipeline_profile, model_id, model_hash
  • Time binding: frame_id/pts reference, and event window start_pts–end_pts (or equivalent)
  • Explainability: rule_id, zone_id, track_id, confidence, key attributes (as needed)
  • Debug hooks: queue_depth, drop_reason, throttle_state (prove why outputs degrade)
Acceptance checklist (measurable): (1) bounded-latency event emit, (2) metadata explains “why/where”, (3) model/pipeline tags make results reproducible across upgrades and RMAs.
On-Camera / Edge Analytics: Output Contract Inference + decision + packaging happen on the camera Sensor Frames ISP Image prep Preproc Resize / norm NPU / DSP Inference Postproc Track / rules Outputs (must be verifiable) Events event_id start_pts / end_pts Metadata track_id rule_id / zone_id ROI Map tiles / QP map Sub-stream low-res / ROI Masked privacy Identity tags (reproducible) device_id · fw_ver · model_id · model_hash · profile Constraints: thermal latency memory fault recovery
Figure F1. A camera-side analytics pipeline must output verifiable events/metadata/ROI/sub-streams/masked video, with identity tags and bounded-latency evidence hooks.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F1
ICNavigator | On-Camera / Edge Analytics | Fig F1 (Pipeline + Output Contract)

H2-2. Reference Architecture: Multi-Stream Ingest, Split-Pipelines, and Where Work Runs

Purpose: provide a canonical block-level architecture that maps tasks to engines (ISP/Codec/NPU/DSP/CPU) and defines the minimum observability (timestamps + counters) needed to debug field behavior.

Card A — Canonical architecture rules (task → engine)

  • Multi-stream ingest: produce at least a main stream (quality) and an analytics stream (stable), plus snapshot/clip outputs as needed.
  • Split-pipeline principle: video quality optimization and analytics stability are different objectives; separate branches avoid “good video but unstable analytics”.
  • Engine mapping: ISP prepares images; Codec encodes streams; NPU runs CNN inference; DSP handles lightweight transforms/filters/post ops; CPU orchestrates scheduling, rules, and event packaging.
  • Shared resource reality: DDR/DMA is the bottleneck hub—bitrate/motion spikes can starve preproc/inference even when NPU is “fine”.
  • Deterministic behavior: define queue limits and drop policies (with reasons) so the camera degrades safely instead of failing silently.
Streams Main stream Sub-stream(s) Analytics downscale Snapshot / event clip

Card B — Minimum observability (timestamps + counters)

  • Pipeline timestamps (locate where time is spent): ts_isp_out, ts_preproc, ts_infer_done, ts_event_emit
  • Engine utilization (see which unit is saturated): npu_util%, dsp_util%, codec_util%
  • Memory / bandwidth (prove contention): ddr_bw (or read/write split), dma_queue_depth
  • Health / backpressure (prove safe degrade): queue_depth, frame_drop_count, drop_reason, throttle_state
Debug principle: timestamps show where latency expands; counters show why (codec bursts, DDR contention, thermal throttling, queue backpressure).
Reference Architecture (Video / Analytics / Control) Multi-stream ingest + split pipelines + shared DDR/DMA observability DDR / Shared Memory frames · tensors · ROI · masks DMA queues Video lane (quality) Analytics lane (stability) Control lane (orchestration) ISP Encoder Main stream Sub-stream Preproc NPU DSP/CPU Events CPU Scheduler Rules / Config Health Monitor Minimum observability ts_isp_out · ts_preproc · ts_infer_done · ts_event_emit npu_util% · dsp_util% · codec_util% · ddr_bw queue_depth · drop_reason · throttle_state
Figure F2. A practical camera-side architecture uses split pipelines (video vs analytics) and treats DDR/DMA as the shared bottleneck; debug requires timestamps + resource counters.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F2
ICNavigator | On-Camera / Edge Analytics | Fig F2 (Video/Analytics/Control + DDR/DMA)

H2-3. NPU/DSP Pipeline Scheduling: Deterministic Latency Under Thermal & Load

Goal: keep inference stable when bitrate spikes, motion increases, or thermal throttling occurs—by using explicit scheduling modes, queue backpressure, and graceful degradation with measurable evidence.

Card A — 3 scheduling patterns (when to use + how they fail)

  • Fixed-FPS inference: targets a constant analytics rate for predictable alarms. Requires hard queue limits; otherwise latency grows without bound under bursts.
  • Adaptive inference: adjusts infer_fps and/or input size based on queue_depth or p95_latency_ms. Must enforce a minimum service level (avoid long blind gaps).
  • Event-driven inference: runs on triggers (scene change, zone hit, external input). Needs debounce/cooldown and a periodic “heartbeat” run to avoid trigger loss.
Backpressure should be a controlled degrade (bounded queues + reasoned drops), not accidental random stalls. Timestamp continuity must remain valid even when frames are skipped or dropped.
Backpressure queue depth caps drop policy frame-skip strategy graceful degrade

Card B — Debug checklist (first 2 measurements)

  • First measure: per-stage queue_depth (preproc / infer / postproc). Identify where backlog accumulates.
  • Second measure: infer_fps with throttle_state. Determine whether collapse is thermal/clock-driven or contention-driven.
  • Latency evidence (minimum table fields): queue_depth, infer_fps, p95_latency_ms, throttle_state
  • Drop reason codes (must be explicit): DROP_QUEUE_FULL, DROP_THERMAL, DROP_BW_LIMIT
Interpreting drop codes: DROP_QUEUE_FULL proves queue caps work; DROP_THERMAL indicates clock/thermal limits; DROP_BW_LIMIT suggests DDR/bus contention (often burst encode complexity).
Scheduler State Machine (Deterministic Latency) Queues + drop gates keep p95 latency bounded under load and thermal throttling NORMAL stable queues CONGESTED q rising / bursts THROTTLED clock/thermal cap RECOVERY cooldown q>q_hi temp>t_hi cooldown q<q_lo & stable Queues + caps (backpressure) Hard limits prevent unbounded latency growth Q_preproc cap: q_pre_max fill headroom Q_infer cap: q_inf_max fill headroom Q_post cap: q_post_max fill Drop gates (explicit) Reason codes must be logged DROP_QUEUE_FULL q > cap DROP_THERMAL throttle_state DROP_BW_LIMIT ddr/bus contention KPI: infer_fps p95_latency_ms queue_depth throttle_state
Figure F3. Deterministic latency comes from explicit queue caps and logged drop gates; the state machine encodes safe degradation and recovery under load and thermal throttling.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F3
ICNavigator | On-Camera / Edge Analytics | Fig F3 (Scheduling State Machine + Queues + Drop Gates)

H2-4. Pre-Processing for Inference: Resize, Normalize, ROI Crop, and Sensor Variance Handling

Goal: prevent “works in the lab, fails in the field” by treating pre-processing as a measurable input contract: stable tensors despite exposure/WDR/IR variance and quantization constraints.

Card A — Common preproc mistakes (field-failure patterns)

  • Color space mismatch: YUV/RGB conversion or matrix mismatch changes the model’s world (accuracy collapses with “no obvious bug”).
  • Aspect ratio errors: stretching instead of letterboxing introduces geometry distortion; boxes drift and scale incorrectly.
  • Normalization mismatch: wrong mean/std or scaling range (0–1 vs 0–255) shifts distributions and breaks thresholds.
  • ROI crop coordinate mismatch: ROI defined in one coordinate space but applied in another; results look “randomly wrong”.
  • Over-correction: aggressive lens correction can warp edges; keep corrections light and consistent.
  • IR/WDR discontinuities: day/night or WDR mode switches cause abrupt distribution shifts; stabilize via profiles, not ad-hoc tuning.
  • INT8 quant pitfalls: incorrect scale/zero-point or per-channel drift produces silent accuracy loss.
Preproc is often the first root cause because it sits between real-world sensor variance and fixed model expectations. Fixing inference without fixing preproc is usually ineffective.

Card B — Minimal telemetry to prove preproc mismatch

  • Per-frame tensor stats: input_mean, input_std (overall or per-channel) to detect distribution drift.
  • Sensor variance bins: exposure_bin, gain_bin, wdr_mode to correlate accuracy drops with capture regimes.
  • ROI accounting: roi_crop_count and a coordinate tag (e.g., roi_coord_space) to confirm where ROI is applied.
  • Quant identity: quant_scale_id (or equivalent profile ID) to prove INT8 configuration consistency.
  • Optional canary proof: store a tiny downscaled debug thumbnail or a canary_hash (privacy-safe) to detect input pipeline changes.
Target question: “What tensor did the model actually receive?” Telemetry should answer this without needing raw high-resolution frames.
Pre-Processing as an Input Contract Stabilize tensors across exposure/WDR/IR variance without turning this into an ISP tuning tutorial Sensor variance exposure / gain WDR mode IR on/off noise / motion Preproc chain (must be consistent) Color space YUV ⇄ RGB Resize downscale Letterbox keep AR Normalize mean / std μ σ ROI crop coord space Quantize INT8 scale Telemetry tags input_mean/std · exposure_bin · quant_scale_id Optional proof canary_hash Stable tensor output (input contract)
Figure F4. Treat pre-processing as a measurable input contract: sensor variance enters the chain, telemetry tags prove tensor stability, and quantization identity prevents silent INT8 drift.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F4
ICNavigator | On-Camera / Edge Analytics | Fig F4 (Preproc Chain + Variance + Telemetry Tags)

H2-5. Post-Processing: Tracking, Attributes, and Event Logic Without False Alarms

Goal: turn raw detections into stable, explainable events—the part operators and customers actually feel—by tuning track lifecycle, rule hysteresis, and scene-aware thresholds.

Card A — Event stability knobs (what keeps alarms from “chattering”)

  • Track lifecycle: require confirmation before emitting events (min_hits), tolerate brief occlusion (max_lost), and bound association jumps (max_jump_px).
  • Association gates: tune matching strictness (assoc_iou_gate / distance gate) to reduce ID churn without merging distinct targets.
  • Rule hysteresis: separate trigger vs release conditions so boundary jitter does not spam alarms (e.g., enter threshold > exit threshold).
  • Debounce & hold: use time-based stability windows (debounce_ms, hold_ms) to prevent one-frame spikes from triggering.
  • Scene profiles: keep thresholds per class and per capture regime (scene_profile_id for day/night/WDR/IR) rather than a single global number.
Engineering intent: prefer bounded, explainable stability over “maximum sensitivity”. If alarms require operator trust, every trigger must be reproducible with a reason code and track evidence.
trigger hold release hysteresis debounce

Card B — False alarm triage fields (minimum evidence to debug fast)

  • Track metrics: track_id, age, lost_count, reid_count, zone_hits (prove continuity vs churn).
  • Alarm reasons: log rule state transitions: RULE_X_TRIGGER, RULE_X_HOLD, RULE_X_RELEASE.
  • Profile identity: include scene_profile_id and per-class thresholds in the event context so “night-only false alarms” are attributable.
  • Fast discriminators: frequent RULE_X_RELEASE with high lost_count suggests occlusion/visibility; high reid_count suggests association gates too strict.
The operator-facing question is “Why did it alarm?” A good pipeline answers with: track evidence (continuity) + rule state (trigger/hold/release) + profile identity.
Post-Processing: Stable Events (No False Alarms) Detections → Tracking → Rule Engine → Events, stabilized by hysteresis + debounce Detections boxes + conf class_id Tracker track_id + age lost_count / reid Rule Engine zones / dwell line-cross / loiter Events alarm + meta Alarm stability loop Hysteresis + debounce + hold/release states TRIGGER enter_cond met HOLD hold_ms RELEASE exit_cond stable debounce exit & timeout re-arm Reason codes: RULE_X_TRIGGER · RULE_X_HOLD · RULE_X_RELEASE
Figure F5. Post-processing converts detections into stable events using track continuity plus a rule-engine stability loop (hysteresis, debounce, hold, release) with explicit reason codes.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F5
ICNavigator | On-Camera / Edge Analytics | Fig F5 (Detections→Tracker→Rule Engine→Events + Hysteresis)

H2-6. ROI Coding and Sub-Streams: Bandwidth Savings Without Losing Evidence

Goal: generate ROI maps and sub-streams on-camera so bandwidth drops while evidence remains consistent—by aligning ROI to encoder grids and time-bases, and by logging measurable alignment checks.

Card A — ROI/sub-stream design patterns (what to build)

  • ROI coding from analytics: detections/tracks produce an ROI map (tile grid), which drives a foreground/background QP-map concept for the encoder.
  • Analytics sub-stream: a low-resolution, stable stream optimized for inference consistency (fixed cadence, low jitter, predictable bitrate).
  • Evidence burst: a short high-bitrate clip around an event (pre/post window) to preserve forensic detail without keeping the whole stream expensive.
  • Consistency contract: ROI map must align to the encode frame grid; sub-stream and main stream must remain timestamp-aligned.
ROI is only valuable if it is encoder-aligned. A visually plausible ROI overlay can still be ineffective if the ROI grid does not match the encoder’s tile/macroblock structure.
ROI map tile grid foreground QP background QP evidence burst

Card B — Measurable knobs & failure signatures (how to verify)

  • Encode telemetry (minimum): roi_map_id, roi_coverage%, avg_qp_fg/bg, substream_bitrate, idr_interval.
  • Alignment checks (must be explicit): roi_grid_mismatch_count, ts_skew_ms(main,sub).
  • Common failure signatures: bitrate does not drop (ROI too large), foreground detail still poor (QP not applied), evidence clip time mismatch (skew/jitter), ROI “swims” (grid mismatch).
  • Guardrails: cap roi_coverage%, enforce grid quantization, bound ts_skew_ms with hard checks and corrective resync.
Engineering intent: savings are secondary to correctness. If alignment checks fail, treat ROI/sub-stream as untrusted and fall back to safe defaults.
ROI Coding: Encoder-Aligned Tiles → QP Map Foreground tiles preserve evidence; background tiles suppress bitrate—only if grid/time alignment is verified Detections / Tracks boxes → tracks track_id + zone_hits ROI tile grid encoder-aligned roi_coverage% · roi_map_id Encoder QP map FG low QP / BG high QP avg_qp_fg/bg · idr_interval Alignment checks roi_grid_mismatch_count ts_skew_ms(main,sub)
Figure F6. ROI coding is an encoder-aligned tile map that drives a QP map: foreground tiles preserve evidence while background tiles suppress bitrate—verified by grid and timestamp alignment checks.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F6
ICNavigator | On-Camera / Edge Analytics | Fig F6 (ROI Tiles → Encoder QP Map + Alignment Checks)

H2-7. Privacy Masking: Redaction That Survives Scaling, PTZ, and Scene Changes

Goal: make privacy masking reliable across zoom/pan/warp and changing scenes, prevent drift or leaks, and enforce a fail-safe policy when detection or system health degrades.

Card A — Masking strategies & drift causes

  • Mask sources: detector boxes (faces/plates), fixed polygons (privacy zones), or hybrid (polygons as a safety net + detector-follow masks for moving targets).
  • Why drift happens: mask defined in the wrong coordinate space (tensor vs encode frame), zoom/warp changes the mapping, or detector jitter reshapes boxes frame-to-frame.
  • Stabilize first, then apply: track the mask target (track_id) and bound the update rate so masks do not “swim” during PTZ and motion bursts.
  • Transform chain (concept): Detect → Track → Transform (PTZ/warp profile) → Apply (pre-encode). Each step should be attributable to a profile ID rather than hidden behavior.
Engineering intent: masking must be pre-encode and coordinate-consistent. If any mapping is uncertain, enlarge the mask rather than risk exposure.
mask drift coord_space PTZ/warp pre-encode

Card B — Fail-safe policy + counters (never leak)

  • Fail-safe triggers: low confidence (mask_conf_min below threshold), unstable detections (mask_boxes_n collapsing), or system degradation (queue/thermal states).
  • Fail-safe actions: expand masks (mask_expand_px), enforce minimum mask coverage, and hold masks through short detector dropouts.
  • Mask telemetry (minimum): mask_boxes_n, mask_area%, mask_conf_min, mask_expand_px, mask_fail_safe_active.
  • Leak spot-check (concept): run randomized checks and count suspected exposures without storing raw frames; gate output if checks fail.
Treat masking as a safety system: when uncertain, bias conservative. A single unmasked frame can be an unacceptable failure, even if “average” quality looks fine.
Privacy Masking: Drift-Resistant Redaction Detect → Track → Transform → Apply (pre-encode) → Verify gate, with fail-safe override Detect face / plate Track track_id + hold Transform PTZ / warp profile Apply pre-encode mask Encoder main / sub streams Verify gate (anti-leak) Randomized spot-check counters + output gating (no raw frame retention) Telemetry: mask_boxes_n · mask_area% Counters: leak_spotcheck_count · leak_suspect_count Fail-safe masking If uncertain or degraded: expand masks + minimum policy Fields: mask_conf_min · mask_expand_px · mask_fail_safe_active override
Figure F7. Privacy masking pipeline with a verify gate and a fail-safe override path. Masking should be applied pre-encode and remain stable through PTZ/warp via track + transform chaining.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F7
ICNavigator | On-Camera / Edge Analytics | Fig F7 (Detect→Track→Transform→Apply(pre-encode)→Verify + Fail-safe)

H2-8. Metadata & Event Packaging: What to Output, How to Timestamp, How to Reconcile Streams

Goal: define an on-camera analytics payload contract that downstream systems can trust—clear payload types, unambiguous timestamps, and a deterministic way to reference the exact video segment on main and sub streams.

Card A — Payload contract (what to output)

  • Events: alarm + context; must reference video precisely. Minimum fields: event_id, zone_id, confidence, model_id, start_pts, end_pts, frame_id_ref.
  • Tracks: continuous objects for reconciliation and audit. Fields: track_id, class_id, first_pts, last_pts, age, lost_count, optional attributes.
  • Snapshots: key frames for fast review. Fields: snapshot_id, frame_id/pts, stream_id, optional crop boxes.
  • ROI maps: encode/control artifacts. Fields: roi_map_id, roi_coverage%, grid metadata (tile size/profile).
  • Mask descriptors: redaction geometry descriptors. Fields: mask_id, polygons/boxes, coord_space, transform_id, apply_stage=PRE_ENCODE.
Contract intent: downstream systems should never infer meaning from “missing fields”. If a payload cannot reference video reliably, mark it degraded and provide explicit flags.

Card B — Timestamp & alignment checklist (how to reconcile)

  • Use two time bases: monotonic device time for ordering + pts/frame_id for video referencing; add wallclock if available.
  • Reference the exact segment: events include start_pts/end_pts and a frame_id_ref that maps to both main and sub streams.
  • Skew checks: log ts_skew_ms(main,sub). If skew exceeds the bound, mark video reference degraded.
  • Light integrity (non-crypto): include sequence_no and payload_hash to detect truncation, drops, and reorder without defining signature schemes here.
Engineering intent: “Event references video” must be deterministic. A payload that cannot be reconciled to a clip is not forensic-grade.
event_id start_pts end_pts frame_id_ref sequence_no
Event → Video Binding (Main/Sub Streams) A payload contract that points to the exact clip using pts/frame_id, with skew checks and sequence continuity Event payload what downstream trusts event_id · zone_id · confidence · model_id start_pts · end_pts · frame_id_ref Reference resolver maps pts/frame_id to stream clips Checks: ts_skew_ms(main,sub) Main stream clip clip(start_pts, end_pts) stream_id=MAIN · pts range Sub stream clip clip(start_pts, end_pts) stream_id=SUB · pts range Light integrity (non-crypto) Detect drops/reorder/truncation without defining signing schemes here Fields: sequence_no · payload_hash · payload_len
Figure F8. A clean payload contract binds events to exact video segments using pts/frame_id_ref, reconciles main/sub streams via skew checks, and adds light continuity fields (sequence/hash).
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F8
ICNavigator | On-Camera / Edge Analytics | Fig F8 (Event→(frame_id,pts)→Main/Sub Clips + Skew + Sequence)

H2-9. Model Lifecycle on Camera: Profiles, Versioning, Rollback, and A/B Validation

Goal: prevent “updates made accuracy worse” and make field behavior explainable and reversible through auditable identities, profile switching rules, staged rollout, and rollback gates.

Card A — Lifecycle policy (safe and explainable)

  • Identity contract: treat “model” as a bound set (weights + quant + labels + preproc profile). Always publish model_id, model_hash, quant_profile, labels_ver, and pipeline_profile.
  • Scene profiles: define explicit profiles (day/night, IR on/off, indoor/outdoor) with auditable switching rules and hysteresis. A profile is a parameter pack (thresholds, trackers, mask policy levels), not an opaque “auto mode”.
  • Safe update flow: stage → warmup → validate → commit (or rollback). Validation can be time-sliced A/B (alternating windows) or shadow-run (compute-only, no event emission).
  • Rollback hierarchy: if behavior degrades, first attempt a conservative profile, then revert model/hash. Always store a last-known-good target and a reason code.
Engineering intent: a field update should be reversible in minutes and explainable from logs without subjective debates.
model_hash quant_profile labels_ver pipeline_profile A/B validation

Card B — What to log for blame-free postmortems

  • Update timeline: model_apply_start_ts, model_apply_end_ts, warmup_frames, active_model_id, active_model_hash.
  • Profile context: scene_profile_id, optional profile_switch_ts + profile_switch_reason (e.g., IR toggle, lux threshold, schedule).
  • Accuracy proxies (field-safe): event_rate, false_alarm_rate_proxy, optional confidence_histogram_id.
  • Rollback accountability: rollback_reason (e.g., PROXY_SPIKE, P95_LATENCY, THERMAL, PROFILE_FLAP, OPERATOR_FORCE), plus rollback_from_hash → rollback_to_hash.
Practical outcome: “accuracy worse after update” becomes a searchable diff (model/hash/profile/time window), not a guess.
Model Lifecycle: Stage → Validate → Commit / Rollback Auditable identity + proxy metrics gate + explicit rollback reason Old model model_id / model_hash Stage download · prepare · warmup warmup_frames Validate time-sliced A/B or shadow-run Metrics gate (proxy + health) Decide commit vs rollback based on field-safe signals event_rate · false_alarm_proxy p95_latency · thermal_state Commit set active + audit Rollback revert to LKG rollback_reason pass fail Audit fields (minimum) active_model_hash · scene_profile_id · model_apply_start/end · warmup_frames
Figure F9. A staged model update flow with an auditable metrics gate. Updates become explainable and reversible via identity binding, validation, and explicit rollback reasons.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F9
ICNavigator | On-Camera / Edge Analytics | Fig F9 (Old→Stage→Validate→Commit/Rollback + Metrics Gate)

H2-10. Performance Budgeting: End-to-End Latency, DDR Bandwidth, and Worst-Case Scenarios

Goal: provide a practical budgeting method for p50/p95 latency and bandwidth that survives worst-case scenes (crowds, rain, IR switching, PTZ motion) without guesswork.

Card A — Budgeting method (practical steps)

  • Define an SLO: pick a forensic-grade bound (e.g., event emission p95 ≤ target ms) rather than average latency.
  • Decompose latency: instrument stage timestamps for capture → preproc → infer → postproc → event emit → encode packaging.
  • Measure p50/p95 per stage: record stage_ms_p50/stage_ms_p95 and the end-to-end p95; treat queueing as a first-class signal.
  • Budget shared DDR: track total bandwidth and engine utilization; plan for codec bursts and ISP spikes during motion/noise.
  • Worst-case validation: run a scenario matrix (crowds, rain/snow, IR toggles, PTZ sweeps, night gain) and confirm the p95 bound holds or degrades safely.
Key principle: p95 breaks are usually queue/backpressure + contention, not a single slow compute stage.
p95 queue_depth ddr_bw_total thermal_state

Card B — If p95 breaks: which 2 counters first

  • Counter #1: max_queue_depth (or queue depth at the inference boundary) — proves whether latency is queueing-driven.
  • Counter #2: ddr_bw_total (or DDR utilization) — indicates contention from ISP/codec bursts versus compute starvation.
  • Fast discriminator: queue↑ + DDR↑ → contention; queue↑ + DDR flat → scheduling/locks; DDR↑ + codec_util↑ → bitrate bursts; p95↑ + thermal_state throttle → clock collapse.
  • Budget table fields (minimum): stage_ms_p50/p95, ddr_bw_total, codec_util%, npu_util%, max_queue_depth, thermal_state.
Debug intent: reduce guesswork. Two counters should tell whether to attack queueing, contention, or thermal throttling first.
Performance Budget: Latency Blocks + Shared DDR Hub Measure p50/p95 per stage, then confirm worst-case scenes do not break p95 bounds Capture p95_ms Preproc p95_ms Infer p95_ms Postproc p95_ms Event emit p95_ms Pack p95 Shared DDR hub Contention shows up as queueing and p95 spikes ddr_bw_total max_queue_depth ISP DMA bursts Codec bitrate spikes NPU infer_fps CPU/DSP postproc Worst-case triggers crowds · rain/snow · IR switching · PTZ motion · night gain effect: codec/ISP bursts + DDR contention + queue growth Failure signatures p95_latency↑ · max_queue_depth↑ · infer_fps↓ · thermal_state=THROTTLED first triage: max_queue_depth + ddr_bw_total
Figure F10. A practical budgeting view: latency is measured as stage blocks (p50/p95), while DDR contention is modeled as a shared hub connecting ISP/codec/NPU/CPU. Worst-case scenes often manifest as bandwidth bursts and queue-driven p95 spikes.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F10
ICNavigator | On-Camera / Edge Analytics | Fig F10 (Latency Blocks + Shared DDR Hub + Worst-case Triggers)

H2-11. Signing & Watermarking at the Camera: What to Sign, Where to Embed, How to Verify

Goal: lock camera outputs (events, snapshots, ROI maps, clip manifests) to device identity and timestamps, while embedding on-frame evidence watermarks—without expanding into full key-management or boot chains.

Card A — Minimal secure integrity model for camera outputs

  • Sign structured outputs (tamper detection + attribution): events/tracks, snapshots, ROI maps, and a clip manifest that lists hashes for frames/segments inside [start_pts…end_pts].
  • Bind to context (explainable behavior): include device_id, fw_ver, model_id, model_hash, scene_profile_id, and the reference frame_id/pts that the event points to.
  • Canonicalize before hashing: normalize field order and encoding so the same payload produces the same hash across firmware builds (PAYLOAD_FORMAT_ERR becomes diagnosable, not random).
  • Watermark video at a forced stage: apply watermark at a pre-encode compositor stage so every encode path passes through a single insertion point (prevents ENCODE_PATH_BYPASS leaks).
Practical rule: signatures protect “what the camera decided and emitted”; watermarks protect “what a viewer later sees and re-shares”.
clip_manifest payload_hash start_pts/end_pts pre-encode watermark

Card B — Verification hooks and measurable failure codes

  • What to verify (sample-based): (1) payload signature, (2) clip manifest signature + segment hash chain, (3) watermark detectability on sampled frames.
  • Counters: sign_ok/sign_fail, watermark_insert_ok/watermark_insert_fail, verify_sample_fail_rate (or verify_sample_ok/verify_sample_fail).
  • Failure taxonomy (camera-side, actionable): KEY_UNAVAILABLE (engine/SE not ready), PAYLOAD_FORMAT_ERR (canonicalization mismatch), ENCODE_PATH_BYPASS (watermark stage skipped), TIME_BINDING_ERR (event pts/frame reference inconsistent).
  • Fast triage: sign_fail↑ with KEY_UNAVAILABLE → check SE/crypto engine readiness; sign_fail↑ with FORMAT_ERR → compare canonical schema version; watermark_fail↑ or BYPASS↑ → enforce single compositor stage for all streams.
MPN examples (camera-side integrity building blocks)

Examples only (common in embedded security/video designs). Final selection depends on availability, crypto needs, interfaces, and certification targets.

  • Secure element / key storage (I²C/SPI): Microchip ATECC608B (ECC + secure key storage), Microchip ATECC608A, NXP EdgeLock SE050, STMicroelectronics STSAFE-A110, Maxim/Analog Devices DS28C36.
  • MCU/MPU with hardware crypto acceleration (SHA/AES/ECC engines for hashing/signing pipelines): ST STM32H743 / STM32H753, ST STM32U575, NXP i.MX RT1062, Renesas RA6M4.
  • RTC for stable timestamp reference (when wallclock is required in addition to monotonic time): Maxim/Analog Devices DS3231M, NXP PCF2129, Micro Crystal RV-3028-C7.
  • NVM for manifests/audit logs (append-friendly storage for hashes/records): Winbond W25Q128JV (QSPI NOR), Macronix MX25L128, Micron MT25QL128.
Signing & Watermarking Points (Camera-side) Bind outputs to device + timestamps; enforce a single insertion stage to prevent bypass Lane A — Payload integrity (structured outputs) Payload events · tracks · ROI · manifest Canonicalize schema_ver Hash payload_hash Sign sig Emit event payload + signature Bind to context: device_id · fw_ver · model_hash · scene_profile_id · frame_id/pts · sequence_no Lane B — Visual evidence (video watermark) Video frames main/sub/evidence Watermark apply pre-encode compositor Encode all paths Stream out watermarked pixels + metadata Failure: ENCODE_PATH_BYPASS (watermark skipped) Modes: per-event watermark (strong) · continuous low-rate watermark (sampling-friendly) Verify gate sample verify (payload + video) verify_sample_fail_rate sign_ok/fail wm_ok/fail Codes: KEY_UNAVAILABLE · PAYLOAD_FORMAT_ERR · TIME_BINDING_ERR
Figure F11. Camera-side integrity points: structured outputs are canonicalized, hashed, and signed; video frames are watermarked at a forced pre-encode stage. A verification gate samples both lanes and reports measurable fail rates and failure codes.
Cite this figure: Security & Surveillance › On-Camera / Edge Analytics — Figure F11
ICNavigator | On-Camera / Edge Analytics | Fig F11 (Payload Sign + Video Watermark + Verify Gate + Bypass Failure)
Implementation note (camera-side only): keep signature and watermark status as part of the emitted contract so downstream systems can reject or flag evidence when sign_fail or watermark_insert_fail exceeds a threshold.

Request a Quote

Accepted Formats

pdf, csv, xls, xlsx, zip

Attachment

Drag & drop files here or use the button below.

H2-12. FAQs (On-Camera / Edge Analytics)

Each answer stays on-camera, points back to measurable evidence fields, and maps to chapters above (no scope creep into VMS/NVR or full key management).

Analytics FPS drops at night—thermal throttle or DDR contention?

Short answer: Separate “heat throttling” from “memory contention” using paired counters, not guesses.

  • Evidence to check: infer_fps, p95_latency_ms, thermal_state/throttle_state, ddr_bw_total, max_queue_depth, codec_util%.
  • Discriminate: thermal_state=THROTTLED with broad utilization drop → thermal; rising ddr_bw_total + queue growth → DDR contention.
  • First fix: Cap worst-case bitrate (night NR/IR), enforce queue limits, and switch to adaptive inference with a deterministic drop policy (DROP_BW_LIMIT/DROP_QUEUE_FULL).
→ H2-3→ H2-10
Detections look fine but events are unstable—tracker or rule hysteresis?

Short answer: “Event instability” is usually a tracking lifecycle problem or a hysteresis/debounce problem in the rule engine.

  • Evidence to check: track_id, age, lost_count, reid_count, zone_hits, plus RULE_X_TRIGGER/RULE_X_HOLD/RULE_X_RELEASE.
  • Discriminate: High lost_count/reid_count → tracker; stable tracks but rapid TRIGGER/RELEASE → hysteresis too weak.
  • First fix: Add minimum dwell/hold time, debounce, and release hysteresis; then tighten “lost” handling (avoid immediate track reset).
→ H2-5
ROI coding saves bandwidth but faces become unrecognizable—ROI map design or QP bounds?

Short answer: The common failure is either ROI coverage misses faces/plates, or QP bounds crush foreground detail.

  • Evidence to check: roi_map_id, roi_coverage%, avg_qp_fg/bg, substream_bitrate, idr_interval.
  • Discriminate: Low roi_coverage% or frequent misses → ROI design; extreme fg QP or fg/bg gap → QP bounds.
  • First fix: Expand face/plate ROI tiles, set a foreground QP ceiling, and add a short “evidence burst” clip around events.
→ H2-6
Sub-stream timestamps don’t match main stream—what fields prove skew?

Short answer: Prove mismatch with explicit skew counters and shared frame references, then enforce a single timebase.

  • Evidence to check: ts_skew_ms(main,sub), frame_id_ref, start_pts/end_pts, sequence_no, per-stream idr_interval.
  • Discriminate: Large skew plus event referencing the “wrong” PTS window indicates broken reconciliation, not model accuracy issues.
  • First fix: Derive both streams from one PTS source, resync at IDR boundaries, and carry sequence_no end-to-end.
→ H2-6→ H2-8
Privacy masks drift during PTZ—tracking transform or coordinate mismatch?

Short answer: PTZ/warp changes coordinate space; mask drift is usually a transform-chain mismatch rather than detector quality.

  • Evidence to check: mask_boxes_n, mask_area%, mask_expand_px, (optional) ptz_state_id/warp_id to tag the active transform.
  • Discriminate: Stable tracks but drifting masks during PTZ implies transform/space mismatch; unstable boxes implies tracking.
  • First fix: Apply masks in the same stabilized coordinate space used for encode, and lock transform IDs per frame.
→ H2-7
Masks sometimes disappear under load—fail-safe policy or pipeline bypass?

Short answer: Under load, masks disappear either because fail-safe is not enforced, or because scheduling/paths bypass the masking stage.

  • Evidence to check: mask_fail_safe_active, mask_conf_min, queue_depth/max_queue_depth, drop codes (DROP_QUEUE_FULL/DROP_THERMAL), and any “bypass” counter.
  • Discriminate: Fail-safe inactive while confidence is low → policy issue; queue spikes + stage skipped → pipeline bypass.
  • First fix: Reserve budget for masking, enforce a minimum mask policy when degraded, and prevent stage skipping in degraded modes.
→ H2-7→ H2-3
Model update reduced accuracy—how to make rollback provable?

Short answer: Make updates reversible by logging “apply → validate → commit/rollback” with explicit model identity and gates.

  • Evidence to check: model_apply_start/end, warmup_frames, model_id, model_hash, labels_ver, quant_profile, rollback_reason.
  • Discriminate: A/B time-sliced run shows whether regression is model/profile or scene-profile switching.
  • First fix: Stage deployment, keep the previous model resident, and trigger rollback on proxy metrics (event-rate anomalies, false-alarm proxies).
→ H2-9
CRC/logs say OK but behavior changed—preproc normalization mismatch?

Short answer: CRC “OK” only proves the file is intact; field regressions often come from preproc and quantization profile drift.

  • Evidence to check: input_mean/std, exposure/gain bins, roi_crop_count, quant_scale_id, and a canary hash (thumbnail/tensor hash).
  • Discriminate: Same model hash but different input_mean/std or quant_scale_id indicates normalization mismatch.
  • First fix: Version the preproc profile, lock colorspace/normalize/letterbox parameters, and validate on canary frames before rollout.
→ H2-4
False alarms only in rain/snow—what worst-case knobs matter?

Short answer: Rain/snow creates worst-case motion noise and bitrate spikes; instability is often a mix of budget breakage and weak rule hysteresis.

  • Evidence to check: p95_latency_ms, ddr_bw_total, max_queue_depth, thermal_state, plus lost_count and RULE_X_HOLD/RELEASE.
  • Discriminate: If p95 and queues rise with false alarms, fix budget first; if tracks are stable, strengthen hysteresis.
  • First fix: Require multi-frame confirmation, tighten release hysteresis, and degrade inference deterministically under worst-case load.
→ H2-10→ H2-5
How do I bind an event to the exact video segment?

Short answer: Bind events to video using explicit frame references and a PTS window, and keep the binding stable across main/sub streams.

  • Evidence to check: event_id, frame_id_ref, start_pts/end_pts, stream_id, sequence_no, payload_hash.
  • Discriminate: A valid event must replay the same PTS window on both streams within measured skew bounds.
  • First fix: Emit a clip manifest per event window and ensure both streams share one monotonic timebase + PTS lineage.
→ H2-8
Should I sign video frames or just the event payload?

Short answer: Full-frame signing is expensive; the practical approach is signing payloads plus a clip manifest (chunk hashes), and using watermarks for visual redistribution checks.

  • Evidence to check: payload_hash, sequence_no, start_pts/end_pts, clip_manifest, sign_ok/sign_fail.
  • First fix: Sign payload + manifest on-camera using a secure key store (e.g., ATECC608B or SE050) and verify by sampling downstream.
→ H2-11→ H2-8
Watermark verification fails sporadically—where to look first?

Short answer: Sporadic failures usually mean path inconsistency (bypass) or worst-case resource collapse during spikes (DDR/thermal/queues).

  • Evidence to check: watermark_insert_ok/fail, verify_sample_fail_rate, failure codes (ENCODE_PATH_BYPASS, KEY_UNAVAILABLE), plus ddr_bw_total, max_queue_depth, thermal_state.
  • Discriminate: Fail with BYPASS → enforce pre-encode compositor; fail with queue/thermal spikes → budget headroom issue.
  • First fix: Force one insertion stage for all streams, reserve compute/memory headroom, and run a periodic self-test sampling hook.
→ H2-11→ H2-10
Tip for maintenance: keep the evidence field names stable over firmware versions; changing field semantics silently will break downstream triage and “blame-free” postmortems.
icnavigator

ICNavigator helps engineers find verified ICs, alternatives, and fast quotes.

Explore
Categories
  • Power Management
  • MCU
  • Automotive ICs
  • Sensors
  • Interface & Transceivers
  • Analog Front-End
Get in Touch
  • Email: hello@icnavigator.com
  • WeChat/WhatsApp: +86-xxxx

© 2025 ICNavigator. All rights reserved.