H2-1. Definition & System Boundary (What an NVR “owns”)

This chapter locks the scope: an NVR/edge recorder is a hardware system that ingests many IP video streams, keeps recording continuous under worst-case bursts, and preserves recoverable storage + crash-safe logs across reboots and brownouts.

• Reliability path: ingress → buffering/backpressure → durable writes → reboot recovery evidence.
• Throughput budgeting: GbE/10GbE receive capacity, DDR/bus contention, decode/transcode accelerators, disk tail latency.
• Power-loss behavior: hold-up sizing, brownout detection, “flush then stop” policy.
• Operational evidence: event logs, counters, health metrics that remain readable after crashes.

• Time sources: RTC/NTP mentioned as labels only (no timing architecture).
• Storage modes: RAID / filesystem journaling mentioned at decision level only.
• OS/software: treated as “exists” but not optimized here.

• VMS ingest & AI box: multi-stream inference scheduling, cross-camera analytics.
• Recording integrity & compliance: non-repudiation signatures, WORM, evidentiary audit chains.
• Video pipeline security deep dive: key ceremonies, TRNG internals, full secure-boot chain design.
• Cameras/optics: sensor/ISP/PTZ/illumination details.
• PoE switch/PSE: allocation/metering/thermal design in switch infrastructure.

Record-only systems are dominated by I/O integrity: NIC receive behavior, buffering, and disk write tail latency. Record + decode/transcode adds hard constraints: accelerator concurrency, shared DDR bandwidth, and thermal throttling, where export/preview workloads can steal budget from recording unless backpressure policy is explicit.

H2-2. Workload Model: Streams, Bitrate, and “Worst-Case” Math

The goal is an auditable model: the recorder either proves margin (no gaps) under worst-case bursts and storage tail latency, or it fails with a measurable counter. Average bitrate is not an acceptance criterion.

• N = number of concurrent streams.
• B_i = per-stream bitrate distribution (use P95/P99 or peak window, not average).
• C_i = codec/profile mix (drives decode/transcode complexity and memory bandwidth).
• M = mode flags: record-only, preview decode, multiview, export transcode.
• R = retention target (days) and storage policy (how much overwrite pressure exists).

• Record-only: bottleneck typically becomes disk tail latency or NIC bursts.
• Decode-for-preview: adds decode engine concurrency + DDR bandwidth sharing risk.
• Transcode-export: adds encode engines + thermal throttling and can starve recording without explicit priority/backpressure.

• Use a peak window: model ΣB over 1–5 minute peaks (events, motion bursts), not daily averages.
• Include tail latency: storage is judged by P95/P99 write latency and queue depth, not “MB/s”.
• Consider concurrency: preview/transcode + recording + disk background behavior (GC / remap / SMR) may overlap.
• Margin is policy-driven: define what can degrade first (preview frames) while recording stays continuous.

• Disk write budget: BW_write ≈ ΣB_peak × (1 + container/fs overhead). Keep headroom for tail latency (policy-defined).
• NIC budget: track PPS and RX drops; if PPS rises (small packets), interrupt/DMA pressure can break “MB/s” assumptions.
• Compute budget: if preview/transcode exists, require measurable decode/encode utilization and fallback counters (no silent software decode).
• DDR/shared-bus budget: if DDR contention appears, recording loss often shows up as buffer watermark hits first.

• Soak test: at configured N and ΣB_peak, continuous recording for T hours with zero timeline gaps (or below a strict, declared threshold).
• Interference test: preview/transcode enabled: recording remains continuous and any degradation is visible in drop-reason counters (not silent failure).
• Tail-latency test: under worst observed write latency (P99), buffer watermarks must stay below “panic” and drops must be attributable to a specific subsystem counter.

H2-3. Network Ingest Front-End (GbE/10GbE) & Jitter Buffering

Multi-stream recording fails when bursts and jitter are not converted into stable per-stream consumption. This chapter treats ingest as a hardware queueing problem (NIC/PHY/DMA/buffers), not a protocol tutorial.

• PHY → MAC: physical link quality and framing; errors here are wire-side, not “software”.
• RX queues / RSS: distribute incoming traffic to reduce single-queue overflow under bursts.
• DMA rings: descriptor availability is finite; ring starvation creates hard drops.
• Jitter buffer: absorbs burstiness/reordering and normalizes delivery timing.
• Stream demux → per-stream queues: isolates streams so one problematic source does not collapse the whole recorder.

• Wire-side loss: PHY link errors, CRC/FCS errors, unstable cabling/SFP issues (prove with RX errors).
• NIC-side drops: RX ring overflow, descriptor starvation, IRQ storm, queue imbalance (prove with ring drops).
• Reorder / late arrival: packets arrive but too late for the target playback/record window (prove with late counters).
• Consumer-side overload: downstream contention (SoC/PCIe/DDR) causes per-stream queues to hit watermarks.

H2-4. Decode / Transcode Architecture (ASIC vs SoC vs GPU) — What Bottlenecks First

Decode/transcode success is determined by shared bandwidth, concurrency, and thermals. Under many streams, “compute present” is not enough—DDR/NoC contention and hidden fallback are common failure paths.

• Decode engines: fixed-function or ASIC blocks; concurrency is finite and profile-dependent.
• Scaler / compositor: multiview and preview paths add memory traffic even without transcoding.
• Encode engines: export or re-stream workloads multiply DDR reads/writes.
• Shared DDR / NoC: the usual first bottleneck under concurrency; congestion shows up as queue watermark hits.
• Thermal limits: throttling can reduce sustained engine throughput without obvious “errors”.

• Decode-only (preview): adds decode + display path; failure often appears as late frames when DDR is tight.
• Transcode (export): adds decode + encode; DDR traffic increases sharply and thermals become first-order.
• Recording-first rule: if export/preview competes with recording, policy must degrade preview/export first.

• Engine utilization can look acceptable while DDR/NoC congestion increases.
• Congestion triggers buffer watermark hits → frame drops with specific reason codes.
• Some codec profiles trigger fallback (software decode/encode) causing CPU/power spikes and new drop patterns.

• Cap concurrency: define maximum simultaneous decode/transcode sessions and log when limits trigger.
• Protect recording: prioritize write pipeline; allow preview/export to drop first with explicit counters.
• Detect fallback: require a visible counter for HW→SW transitions; treat any fallback as a capacity reduction event.
• Thermal margin: validate sustained performance at worst ambient; throttling must be logged as a cause, not a mystery.

H2-5. Memory, Buffering & Backpressure (Why “Dropped Frames” Happen)

Dropped frames are rarely “random.” They are usually a visible sequence: a specific buffer reaches a watermark, backpressure propagates upstream, and a defined drop/degrade policy triggers. The engineering goal is to make every loss traceable to a buffer level + a reason code.

• Rings (ingest): NIC RX queues / DMA rings absorb micro-bursts until the CPU/SoC can service descriptors.
• Queues (isolation): jitter buffer and per-stream queues normalize timing and keep one stream from collapsing all streams.
• Work queues (compute): decode/compose queues represent shared DDR/NoC pressure under concurrency.
• Coalescing buffers (storage): write buffers batch small writes and hide short write stalls—until tail latency breaks the budget.

• If NIC ring watermarks hit first: ingress servicing is behind (queue imbalance / IRQ pressure / DMA descriptor starvation).
• If jitter buffer stays high (or oscillates) while rings are clean: reorder/late-arrival dominates; buffer sizing or source burstiness is the driver.
• If decode/work queues climb first: compute or shared DDR/NoC congestion is starving consumers (often before “engine utilization” looks maxed).
• If write buffer climbs first: storage tail latency is forcing backpressure (H2-6 proves it with P95/P99 + queue depth).

• Degrade non-critical paths first: pause/limit multi-view preview, reduce preview FPS, or cap simultaneous export/transcode.
• Prefer controlled drops over silent corruption: drop events must be explicit, counted, and timestamped.
• Per-stream isolation: throttle or deprioritize a misbehaving stream instead of letting global queues saturate.
• Keep policies observable: any degrade/drop action must increment a policy counter with a reason code.

H2-6. Storage Subsystem: SATA / NVMe, SSD/HDD Mix, and Tail Latency

Recording continuity is limited by tail latency and error recovery, not peak throughput. A “fast” drive can still break video continuity if P99 write latency exceeds the buffer budget and forces backpressure (H2-5).

• Latency distribution: P95/P99 write latency determines how large the write buffer must be to avoid stalls.
• Queueing behavior: queue depth and concurrency define how the system behaves under many streams.
• Error recovery: retries and media remapping turn into long stalls that look like “random gaps” unless logged.
• Power-loss behavior: unsafe shutdown events are a reliability signal; PLP presence changes the risk profile.

• SATA: simpler attachment and predictable behavior, but concurrency and queueing can become the limiter under many streams.
• NVMe: higher parallelism and deeper queues, but bottlenecks can migrate to PCIe/SoC/DDR contention under heavy mixed workloads.
• Design implication: storage choice must be validated with the real stream mix and the recorder’s coalescing strategy, not datasheets alone.

• HDD stalls: retries, background reallocation, and write-cache behavior can create latency spikes that propagate to write buffers.
• SSD stalls: garbage collection, cache exhaustion, and thermal derating often show up as rising P99 latency over time.
• Mixed media: a single worst tail-latency member can dominate the array/volume behavior unless isolation is designed in.

• Pros: capacity scaling and fault tolerance.
• Cons: write amplification and rebuild modes that can dramatically change tail latency; rebuild must be treated as a validation scenario.
• Rule: any degraded/rebuild state must be reflected in logs as an explicit mode so continuity impact is explainable.

H2-7. Power Tree, Hold-Up, and “Graceful Stop” Under Brownout

Hold-up is not “staying on.” It is a time-budgeted transaction: after power-fail detection, the recorder must stop new work, flush critical buffers, and commit index + metadata + event logs before rails collapse. Every shutdown must be explainable with a timestamped sequence.

• Recording index / segment table: last segment boundary, offsets, and continuity markers.
• Write journal / allocation state: minimum state required to avoid “phantom” corruption after reboot.
• Power-loss event snapshot: event code + counters (buffer watermarks, ring drops, storage latency).

• Storage domain: NVMe/SATA controller + drive power is the first dependency for real persistence.
• DDR / memory domain: if commit relies on in-memory buffers/journals, DDR must outlive the commit window.
• SoC core + interconnect: the minimal compute domain that runs the flush/commit state machine.
• Power-fail detect / supervisor: provides early warning and deterministic thresholds for mode transitions.
• NIC is not highest priority: during brownout the correct action is stop ingest, not continue intake.

• T0 — Detect: power-fail event asserted (PGOOD/comparator/ADC) and timestamped.
• T1 — Enter flush mode: freeze non-critical tasks; elevate commit path priority.
• T2 — Stop ingest: stop accepting new segments/streams; prevent buffer growth.
• T3 — Commit window: commit index + metadata + log snapshot; request final storage flush.
• T4 — Cut: when voltage/time budget is exceeded, the system must stop safely (no silent partial writes).

H2-8. Event Logging & Health Metrics (Crash-Safe, Power-Loss-Aware)

Without structured, crash-safe logs, every outage becomes guesswork. The recorder must produce machine-readable events that survive crashes and power-loss, enabling post-mortem reconstruction of buffer, storage, thermal, and reset conditions—without relying on external platforms.

• event_code: stable identifier for the condition (power-fail, watermark hit, disk stall, watchdog).
• subsystem: power / NIC / storage / compute / thermal / watchdog.
• timestamp_source: RTC or monotonic counter (record which one).
• counter_snapshot: key counters at the moment (ring drops, queue depth, P99 latency, throttles).
• severity (optional): info / warn / error to aid filtering and triage.

• Append-only: logs are appended, not overwritten, reducing corruption blast radius.
• Segmented commits: log segments close with a CRC; power-loss may drop the last partial segment only.
• Self-check: on boot, verify CRC and report results as health events (pass/fail counts).

• Must record: power-fail detect → counters snapshot → flush mode entered → ingest stopped → commit success/fail.
• Missing steps are failures: absence of “flush entered” or “commit complete” is an explicit diagnostic signal.
• Keep it local: this chapter focuses on reliable recording and export, not on compliance signatures or non-repudiation.

• Thermal: max/avg temperatures, throttle flags, fan tach fault counters.
• Storage: P95/P99 write latency trend, queue depth, SMART error markers, unsafe shutdown counts.
• Network: RX errors, ring drops, reorder/late packet counts.
• Reset path: reboot reason register, watchdog reset counts, brownout counters.

H2-9. Minimal Security Posture for NVR Hardware (No Deep Dive)

An NVR does not need an enterprise security treatise to be credible. It needs a minimum hardware security posture that is observable, logged, and hard to bypass accidentally: verified boot presence, bounded credential/key storage, and physical tamper events captured as structured logs. Deep implementation details belong on the dedicated pages for video pipeline security and recording integrity.

• In scope: minimum “have/measure/log” posture for boot, keys/credentials, and tamper.
• Out of scope: secure boot chain internals, key ladder details, stream encryption design, non-repudiation signatures, WORM storage policies.
• Bridge: this chapter only defines observable flags and event log contracts for auditability.

• Presence: a verified/secure boot step must exist before the recorder enters normal operation.
• Deterministic failure: verification failure must lead to a bounded behavior (blocked boot or explicit recovery mode).
• Evidence: expose a secure boot status flag and a firmware build identifier for logging and support triage.

• Version attestation-lite: export a minimal statement such as verified boot = yes/no, firmware version, build ID/hash ID.
• Update events: record update success/fail and rollback occurrence as structured events (no platform details required).
• No silent behavior changes: changes to recording behavior must correlate to a logged firmware/config event.

• Key boundary: if disk encryption or protected credentials are used, keys must live in a controlled boundary (SoC key store / secure element / TPM-class storage).
• Separation: avoid placing long-term keys inside user-accessible plain storage where they can be copied with the disk.
• Evidence: expose an encryption enabled flag and a credential storage mode indicator for logs and factory QA.

• Chassis open: record a tamper event when enclosure switch triggers (or when mechanical seal state changes).
• RTC/battery removal: record a specific event when RTC resets or backup supply is removed; capture timestamp source change.
• Link to H2-8: tamper signals are only useful if they enter the structured event log with an event code and counter snapshot.

H2-10. Validation Plan (Throughput, Power Cycling, Thermal, Fault Injection)

Validation must be repeatable and measurable. The goal is not “it usually works,” but “under defined worst-case workloads and injected faults, every drop has a reason code, every recovery time is bounded, and power-loss sequences produce complete logs (detect → flush → stop ingest → commit).”

• Workload profile: stream count, codec profiles, aggregate bitrate, and burst patterns.
• Modes: record-only, preview decode, export transcode (if supported) under the same baseline.
• Telemetry enabled: buffer watermarks, ring drops, storage latency percentiles, throttles, reset reasons.

• Procedure: sustained recording at worst-case aggregate bitrate; include periodic bursts (scene changes / I-frame spikes).
• Observe: P95/P99 write latency time series, queue depth, watermark hits, drop counters and reason codes.
• Pass/Fail: no silent failures; drops (if any) must be explainable by logged reason codes and bounded in rate.

• Procedure: random power cuts at different phases; include short interruptions and controlled voltage dips.
• Observe: detect→commit (ms), unsafe shutdown count, journal replay count, index continuity markers.
• Pass/Fail: complete event sequence exists for each event; recovery time to recording is bounded and recorded.

• Procedure: raise ambient or restrict airflow until throttling occurs; repeat under steady and burst workloads.
• Observe: throttle flags, temperature maxima, fan faults, P99 latency drift, drop reason codes.
• Pass/Fail: degradation is policy-driven and logged (no unexplained discontinuities).

• Network: short link drops, packet loss/reorder bursts → observe jitter occupancy, late packets, recovery time.
• Storage: simulated drive stalls or single-drive offline events → observe write latency spikes, error events, fallback mode.
• Compute: induce contention or downclock events → observe decode queue watermarks, fallback counters (if available).

H2-11. Field Debug Playbook (Symptom → Evidence → Isolate → First Fix)

This playbook is designed to be usable on a bench or in the field. Each symptom starts with two fastest measurements, uses a single discriminator to collapse root-cause into one domain, and ends with a first fix that is minimally invasive. Every step maps back to the evidence contracts established earlier (ingest counters, buffer watermarks, tail latency, hold-up timeline, and event logs).

• Lock the workload ID first: record-only vs preview decode vs export transcode (maps to H2-2).
• Only trust observable evidence: counters, latency percentiles, watermarks, reboot reasons, and power-loss sequence events (maps to H2-3/H2-5/H2-6/H2-7/H2-8).
• Collapse to one domain: NIC/ingest, buffers/DDR, decode/compute, disk/storage, power/logging.
• First fix must be reversible: policy/threshold/buffering/priority before architecture changes (maps to H2-10).

The part numbers below are example MPNs commonly used for NVR-class designs. Always verify availability, speed grade, and platform support. The intent is to give a concrete BOM anchor for each “first fix” direction without diving into deep security or camera-ISP domains.