Radar-Camera Sensor Fusion: Time Sync, TSN, Calibration
← Back to: Security & Surveillance
Radar-Camera Sensor Fusion works only when time sync, latency determinism, and calibration traceability are all controlled at the node. This page turns “fusion feels wrong” into a measurable evidence chain—offset/jitter, end-to-end latency, and cal/version tags—so issues can be isolated and fixed without scope creep.
H2-1. Definition & Boundary: What “Radar-Camera Fusion” Owns
Intent: lock the page boundary upfront so every later section stays vertical (no scope creep).
Definition (one sentence): A radar-camera fusion node aligns mmWave radar observations and camera frames into a shared time base, spatial frame, and object identity, then outputs fused tracks/ROI/events with traceable timestamps suitable for validation and field debugging.
This page OWNS (deep coverage)
- Time alignment: PTP/1588 (node-side), hardware timestamps, sync state, and how offset/jitter maps to fusion errors.
- Space alignment: extrinsic calibration and how to prove/measure drift via residuals and cross-sensor consistency.
- Object alignment: projection + gating + association + tracking stability, with evidence that explains mis-associations.
This page REFERENCES ONLY (no deep dives)
- PoE / power / surge: only as “what to measure” when sync/frames glitch; no PSE/switch topology.
- VMS/NVR/cloud: only “what streams/metadata are exported”; no ingest throughput or storage design.
- Single-sensor tuning: radar TRx or camera ISP internals only to define outputs and timing; no image-quality tuning.
This page BANS (out of scope)
- PoE switch/PSE architecture, grandmaster timing-network planning.
- NVR/VMS platform sizing, recording/WORM/compliance chains.
- PTZ mechanics, thermal ROIC/TIA, starlight/ANPR ISP tuning, perimeter intrusion radar product design.
Typical outputs (engineer-facing)
- Must-have: fused object list (position/velocity/confidence/track_id) + per-object timestamp +
sync_state. - Nice-to-have: ROI/event metadata (tripwire/loiter/zone), association confidence.
- Optional raw: radar cube / point cloud / video stream for offline correlation and failure replay.
- sync_error (ns): offset/jitter statistics at the fusion-critical timestamp domain (use percentiles, not only averages).
- end-to-end_latency (ms): from capture timestamp to network egress timestamp (again: P50/P99/P99.9).
- timestamp_source: enum HW / PTP / SW, attached to logs/packets so downstream debug can trust timing provenance.
H2-2. System Architecture & Dataflow: From Raw Sensors to Fused Tracks
Intent: define the full end-to-end dataflow and a single “timestamp language” so every later metric can map back to one tap point.
End-to-end dataflow (only what fusion needs)
- Radar path: ADC samples → range/Doppler/angle processing → radar cube / point cloud / tracks (choose output based on bandwidth and debug needs).
- Camera path: image sensor → ISP pipeline → frames + metadata (exposure, gain, frame_id, sensor timestamp).
- Fusion path: time align → projection → association → tracking → object list / ROI / events (each carries timestamp provenance).
Three mandatory timestamp tap points
- t_capture: closest to sampling (prefer HW/PTP domain). One per radar frame/chirp group and one per camera frame.
- t_fuse: at fusion boundary (recommend both start and end to separate compute time vs queueing).
- t_tx: at network egress (distinguish SW enqueue vs HW egress timestamp when available).
What metadata must exist for correlation
- Stable IDs:
frame_id(camera),radar_frame_id(radar), and a log-levelsync_epoch_idafter every re-sync. - Provenance:
timestamp_source(HW/PTP/SW),sync_state(locked/holdover/free-run),cal_version. - Debug counters: queue depth, drop counters, processing time per stage (used later in the field playbook).
| Tap point | Where it lives | Preferred source | Minimum fields to log | Used to prove |
|---|---|---|---|---|
| t_capture | Radar front-end / camera sensor domain | HW timestamp in PTP-disciplined clock domain | frame_id, t_capture, timestamp_source, exposure/gain (camera), radar_frame_id |
Sampling time truth; separates sensor timing from compute/network artifacts |
| t_fuse | Fusion boundary (before/after association & tracking) | PTP time read + deterministic scheduling | t_fuse_start, t_fuse_end, stage runtimes, gating/association counters |
Compute latency vs queueing latency; identifies CPU contention or buffer growth |
| t_tx | NIC queue / PHY egress | HW egress timestamp when supported; else SW enqueue timestamp | t_tx, queue latency, drop counters, traffic class (PTP/control/video) |
Network determinism; exposes TSN/QoS misconfiguration and long-tail queue spikes |
H2-3. Time Sync & Clock Tree: PTP/1588 + HW Timestamping (Node-Side)
Intent: the deepest chapter of this page. Explain time sync only from the node perspective (no grandmaster/network-wide design).
Fusion accuracy depends on a single, traceable time base. The key requirement is not “PTP enabled” but where timestamps are created (HW vs SW) and how clock domains remain consistent across radar, camera, compute, and Ethernet.
Node clock tree (what must exist)
- Reference: XO/TCXO/OCXO provides the local base stability (holdover behavior).
- Jitter-clean PLL: cleans short-term phase noise before feeding sensitive domains (radar sampling, ISP, PHY ref).
- Distribution: one disciplined reference is fan-out to Radar, Camera/ISP, Compute, and Ethernet PHY.
- Domain labeling: every timestamp must declare
timestamp_sourceandsync_state.
Why SW timestamps cause fusion drift
- Scheduling jitter: user-space timestamping moves with interrupt load and run-queue contention.
- Buffer uncertainty: frames/packets sit in queues before the application sees them; SW time measures “arrival to app”, not “sampling”.
- Long-tail effect: P99.9 spikes break association gates even if average latency is fine.
Evidence deliverables (must be recordable)
- Measurement 1:
ptp_offset(ns) — report P50/P99/P99.9 and detect step changes. - Measurement 2:
ptp_jitter(ns) — windowed deviation or P99.9 of offset differences. - Decision: offset/jitter long-tail ↔
track_id_switch/ “jump” events correlate in time.
| Timestamp | Where it should be generated | Preferred source | What it proves | Common failure signature |
|---|---|---|---|---|
| t_capture | Radar sampling edge / camera sensor exposure start/end (closest possible) | HW (PTP-disciplined) | Sampling time truth across sensors | Drift between radar vs camera timestamps after load/temperature changes |
| t_tx | MAC/PHY egress (or NIC HW timestamp when available) | HW (MAC/PHY) | Network determinism independent of application delay | Offset stable but egress time has long-tail spikes (queue/QoS issue) |
| SW time | Application receive/processing time | SW only (diagnostic) | Compute scheduling / backlog visibility | “Looks fine on average” but association fails under burst load (P99.9 hidden) |
timestamp_source, sync_state).
H2-4. Latency & Jitter Budget: What Must Be Deterministic for Fusion
Intent: convert “feels inaccurate” into an auditable budget. Every stage gets a P99.9 latency/jitter allowance and a visible failure mode when exceeded.
Fusion fails on long-tail timing spikes. Budgets must be expressed in P99.9 (or worst-in-window), not averages. A deterministic pipeline is achieved by bounded queues, fixed processing depth, and controlled scheduling.
Budget method (usable template)
- Split end-to-end into: Capture → Process → Fuse → Tx.
- Measure each segment with H2-2 tap points: t_capture, t_fuse, t_tx.
- Report P50/P99/P99.9 and track step changes (load, temperature, mode switches).
- Attach one failure mode per segment (association fail, ID switch, ghost targets).
Evidence deliverables (must be recordable)
- Counter 1:
frame_queue_depth(camera/ISP output queue) - Counter 2:
radar_processing_time(per radar frame / batch) - Decision: jitter over budget →
association_fail_rateandtrack_id_switchrise
Stages that must be deterministic (and why)
- Frame buffers: variable queue depth creates variable alignment error (time-align becomes wrong even if offset is fine).
- ISP pipeline depth: mode-dependent pipelines change latency unless locked to a fixed configuration.
- Radar chirp/processing window: missed windows create bursty outputs and “lumpy” track updates.
- Network egress queue: uncontrolled queueing inflates t_fuse→t_tx long-tail, masking true capture time.
| Budget segment | Measured by | Primary counters | What to watch (P99.9) | Failure mode when exceeded |
|---|---|---|---|---|
| Capture→ISP/Radar output | t_capture + stage timestamps | frame_queue_depth, sensor drop counters |
Queue depth growth, bursty frame delivery | Time-align uses stale frames → projection residual spikes |
| Radar output→Fusion start | t_capture→t_fuse_start | radar_processing_time, radar backlog |
Processing time long-tail, missed windows | Track update jitter → association gating mismatch |
| Fusion compute | t_fuse_start→t_fuse_end | stage runtimes, CPU contention indicators | Compute spikes, cache/memory contention | Association fails under load; track_id churn |
| Fusion→Tx egress | t_fuse_end→t_tx | NIC queue latency, drop counters | Egress queue long-tail, traffic-class starvation | Events arrive late/out-of-order; false alarms or missed triggers |
H2-5. Calibration Stack: Intrinsic, Extrinsic, and Time-Offset Calibration
Intent: separate calibration into three layers, then lock what must be stored and how records remain traceable (versioned, CRC-checked, rollback-guarded).
Fusion quality depends on three independent calibration layers: camera geometry (intrinsic), cross-sensor pose (extrinsic), and cross-sensor time alignment (time-offset). A stable system requires two things: (1) measurable residuals and (2) calibration records that cannot be silently overwritten.
Layer 1 — Intrinsic (camera model)
- What it owns: lens distortion + projection geometry that controls pixel-domain projection error.
- Typical symptom: residual grows toward image edges; ROI appears consistently “bowed” or shifted by FOV region.
- What to measure:
reprojection_residual_px(report P50/P95/P99) by image region (center vs edge).
Layer 2 — Extrinsic (radar ↔ camera pose)
- What it owns: rigid transform between radar frame and camera frame (mounting angle/position).
- Installation tolerance link: small bracket rotations can create large pixel errors at long range; vibration can shift extrinsics over time.
- Typical symptom: residual is globally biased (consistent shift/rotation), often after re-mounting or mechanical shock.
rig_id/sensor_id so a valid record cannot be applied to the wrong mechanical assembly.
Layer 3 — Time-offset (capture alignment)
- What it owns: fixed capture offset between radar and camera, plus any temperature/load dependent drift that behaves like offset.
- Why it matters: fast movers show systematic projection mismatch when captures are not time-aligned even if geometry is correct.
- Typical symptom: mismatch scales with target speed; errors correlate with
sync_statechanges or temperature transitions.
Calibration record contract (minimal, auditable)
- Identity:
sensor_id,rig_id,cal_timestamp,timestamp_source - Versioning:
cal_version(monotonic),active_slot(A/B),rollback_guard - Integrity:
crc32over payload, reject-and-keep-old on CRC fail - Payload: intrinsic parameters hash, extrinsic parameters,
time_offset_ns
Evidence deliverables (must be recordable)
- Field 1:
cal_version— included in every fused output/event for traceability. - Field 2:
reprojection_residual_px— track P95/P99 trends; segment by image region when possible. - Field decision: residual ↑ vs sync error ↑:
- If
sync_error_ns(or offset/jitter) spikes and residual rises in the same window → time alignment is the primary suspect. - If sync metrics are stable but residual drifts after re-mount/shock and shows global bias → extrinsic is the primary suspect.
- If
H2-6. Sensor Alignment & Association: Making Two Modalities Agree
Intent: explain object-level fusion mechanisms with hardware/time constraints and explainable gating, without turning into a paper-style algorithm survey.
Object-level fusion requires two steps that must stay auditable: (1) project radar measurements into the camera view using calibrated geometry,
then (2) associate candidates using explainable gates (position/velocity/confidence) that remain stable under the node’s latency/jitter budget.
The health of association is tracked by two metrics: association_fail_rate and track_id_switch_rate.
Coordinate chain (hardware-friendly)
- Radar measurement: range/angle (and optionally Doppler) expressed in radar frame.
- Extrinsic transform: map radar frame → camera frame using the rig pose record.
- Intrinsic projection: camera frame → pixel plane; derive expected ROI and uncertainty envelope.
- Time alignment: apply time-offset (or compensate using sync metrics) before comparing fast movers.
Explainable association gates (minimal set)
- Position gate: projected radar point/track falls inside a pixel window (ROI + margin).
- Velocity gate: consistency between radar radial velocity and camera-derived motion direction/magnitude (coarse is enough).
- Confidence gate: reject low-quality candidates (radar SNR/quality + camera detection confidence) to avoid ID churn.
Evidence deliverables (must be recordable)
- Metric 1:
association_fail_rate— percentage of radar tracks with no valid camera match (per window). - Metric 2:
track_id_switch_rate— frequency of identity churn for matched tracks. - Field decision guide:
- Fail rate high, ID switches low → gates too strict or projection residual is high (calibration suspect).
- ID switches high → timing jitter spikes or ambiguous multi-target gating (time-offset/jitter suspect).
- Both high and synchronized with
ptp_jitteror queue depth spikes → return to sync/budget chapters first.
H2-7. Ethernet/TSN Design (Node Perspective): QoS, VLAN, Traffic Shaping
Intent: keep the fusion node’s egress predictable and auditable (classification → queues → shaping), without covering switch selection or network-wide TSN planning.
Fusion stability depends on whether the node can protect time (PTP), control outputs (object list/alarms),
and traceability (metadata/logs) from bandwidth-heavy video bursts. The goal is a node-side contract:
traffic classes are explicit, queues are deliberate, and determinism is validated using tx_queue_latency and packet_drop_counter.
Traffic class contract (node-side)
| Class | Examples | Time sensitivity | Drop tolerance | Recommended queue intent |
|---|---|---|---|---|
| PTP | Sync/Delay messages, ToD/1PPS related packets | Highest — jitter directly converts to fusion misalignment | Near-zero | Strict priority / isolated queue; minimal shaping |
| Control | Object list, alarms, door/zone triggers, PTZ control feedback | High — affects closed-loop response and event timing | Low | High-priority queue; protected from video bursts |
| Metadata | Calibration/version tags, health counters, audit/event logs | Medium — must remain traceable for forensics | Low–medium | Mid-priority queue; stable delivery preferred |
| Video | Main/sub streams, snapshots, replay bursts | Lower — latency can vary (within acceptable UX) | Medium (sub stream can tolerate more) | Low-priority queue; shaped/policed to protect others |
QoS queues (node perspective)
- Classifier: map each traffic class into a dedicated queue (at least 4 queues recommended).
- PTP first: strict priority or reserved scheduling slot to minimize queueing jitter.
- Control before video: protect object list/alarms from video bursts (especially during scene changes / I-frames).
- Metadata preserved: avoid silent drops that break post-incident reconstruction.
tx_queue_latency long-tail up for PTP/control, association failures and track jumps increase.
TSN hooks (purpose + verification points)
- 802.1Qbv (time-aware shaping): reserve deterministic egress windows for PTP/control when video is bursty.
- Verify Qbv: P99.9
tx_queue_latencyfor PTP/control drops during scheduled windows; video is smoothed outside windows. - 802.1Qci (per-stream policing/filtering): prevent abnormal/misbehaving video traffic from exhausting buffers.
- Verify Qci: policing counters increment during anomalies while PTP/control counters remain stable (no collateral delay/drops).
Evidence deliverables (must be measurable on the node)
- Measurement 1:
tx_queue_latency— tracked per class (at least PTP vs Control vs Video), with P99/P99.9 emphasis. - Measurement 2:
packet_drop_counter— tracked per class; video drops should occur before control/PTP drops. - Field decision: if long-tail queue latency rises with sync jitter/offset, prioritize QoS/shaping correctness before tuning fusion logic.
H2-8. Hardware Building Blocks & Example BOM (MPNs Optional)
Intent: provide a practical module-level BOM map and selection checks, without stealing deep-dive content from sibling pages.
This chapter is a system build checklist: each block is mapped to device categories and evaluated against three binding criteria:
timestamp capability, deterministic latency support, and thermal headroom.
The purpose is not to list parts, but to ensure each module can support time/latency determinism and sustained operation.
mmWave Radar block (node-side view)
- Device categories: TRx front-end, radar AFE/ADC, radar processor/MCU, reference clock input.
- Timestamp capability: exposes frame/chirp timing markers or supports capture time tagging at the node boundary.
- Deterministic latency: stable processing pipeline timing for radar cube/point cloud/track outputs.
- Thermal headroom: worst-case RF/processing load must not throttle timing stability.
Camera + ISP output contract
- Device categories: image sensor, ISP (in-sensor or SoC), metadata path (exposure/gain/frame_id).
- Timestamp capability: frame start / frame_id must be observable and alignable to a common time base.
- Deterministic latency: fixed pipeline depth is preferred (avoid mode-dependent jitter surprises).
- Thermal headroom: temperature-driven drift must be monitored; thermal design must not destabilize frame timing.
Fusion compute (SoC / MCU / FPGA)
- Device categories: SoC/MPU, NPU/DSP, FPGA (optional), DMA/interconnect.
- Timestamp capability: supports ingest time stamping and consistent time domain across radar + camera paths.
- Deterministic latency: real-time scheduling options (dedicated cores/RT island), stable DMA priorities.
- Thermal headroom: sustained NPU/codec load must not push into throttling that increases jitter.
Clocking + network (node-local)
- Clocking categories: XO/TCXO, jitter cleaner PLL, reference clock distribution, RTC/holdover (optional).
- Network categories: Ethernet PHY (HW timestamp preferred), optional small switch (node-local), queue resources.
- Timestamp capability: HW timestamping at MAC/PHY is preferred for stable PTP measurement.
- Deterministic latency: sufficient queue separation and shaping support under video bursts.
Selection checks bound to three criteria (review-ready)
| Module | Timestamp capability (must be true) | Deterministic latency support (must be true) | Thermal headroom (must be true) | Evidence to collect |
|---|---|---|---|---|
| Radar | Time markers or time-tagging at node boundary | Stable output pipeline timing (avoid mode-dependent latency swings) | No throttling under worst-case RF + processing load | Radar processing time stats + sync/jitter logs |
| Camera/ISP | Frame_id / capture timing observable and alignable | Fixed (or bounded) ISP pipeline depth | Frame timing stable over temperature | Frame timing histograms + residual trend by temp |
| Compute | Unified time domain for ingest, fuse, tx | RT scheduling options + stable DMA/IRQ priorities | Sustained load without frequency/power throttling | P99.9 latency budget adherence + CPU/NPU utilization |
| Clocking | Low-jitter reference distribution to timestamp domains | Holdover behavior does not create long-tail drift | Clock stability within operating temperature range | Offset/jitter distributions + drift after warm-up |
| Network | HW timestamp support preferred (MAC/PHY) | Queue isolation and shaping under video bursts | PHY/switch thermals do not degrade error rates | tx_queue_latency + packet_drop_counter per class |
| NVM | Calibration record provenance (timestamp_source) | Atomic slot switch + CRC gating | Data retention across temperature | CRC failures, version rollback attempts, audit logs |
Evidence deliverables (selection must be defensible)
- Criterion 1:
timestamp capability— HW timestamping or equivalent observability for capture/tx domains. - Criterion 2:
deterministic latency support— stable pipelines + queue separation + bounded jitter under load. - Criterion 3:
thermal headroom— sustained performance without throttling-induced jitter or drift.
H2-9. Validation Plan: Bench, Chamber, and Field Correlation
Intent: deliver a repeatable validation SOP. Sync, latency, calibration residuals, and fusion consistency must be measurable and correlatable across bench, temperature chamber, and field.
Validation is not a list of tests. It is a single measurement contract that stays consistent across environments: the same evidence fields, the same tap points, and the same acceptance gates. Bench establishes the baseline, the chamber isolates temperature-driven drift, and the field correlation maps real incidents back to the same metrics.
Instrumentation contract (must exist before testing)
- Timestamp tap points:
t_capture,t_fuse,t_tx(per modality where applicable). - Sync evidence:
ptp_offset,ptp_jitter,timestamp_source(HW/PTP/SW). - Latency evidence: per-stage deltas from the tap points; queue evidence from node egress counters.
- Calibration evidence:
cal_version,reprojection_residual(px)trend, CRC/slot validity (as available). - Fusion evidence:
association_fail_rate,track_id_switch_rate, plus a traceable event ID for incident replay.
Bench SOP (baseline determinism)
- Sync: log
ptp_offset/ptp_jitterunder idle, then under worst-case video burst. - Latency: compute per-stage deltas using
t_capture→t_fuse→t_tx; emphasize P99.9 (not average). - Egress control: record
tx_queue_latency(per class) +packet_drop_counter(per class).
Chamber SOP (temperature-driven drift isolation)
- Time offset drift: track
ptp_offsetvs temperature during a ramp and a soak. - Extrinsic drift: track
reprojection_residual(px)vs temperature (mount/housing compliance). - Coupling check: repeat at light-load and full-load to separate thermal vs load-induced effects.
Field correlation (incident → metric → root cause)
- Trajectory consistency: compare radar speed trend vs video motion/box stability in the same time window.
- ID stability: watch
track_id_switch_ratearound crowding, occlusion, and reflective scenes. - Sync sanity: confirm
ptp_offsettrend and frame timing gaps during the same incident.
Acceptance gates (release-ready)
- Gate 1 (Sync):
ptp_offset/ptp_jitterremain within an upper bound using P99.9 under worst-case traffic. - Gate 2 (Cal):
reprojection_residual(px)stays below a bound across the temperature envelope. - Gate 3 (Fusion):
track_id_switch_ratestays below a bound in a defined field scenario set.
Evidence checklist (minimum)
- Bench:
ptp_offset,ptp_jitter, per-stage latency deltas,tx_queue_latency,packet_drop_counter. - Chamber: offset-vs-temp curve, residual-vs-temp curve, plus “load = light/full” annotations.
- Field: incident IDs linked to
track_id_switch_rateand sync/latency snapshots (same time window).
H2-10. Field Debug Playbook: Symptom → Evidence → Isolate → First Fix
Intent: a field-usable playbook that works with minimal tools. Each symptom routes to two measurable evidence points, one discriminator, and one first fix.
Field issues are rarely “algorithm problems” first. Most repeat incidents come from time-domain drift, latency long-tail under load, or calibration/extrinsic drift. The playbook below forces a consistent workflow: measure two things, separate domains, then apply a reversible first fix.
Symptom A — Track jumps / fused position drifts
- First 2 measurements: (1)
ptp_offsettrend (look for long-tail spikes), (2) frame timing gaps / timestamp deltas fromt_capture. - Discriminator: offset long-tail rises while residual stays stable → time-domain issue. Residual rises while offset stays stable → extrinsic/calibration issue.
- First fix: reduce scheduler jitter (pin critical threads/cores, remove background load), reduce queue depth for critical classes, then re-run time-offset calibration if residual is not the driver.
Symptom B — Mis-association only at high bitrate / multi-stream
- Evidence to collect:
tx_queue_latency(per class),packet_drop_counter(per class), plus CPU/NPU load snapshots. - Discriminator: if PTP/control queue latency long-tail correlates with the incident window, the root cause is egress determinism, not geometry.
- First fix: enforce QoS priorities, rate-limit video bursts, shift to ROI sub-stream strategy, and ensure video drops occur before control/PTP drops.
Symptom C — Alignment degrades after temperature changes
- Evidence to collect:
ptp_offsetvs temperature, andreprojection_residual(px)vs temperature (same time window). - Discriminator: offset changes with temperature but residual stays flat → clock/time offset drift. Residual changes with temperature → extrinsic/mechanical drift.
- First fix: apply temperature compensation for time offset, improve mechanical hold/rigidity for extrinsics, and define a re-calibration policy triggered by temperature thresholds.
Minimum toolset (enough for first isolation)
- Logs/counters:
ptp_offset,ptp_jitter,tx_queue_latency,packet_drop_counter,track_id_switch_rate,reprojection_residual(px). - Time window: an incident ID or timestamp range to pull “before/during/after” snapshots.
- Rule: measure two signals first; do not apply multi-factor changes until a discriminator points to a domain.
H2-11. Data Integrity at the Node (Just Enough): Timestamp Provenance & Audit Hooks
Keep fusion evidence traceable with a minimum, repeatable provenance tag attached to every output, plus audit hooks that record only meaningful state changes. This chapter is node-side only and does not expand into platform signature chains or recording compliance systems.
What this chapter owns
- Timestamp provenance: every output must declare which clock/timestamp domain was used.
- Reproducibility: outputs must reference the exact calibration/config version used at generation time.
- Auditability: state changes (sync degradation, re-sync, cal updates) must be logged with a minimal schema.
What this chapter explicitly does NOT own
- Full key/PKI design, trust chains, WORM storage, non-repudiation platform services.
- Recorder/NVR/VMS compliance pipeline, cloud audit ingestion, or “whole system” security architecture.
- Grandmaster selection/BMCA and network-wide PTP planning (covered by Timing/Sync pages).
1) Minimum Provenance Tag (must travel with every output)
Treat provenance as a uniform tag attached to every exported artifact (object list / alert event / ROI / metadata). If any required field is missing, the node-side evidence chain is broken and field correlation becomes guesswork.
Required fields
- timestamp_source: HW / PTP / SW (and never “unknown”).
- cal_version: monotonic or semantic version of the active calibration set.
- sync_state: LOCKED / DEGRADED / HOLDOVER / FREERUN / RESYNCING.
Recommended fields (still “just enough”)
- config_version: fusion thresholds / gating / ROI policy version.
- clock_id or ptp_domain: disambiguate multi-domain deployments.
- ptp_offset_ns_bucket: bucketed offset for quick correlation without huge logs.
- event_seq: monotonic output sequence (helps detect missing frames/packets).
2) sync_state must be computable and enforceable (not a decorative label)
A useful sync_state is a deterministic node-side decision derived from measurable signals (hardware timestamp availability, PTP lock status, offset/jitter statistics, holdover mode).
Suggested states (node-side)
- LOCKED: offset/jitter within pass thresholds for a defined window.
- DEGRADED: still locked, but long-tail offset/jitter rising (fusion confidence must drop).
- HOLDOVER: PTP lost; local oscillator/RTC holdover active and explicitly declared.
- FREERUN: no valid sync source; timestamps are local-only.
- RESYNCING: recovery window after link down/up or domain switch.
Output policy (just enough)
- LOCKED: normal fusion export.
- DEGRADED: export continues but must tag reduced confidence; avoid aggressive gating.
- HOLDOVER/FREERUN/RESYNCING: freeze fused tracks or export “single-modality only” with explicit flags.
This keeps integrity “engineering-real”: the tag drives behavior, not only logging.
3) Audit hooks: log changes, not constant noise
Provenance tags describe every output. Audit hooks record events that change the interpretation of the output. Keep them minimal, monotonic, and searchable.
Minimum event types (node-side)
- SYNC_THRESHOLD_EXCEEDED: offset/jitter crossed thresholds (include window stats).
- SYNC_STATE_CHANGED: LOCKED→DEGRADED→HOLDOVER…
- CAL_UPDATED: calibration set changed (from/to version + CRC).
- CONFIG_UPDATED: config changed (from/to version).
- LINK_DOWN_UP: link drop/recover that triggers resync.
- TIMEBASE_SWITCHED: PTP↔local or HW↔SW timestamp mode changes.
Minimum per-event fields
- event_id: monotonic counter.
- event_ts: timestamp (also tagged by timestamp_source semantics).
- sync_state: state at the moment of the event (and old/new for transitions).
- cal_version & config_version: snapshot for reproducibility.
4) Hardware building blocks (with concrete MPN examples)
The goal is not “more security”; it is repeatable provenance capture and durable audit logging. Choose parts based on: timestamp capability, deterministic behavior, and retention/robustness.
Ethernet PHY with HW timestamp / PTP support (examples)
- TI DP83867IR/CR — Gigabit PHY, IEEE 1588-related timestamp features (SOF detection).
- TI DP83640 — 10/100 “Precision PHYTER”, IEEE 1588 PTP PHY class.
- Microchip LAN8840 — Gigabit PHY with IEEE 1588 v2 support.
- Microchip LAN8814 — Quad Gigabit PHY with IEEE-1588 timestamping support.
- Marvell Alaska 88E1512 / 88E151x — family supports PTP v2 time stamping (check variant & design).
Tip: prefer PHY/MAC paths that expose a usable PTP clock and deterministic timestamp points; avoid “SW-only” timestamping in fusion nodes.
NVM for calibration + event log (examples)
- SPI NOR Flash (cal/versioned blobs): Winbond W25Q64JV / W25Q128JV.
- FRAM (high-endurance event log): Infineon/Cypress FM25V10 / FM25V20.
- I²C EEPROM (small config snapshots): Microchip 24LC256 / 24AA512.
Minimum storage rules: version + CRC per blob, plus an append-only ring buffer for audit events.
RTC / holdover reference (examples)
- Maxim/ADI DS3231 — temperature-compensated RTC (common, easy bring-up).
- NXP PCF2129 — low-power RTC with battery backup options.
- Microchip MCP79410 — RTC with battery backup features (variant dependent).
Use RTC/holdover only as a declared mode (HOLDOVER), never as a silent substitute for locked PTP time.
Optional audit anchor (examples)
- Microchip ATECC608B — secure element with monotonic counters (use as event_id / change counter anchor).
- NXP SE050 — secure element option for constrained identity anchors (keep policy minimal here).
Keep this “just enough”: use counters/identity hooks to protect event ordering and provenance claims, without expanding into full key lifecycle design in this chapter.
Figure F11 — Provenance Tagging (node outputs + audit hooks)
A single tag schema is attached to radar tracks, video metadata, and fused outputs. Audit hooks record only the moments that change interpretation (sync degradation, resync, cal/config updates).
Suggested ALT: Radar-camera fusion node provenance tags showing timestamp_source, cal_version, sync_state on radar tracks, video metadata and fused outputs, plus audit hook events.
Request a Quote
H2-12. FAQs (12) — Evidence-locked, No Scope Creep
Each answer stays inside node-side evidence: Sync, Latency/Jitter, Calibration, Association, Ethernet/TSN, Provenance tags.
FAQ 01Fusion looks right at install, drifts weeks later—sync or extrinsic?
Short answer: weeks-later drift is usually time holdover/offset long-tail or mount/extrinsic creep; separate them with two curves.
- Measure: ptp_offset/ptp_jitter trend vs time; reprojection_residual(px) vs time/temperature.
- Discriminator: offset long-tail rises while residual stays flat → sync; residual rises with temp cycles while offset stays bounded → extrinsic/mechanics.
- First fix: enforce sync_state downgrade policy or re-run extrinsic + add mechanical retention.
FAQ 02PTP offset is small, but tracks still jump—what else?
Short answer: small average offset can hide latency jitter and association instability that causes track “teleporting.”
- Measure: end-to-end latency P99.9 (t_capture→t_fuse→t_tx); track_id_switch_rate and association_fail_rate.
- Discriminator: latency spikes correlate with jumps → pipeline/queue; stable latency but high ID switches → gating/association constraints.
- First fix: fix deterministic budget (queue depth, DMA priority) or tighten explainable gating with confidence/velocity checks.
FAQ 03Only fails when multiple streams enabled—TSN/QoS issue?
Short answer: multi-stream failures are typically queue contention where metadata/control is delayed behind video bursts.
- Measure: tx_queue_latency per traffic class; packet_drop_counter (and which queue drops).
- Discriminator: PTP/control queue latency stays low but video queue spikes → safe; PTP/control latency spikes → TSN/QoS mis-classification or shaping error.
- First fix: enforce classing (PTP highest, control/objects above main stream), cap bitrate, enable shaping for bursts.
FAQ 04Camera frame rate is stable, radar timing isn’t—where to probe?
Short answer: probe the radar capture timestamp domain and its clock reference path; the issue is often a clock-domain or chirp timing anchor, not video FPS.
- Measure: radar t_capture spacing (chirp/frame cadence); clock-domain status (PLL lock / reference presence) feeding radar TRx/AFE.
- Discriminator: irregular t_capture with stable reference → processing stall; irregular with ref instability → clock tree/PLL/REF distribution.
- First fix: stabilize reference/PLL distribution or isolate radar capture DMA/interrupt priority.
FAQ 05Reprojection residual increases after temperature cycling—mechanical or calibration?
Short answer: temperature cycling usually exposes extrinsic movement or time-offset drift; both look like misalignment but leave different signatures.
- Measure: reprojection_residual(px) vs temperature; time_offset (or offset bucket) vs temperature.
- Discriminator: residual rises while time-offset stays stable → mechanics/extrinsic; time-offset shifts with temp while residual shape stays similar → time-offset calibration/holdover.
- First fix: add mechanical retention or introduce temp-compensated time-offset table and re-cal procedure.
FAQ 06Association fails on reflective targets—how to gate safely?
Short answer: reflective targets can create ambiguous radar points; use conservative, explainable gating that combines geometry + velocity + confidence instead of aggressive proximity-only matching.
- Measure: association_fail_rate split by target class; track_id_switch_rate near specular zones.
- Discriminator: failures spike only when Doppler/angle ambiguity rises → tighten velocity-consistency and confidence thresholds.
- First fix: widen spatial gate but require velocity agreement; down-rank low-confidence radar points in ROI projection.
FAQ 07CRC/logs look clean, but time alignment is wrong—what counters matter?
Short answer: CRC only proves data integrity, not temporal correctness; alignment breaks when timestamps or latency tails shift under load.
- Measure: ptp_offset/jitter window stats; frame timestamp gap and queue depth/latency (P99.9).
- Discriminator: offset long-tail without drops → sync; queue tail spikes without offset change → scheduling/queues.
- First fix: enforce sync_state policy or reduce buffering (fixed pipeline depth, capped queue depth, CPU pinning).
FAQ 08How do I verify HW timestamping is really used?
Short answer: HW timestamping shows up as lower jitter and consistent timestamp points; verify by checking the MAC/PHY timestamp counters and comparing SW vs HW behavior under CPU/network stress.
- Measure: PTP stats (offset/jitter) while stressing CPU + traffic; driver/PHY registers indicating HW timestamp enable and timestamp domain used.
- Discriminator: if jitter explodes with CPU load, timestamps are likely SW-path.
- First fix: enable PHY/MAC HW timestamp path end-to-end and ensure timestamps are taken at the correct ingress/egress point.
FAQ 09What’s the minimum metadata to ship upstream for debug?
Short answer: ship only what preserves the evidence chain: capture/fuse/tx timestamps plus provenance tags, so any upstream system can reconstruct timing, calibration, and sync health without raw sensor dumps.
- Measure/Include: t_capture, t_fuse, t_tx and timestamp_source.
- Include: cal_version, config_version, sync_state, and optional offset bucket.
- First fix: standardize the tag schema across object list, alerts, and metadata to avoid “partial evidence” uploads.
FAQ 10After firmware update, fusion changes—what must be versioned?
Short answer: any change that affects projection/association must be traceable; the minimum is to version calibration, fusion config, and timebase behavior so field results can be reproduced and compared.
- Measure: whether cal_version changed and whether config_version changed.
- Check: timestamp_source and sync_state transitions around reboot/update (resync windows matter).
- First fix: enforce CAL_UPDATED / CONFIG_UPDATED audit events and block silent overwrites (CRC + rollback guard).
FAQ 11Jitter spikes randomly—CPU scheduling or network queue?
Short answer: separate compute jitter from network jitter by correlating timestamp gaps with CPU load and per-queue latency; random spikes usually come from either preemption/IRQ storms or queue bursts.
- Measure: t_capture→t_fuse variance with CPU load/affinity; tx_queue_latency and drop counters per class.
- Discriminator: compute gap spikes without queue spikes → scheduling; queue spikes without compute spikes → QoS/shaping.
- First fix: pin real-time threads / prioritize DMA, and enforce traffic shaping for bursty video streams.
FAQ 12Best practice to recalibrate in field without bricking evidence chain?
Short answer: field recalibration must be an auditable, versioned transaction; never overwrite in place without a reversible record, and always bind outputs to the active cal/config versions during and after the update.
- Measure: before/after reprojection_residual and time_offset, each tagged with cal_version.
- Enforce: CAL_UPDATED audit event (from/to version + CRC) and a rollback-safe slot strategy.
- First fix: use staged update + verification window; downgrade sync_state during RESYNCING to avoid false confidence.
