Likely cause: Real-time task is preempted by management/logging threads or DMA descriptors/credits hit low-watermark under bursts, stalling the data plane.

Quick check: Capture a X_trace_s trace: X_sched_us_max (max scheduler latency), X_cpu_pct_peak (CPU peak), DMA ring X_dma_desc_min (min available), and X_queue_frames_worst (worst queue depth). Jitter spikes that align with X_sched_us_max → CPU contention; spikes that align with DMA low-watermark/underrun counters → DMA starvation.

Fix: Pin cyclic path to RT core, raise priority, reserve DMA channels and descriptor pools, and throttle/decimate non-RT log export (rate limit to X_log_hz_max, move writes off RT path).

Pass criteria: Jitter ≤ X_jitter_ns_rms (p99 over X_minutes) and ≤ X_jitter_ns_pkpk (max); X_cpu_pct_peak not exceeded; DMA underrun/overflow counters = 0 over X_hours.

Likely cause: Image boots, but major.minor policy mismatch, config schema mismatch, or a disabled/changed feature flag blocks join/handshake.

Quick check: Export one “version manifest” bundle and compare against allow-list: X_fw_major_minor, X_cfg_schema_ver, X_stack_artifact_id, and X_feature_flags_hash. If any differ, treat it as gating failure (not link noise).

Fix: Enforce strict allow-list on boot; auto-migrate config only when schema is compatible; otherwise fall back to last-known-good slot and export a gating fault code.

Pass criteria: Join/rejoin completes in ≤ X_rejoin_s_max (max across X_trials); manifest matches allow-list 100%; rollback returns to cyclic-ready in ≤ X_rollback_s_max when gating fails.

Likely cause: Commit is non-atomic (no journal/CRC), or supervisor resets too early, cutting power before the “critical keys” write completes.

Quick check: Run a brownout sweep: log X_bod_irq_to_commit_ms (IRQ→commit done), X_reset_reason, and “journal state” (valid/invalid). Random loss with valid journal → sequencing/hold-up; invalid journal/CRC → journaling issue.

Fix: Use atomic journal (write new record + CRC + pointer flip); prioritize critical keys; ensure hold-up window ≥ X_holdup_ms_min and supervisor delay ≥ X_reset_delay_ms after commit-done signal.

Pass criteria: Critical state loss = 0 across X_brownout_cycles; commit completes in ≤ X_commit_ms_max (max); recovery to cyclic-ready in ≤ X_recover_s_max.

Likely cause: Stall is scheduling/lock contention, not a link error; counters miss it because they update too slowly or only count hard failures.

Quick check: Enable “low-cost trace” for X_trace_s: task switch latency (X_sched_us_max), lock wait time (X_lock_us_max), queue watermark (X_queue_frames_worst), and DMA watermark (X_dma_desc_min). Avoid full debug logs first.

Fix: Add fault snapshot trigger on “stall signature” (e.g., no cyclic progress for X_stall_ms), and gate verbose logs behind rate limits; isolate long operations to non-RT core.

Pass criteria: Stall events = 0 over X_hours at customer load; trace overhead ≤ X_trace_overhead_pct; snapshot capture ≤ X_snapshot_ms_max.

Likely cause: Under stress, queues exceed design depth (store-and-forward pressure), causing deadline misses or controlled drops that the test flags.

Quick check: Add queue watermark counters per class/priority: X_queue_frames_worst plus “drop reason” (overflow, policing, backpressure). Re-run worst-case traffic; if watermark approaches limit or drop reason != 0, it is queue-driven.

Fix: Reserve cyclic queue budget, apply strict priority separation, and move non-cyclic traffic to shaped/limited queues; verify overload policy is deterministic (no lockups).

Pass criteria: Worst-case queue depth ≤ X_queue_frames_worst_limit (max); deadline miss = 0 over X_minutes worst-case run; drop reason counters = 0 for cyclic class.

Likely cause: Slow path is either platform initialization (storage scan, crypto verify, config migration) or link bring-up/state machine waits (timeouts/retries).

Quick check: Add three timestamps: T_init_done, T_link_up, T_first_cyclic. Compute X_init_s=T_init_done−POR, X_link_s=T_link_up−T_init_done, X_cyclic_s=T_first_cyclic−T_link_up. The largest segment is the first target.

Fix: Parallelize non-critical init, postpone heavy diagnostics until cyclic-ready, and bound retries with deterministic fail codes; keep version gating early but time-bounded.

Pass criteria: Boot-to-cyclic-ready ≤ X_boot_to_cyclic_s (p99 over X_boot_trials); no segment exceeds its own budget (X_init_s_max, X_link_s_max, X_cyclic_s_max).

Likely cause: Timestamps are taken in a different clock domain than the actuator control loop (offset/phase not compensated), so “sync OK” does not guarantee phase at the actuator.

Quick check: Log both domains: timestamp clock ID and control-loop timebase ID, plus measured phase error at actuator X_phase_us_meas. If X_phase_us_meas changes with CPU load or port selection, it is a domain/tap mismatch.

Fix: Take timestamps in hardware at the correct boundary, lock timestamp clock to the same disciplined source as the control loop, and apply a single explicit offset model (documented, versioned).

Pass criteria: Timestamp resolution ≤ X_ts_ns_res; actuator phase error ≤ X_phase_us_at_actuator (max over X_minutes); holdover ≥ X_holdover_ms_min without exceeding phase budget.

Likely cause: Logging adds synchronous writes, locks, or bursts of export traffic on the same core/path as cyclic processing.

Quick check: Measure log/export rate X_log_hz_meas and storage write time X_io_us_max while watching X_sched_us_max. If jitter spikes align with X_io_us_max or log bursts, logging is the trigger.

Fix: Enforce: (1) cyclic path cannot block on I/O, (2) logs are buffered in RAM ring, (3) export is rate-limited to ≤ X_log_hz_max and moved to non-RT core/thread, (4) use “event IDs + counters” over verbose strings.

Pass criteria: With diagnostics enabled, jitter remains ≤ X_jitter_ns_rms (p99); export bandwidth ≤ X_export_kbps_max; snapshot capture ≤ X_snapshot_ms_max; cyclic error counters unchanged vs baseline.

Likely cause: A physical loop causes uncontrolled replication (broadcast/multicast or unknown-unicast), overwhelming queues and starving cyclic traffic.

Quick check: Watch three counters: broadcast/multicast rate X_bmc_pps, MAC churn X_mac_moves_per_s, and queue overflow/drops X_drop_overflow. If X_bmc_pps spikes and drops follow, it is a loop storm signature.

Fix: Apply storm control at the system level: rate-limit broadcast/multicast to ≤ X_bmc_pps_max, and define a protective action (temporary port block or isolation) when loop signature persists for > X_loop_ms.

Pass criteria: Under intentional loop injection, cyclic remains stable for X_minutes; overflow drops remain 0 for cyclic class; protective action triggers within ≤ X_loop_detect_ms.

Likely cause: Field power interruptions hit the narrow window where the slot switch metadata is updated but the new image is not yet validated end-to-end.

Quick check: Perform “random cut” tests across a defined window: from T_verify_done to T_first_cyclic, with cut intervals of X_cut_ms_step. Record boot slot selection and rollback reason codes on every cycle.

Fix: Use two-phase commit for slot switching (write intent → validate → finalize), keep rollback metadata in a small atomic journal, and guarantee hold-up ≥ X_holdup_ms_min for the finalize step.

Pass criteria: Across X_cut_cycles random-cut tests, system always boots to a valid image; rollback completes in ≤ X_rollback_s_max; no “stuck between slots” events (count = 0).

Likely cause: Link events trigger an interrupt/event storm that starves the watchdog service or supply dips during cable disturbances cause brownout-like behavior misclassified as watchdog.

Quick check: Correlate timestamps: cable event → X_isr_rate_peak (ISR rate peak), event queue depth X_evtq_depth_worst, and X_reset_reason. If ISR rate and queue depth spike before reset → event handling; if brownout reason/UV flag appears → power integrity.

Fix: Debounce and rate-limit link events, cap event queue growth, and guarantee watchdog service on a higher-priority path; if UV is observed, tighten supervisor thresholds and increase hold-up margin.

Pass criteria: No watchdog resets over X_hours with repeated cable events; ISR rate ≤ X_isr_rate_max; event queue depth ≤ X_evtq_depth_max; reset reason codes match expected (0 unexpected).

Likely cause: Behavior difference comes from non-identical config artifacts (object model, timing defaults, or enabled services), not from the wire itself.

Quick check: Compare three items side-by-side: X_cfg_artifact_hash, X_cfg_schema_ver, and X_feature_flags_hash. Then compare timing defaults: X_cycle_time, X_queue_frames_worst_limit, X_log_hz_max. Differences explain most “stack A vs B” gaps.

Fix: Freeze a single “golden artifact” and generate vendor-specific configs from it; enforce validation on boot (schema + hash); export an artifact mismatch fault code for field support.

Pass criteria: Artifact hash match rate = 100% across X_units; behavior equivalence on defined KPIs (jitter, rejoin time, counters) within ≤ X_delta_pct (max).