• Architecture clarity: what belongs to the grabber (receive → buffer → DMA) vs what must stay elsewhere (detector AFE / ISP / codec / storage).
• Stability mechanics: how to make frame loss, backpressure, and link retraining observable (counters + timestamps), not “random glitches”.
• Sync & traceability: where genlock/trigger and timestamp insertion must live so frame alignment can be verified end-to-end.

• Detector front-ends: CT/FPD/MRI/Ultrasound/OCT sensor AFEs and modality-specific physics.
• Image processing pipelines: ISP/HDR/denoise/compression algorithms beyond minimal framing metadata.
• Recorder/storage deep dive: NVMe/UFS PLP and long-term acquisition storage architecture.

• Inputs: SerDes data + sideband (trigger/genlock/lock-status as applicable).
• Outputs: aligned lanes / framed packets + link health counters (CRC, retrain, lane degrade).
• Responsibility: make link stability observable; expose training state and error rates as measurable signals.

“Minimal” is not “optional”. Without these actions, the system becomes a black box where drops and sync faults cannot be reproduced or audited.

• Alignment: lane deskew / reorder to a stable internal representation.
• Frame boundaries: detect SOF/EOF (or map packet groups to a frame unit).
• Metadata envelope: attach frame_id, timestamp, sync_state, error_flags.
• Buffering: FIFO + DDR with watermarks sized for burst absorption and host jitter.
• Backpressure policy: define what happens when the host cannot keep up (drop mark, throttle, alarm).
• Observability: counters and event logs for overflow, retrain, timebase loss, DMA starvation.

• Output: PCIe DMA writes into pre-allocated host pages (scatter-gather ring), plus completion/timeout statistics.
• Responsibility: sustained throughput, bounded latency jitter, and fault signals that point to a layer (link vs buffer vs DMA).
• Minimum health signals: descriptor starvation, completion timeout, IOMMU fault, and queue depth watermark.

• ROI/stride handling: helps DMA efficiency, but must not become an ISP pipeline.
• Frame re-pack: align to cache/page boundaries for predictable host bandwidth.
• Continuity checks: frame sequence validation and lightweight CRC reporting (no image processing).

Raw pixel rate:
  DataRate_raw = W × H × bpp × fps

Effective required rate (budgeted):
  DataRate_req = DataRate_raw × (1/η_total) × K_burst

Where:
  η_total  = total payload efficiency (encoding + framing + headers + idle gaps)
  K_burst  = burst/jitter factor (host stalls, retrain hiccups, DMA segmentation, cache/NUMA effects)
Multi-source:
  DataRate_req_total = Σ(DataRate_req_i)   (track Avg and Peak separately)

• Average decides whether sustained streaming stays smooth.
• Peak decides whether buffers overflow during bursts (triggers, exposure phases, host scheduling jitter).
• Budget two lines: Avg path for link/PCIe sizing, and Peak path for FIFO/DDR watermarks and “burst window”.

• SerDes lanes: negotiated rate/width + error bursts can reduce effective payload even when raw rate is high.
• FPGA ingress: internal backpressure or alignment resets can create periodic stalls.
• DDR bandwidth: write + read contention under burst can hit watermarks even when average DDR BW “looks fine”.
• PCIe link: downshift (lower speed/width), completion timeouts, or switch/retimer stability events.
• Host RAM / NUMA: remote memory placement or extra copies can collapse application-visible throughput.

1. Using raw pixel rate as “usable bandwidth”.
   Symptom: sustained streaming shows periodic dips despite “enough” headline bandwidth.
   Verify: measure payload efficiency (η_total) and look for protocol/idle gaps and packet overhead in counters.
2. Sizing by average only, ignoring peak bursts.
   Symptom: “random” frame drops aligned with triggers/exposure bursts or host load spikes.
   Verify: FIFO/DDR watermark hits + drop counters with timestamps during burst windows.
3. Assuming PCIe/host memory is a flat, constant resource.
   Symptom: throughput collapses when the system runs other workloads, or when memory is allocated on a remote NUMA node.
   Verify: DMA queue depth, completion latency, and memory locality (NUMA binding) under load.

• ☐ Inputs defined: per source W/H/bpp/fps and number of sources (N).
• ☐ Two rates computed: Avg and Peak (Peak drives buffer sizing).
• ☐ η_total stated per stage (encoding/framing/headers/idle) — do not assume 100%.
• ☐ K_burst stated (host jitter, DMA segmentation, retrain hiccups).
• ☐ Stage-by-stage required rate checked: SerDes → FPGA ingress → DDR → PCIe → Host RAM/NUMA.
• ☐ Headroom documented for each stage and justified (burst, jitter, retrain, contention).
• ☐ Pass/fail signals defined: watermark hits, drops, downshift, completion latency, queue starvation.

• Connector → establishes the insertion loss / reflection reality the rest of the chain must tolerate.
• ESD / CMC (touchpoint) → protection and EMI control without destroying margin (keep this minimal and measurable).
• Redriver / Retimer → compensate loss; optionally restore timing to rebuild margin.
• EQ (CTLE/DFE) → adaptively correct channel distortion; settings must be reportable for debugging.
• CDR / PCS → recover clock/data, expose lock state and training outcomes.
• Lane deskew → align multi-lane timing; expose deskew errors and re-alignment events.
• Packet framing + CRC → create verifiable payload integrity with counters and burst timestamps.

A redriver mainly boosts/compensates amplitude and equalization; a retimer restores timing by re-clocking data. Retimers are used when “amplify only” cannot meet stability and margin requirements.

• Prefer retimer when negotiated link repeatedly retrains, downshifts speed/width, or shows low training margin.
• Prefer retimer when long channels / multiple connectors cause loss and reflections that push EQ to extremes.
• Prefer retimer when deterministic capture is sensitive to jitter bursts (link is “up” but payload errors spike).
• Redriver may be enough when training margin is healthy and errors remain low across temperature and EMI stress.

• Training / lock state: locked, training, degraded, downshifted; include timestamps for state changes.
• Error counters: CRC errors, symbol errors, deskew errors; record bursts (not only totals).
• Retrain log: retrain count + reason code + time correlation with system events (trigger, thermal, EMI).
• Margin proxy: training outcome grade (high/med/low) and selected EQ settings for “before/after” debugging.

• Link flaps (reconnect loops): verify retrain count bursts + lock-state toggles + thermal correlation.
• CRC bursts (payload corruption spikes): verify CRC burst timestamps vs triggers/exposure phases and EMI events.
• Downshift (lower speed/width): verify negotiated rate/width + training outcome grade + channel condition changes.

• If the channel is short with few discontinuities and training margin is healthy → Direct (Endpoint → Root Complex).
• If training margin is low, retrains happen, or the link downshifts under stress → add a PCIe retimer (re-clocking to rebuild margin).
• If multiple endpoints must share one host link, or fan-out/segmentation is required → add a PCIe switch (fabric + arbitration).
• If deterministic capture must remain stable under host workload swings → prefer simpler topology plus strong observability over “hero bandwidth”.

• Negotiation is adaptive: if training fails at a target mode, the link can fall back to a lower speed or fewer lanes.
• Degradation is conditional: temperature, EMI, connector contact, and power noise can push a “barely passing” channel into retrain/downshift.
• Stability requires logging: record speed/width, retrain count, and error bursts with timestamps to reproduce the trigger conditions.

• Discontinuities: connectors, via stubs, and reference plane changes increase reflections and reduce margin.
• Return loss / reflections: low training margin, retrains, and mode fallback can appear even when “it enumerates”.
• Crosstalk: error bursts correlated with high activity or nearby aggressors can collapse payload efficiency.
• Power integrity coupling: link appears up, but completion latency spikes and throughput dips under load.

• NUMA locality: DMA into remote memory can reduce application-visible bandwidth and raise jitter.
• IOMMU mapping: page mapping and translation overhead can matter for bursty, high-rate scatter-gather traffic.
• Resource constraints: BAR/resource allocation issues can affect enumeration and stability (monitor and log).
• Fabric policies: isolation/routing features (e.g., ACS/ARI) can change paths and latency; treat as a throughput variable.

• Negotiated: current speed, width, and any downshift transitions (with timestamps).
• Stability: retrain count + reason code + error burst counters (AER/events where available).
• Correlation: temperature and host load snapshot when transitions occur.

• Sustained throughput: DMA queue depth remains healthy; completion latency does not show periodic spikes.
• Bounded latency jitter: watermark strategy keeps burst absorption within a defined window.
• No silent loss: any drop is surfaced as drop_flag + drop_count + reason + timestamp.

• Descriptor ring: a stable producer/consumer queue for DMA work submission and completion tracking.
• Scatter-gather: frames can land in non-contiguous pages while remaining logically contiguous to software.
• Pre-allocated pages: avoid runtime allocation jitter that converts host load spikes into drops.
• Cache coherency: software reads must see the latest DMA-written data with a defined coherency contract.

• On-board DDR: absorbs bursts and shields the link from host jitter; size via peak-rate burst window and watermark rules.
• Host RAM direct: lowers latency and supports near zero-copy paths, but depends strongly on NUMA placement and host stability.
• Multi-stream fairness: define arbitration so critical streams are protected when buffers approach high watermark.

• High watermark → enter controlled mode (throttle, prioritize, or drop-with-marking).
• Drop-with-marking → set drop_flag, increment drop_count, attach drop_reason and timestamp.
• Starvation detection → detect descriptor starvation and completion timeout; treat as a health event, not a mystery stall.
• Recovery → return to normal only when queue depth and watermarks are back within safe bounds.

• ☐ Host pages are pre-allocated and placed on the correct NUMA node for the DMA device.
• ☐ Ring depth covers the worst-case host scheduling jitter window (defined and tested).
• ☐ DDR/FIFO watermarks match the peak burst window (measured, not guessed).
• ☐ Completion latency and queue depth are monitored; thresholds trigger controlled degradation.
• ☐ Drops/overflows/timeouts are counted and timestamped, with a reason code readable by software.
• ☐ Multi-stream arbitration policy is explicit (fairness/priority) to avoid starvation of critical inputs.

• Inputs: Genlock Ref In, Trigger In, optional 1PPS In, optional time-reference In.
• Outputs: Genlock Ref Out/Loop, Trigger Out (distribution/echo), optional time-reference Out.
• Status: Lock state (locked/holdover/unlocked) and loss-of-lock event flag (readable by host software).
• Metadata fields: frame_id, stream_id, timestamp, and drop/error flags carried alongside frame payload.

• Board ingress: earliest reference to external trigger/ref; useful for system correlation, but includes channel delay uncertainty.
• Post-deserializer: closer to “data valid” after CDR/deskew; good for multi-lane alignment and link-quality correlation.
• Pre-DDR write: ties timing to buffer entry; highlights burst absorption and congestion effects.
• DMA egress: simplest for host visibility, but may include host-side jitter and completion latency variability.

• Alignment looks unstable only under host load: egress (DMA) timestamp includes completion jitter; move earlier (post-deserializer or pre-DDR).
• Multi-stream skew cannot be explained: timestamps were taken before lane alignment; place after deskew/unpack.
• Burst buffering hides real timing: timestamps taken after buffering mask ingress timing; stamp at post-deserializer or pre-DDR depending on target.

• ☐ Frame alignment error is defined (offset/jitter between streams or to a reference).
• ☐ Offset statistics are exposed (min/max/mean and at least one percentile such as P95).
• ☐ Lock state is exposed (locked/holdover/unlocked) and sampled with timestamps.
• ☐ Loss-of-lock events are counted and timestamped (count + duration + recovery time).
• ☐ Each frame carries timestamp + frame_id + flags so software can correlate quality with events.

• Symptom: intermittent corruption, periodic drops, unstable alignment.
• Likely causes: margin too low, crosstalk/EMI bursts, temperature drift, connector issues.
• Fastest checks: CRC counter (total + burst), deskew error count, retrain count + reason, lane degrade/downshift state.

• Symptom: drops cluster during bursts, latency spikes, occasional stalls under load.
• Likely causes: burst window exceeds absorption, unfair arbitration, DDR contention or ECC/timeouts.
• Fastest checks: FIFO overflow count + timestamp, watermark high hit + duration, DDR ECC count, DDR timeout count.

• Symptom: throughput collapse, DMA pauses, instability when system load changes.
• Likely causes: completion latency spikes, IOMMU mapping faults, descriptor ring starvation, resource pressure.
• Fastest checks: completion timeout count, max completion latency, IOMMU fault count/type, descriptor starvation count, DMA queue depth.

• Each critical event is counted, timestamped, and tagged with a reason code.
• Each frame can carry flags (crc_bad / dropped / degraded) so application logs match device counters.
• Correlation fields include current link mode (speed/width) and current buffer state (watermark level).

• Retimers: re-clocking margin degrades with heat; sustained error bursts often start here.
• FPGA SerDes banks: high-speed transceivers are sensitive to temperature and local supply noise.
• DDR buffers: congestion and ECC/timeouts can appear during burst absorption under elevated temperature.
• PCIe switch and PCIe edge region: fabric arbitration + signal integrity sensitivity can trigger mode changes.
• VRM / power stages: ripple and transient response affect SerDes stability and completion latency under load.

• T-retimer: correlates with CRC bursts and retrain spikes when the channel margin is heat-limited.
• T-fpga-bank: correlates with lane errors, deskew stress, and receiver sensitivity changes.
• T-ddr: correlates with watermark duration, buffer occupancy, and ECC/timeout events.
• T-switch/pcie: correlates with link mode stability (speed/width) and completion latency outliers.
• Always align in time: temperature samples must be time-aligned with error bursts, retrains, downshift events, drops.

• Throttle modes: link mode reduction (speed/width), frame-rate limiting, or stream prioritization under high watermark.
• Entry/exit rules: define thresholds and hysteresis to avoid oscillation and repeated retrains.
• Event logging: thermal_throttle_enter/exit with timestamp, duration, current link mode, and error counter snapshots.
• Operator visibility: expose a health state (normal / throttled / critical) to the host application.

• ☐ Link mode is stable: negotiated speed/width does not unexpectedly downshift under sustained load, or any transition is logged.
• ☐ Error bursts are bounded: CRC/retrain counters do not show temperature-correlated runaway behavior.
• ☐ Buffer health is stable: watermark-high duration remains bounded; drops are 0 or explicitly marked with reason+timestamp.
• ☐ Thermal policy is traceable: throttle enter/exit events exist with durations and correlated counter snapshots.

• Common-mode injection: shield termination mistakes can convert external fields into receiver stress and CRC bursts.
• Return-path discontinuity: split references or broken return paths amplify jitter and deskew/CRC failures.
• Ground bounce: fast di/dt on shared return paths can cause intermittent lane errors and retrains.
• Power-to-SerDes coupling: PDN noise shifts margin and can trigger downshifts under full load.

• Connector shielding: define a clear shield termination strategy (and keep it consistent across cable and chassis interfaces).
• Reference ground: preserve a continuous return path near high-speed pairs; avoid routing that forces long return detours.
• I/O boundary: if the system requires isolation, clearly define which signals cross the boundary and how status is returned.
• Countable outcomes: every “touchpoint change” must be evaluated using CRC/retrain/downshift/drops counters, not impressions.

• ☐ Hold the workload constant (same streams, same sustained rate) and change only one touchpoint at a time.
• ☐ Compare CRC bursts, retrain events, link downshifts, and drops over the same time window.
• ☐ Confirm link mode stability improves (speed/width stays consistent) and completion latency outliers reduce.
• ☐ Log results with timestamps so improvements correlate with the modification and environment conditions.

• Input mode: PRBS/BERT or framed test stream (sequence + CRC), sustained at the target lane rate.
• Run time: quick screen (10–20 min) + soak (1–24 h) depending on risk.
• Record fields: link_mode (speed/width), crc_burst_count, retrain_count + retrain_reason, deskew_error_count (if available), lane_degrade_events.
• Pass criteria fields: stable link_mode over the run, no retrain storms, no temperature-correlated runaway of error bursts.

• Input mode: frames with frame_id (monotonic), stream_id, payload_crc; include steady + burst phases.
• Record fields: avg_throughput, peak_throughput, drop_count (by reason), out_of_order_count, duplicate_frame_count, dma_queue_depth_min or descriptor_starvation_count.
• Pass criteria fields: steady-mode drop_count = 0 (or every drop is flagged with reason+timestamp), ordering counters remain 0, sustained throughput matches budget within headroom.

• Input mode: genlock reference + trigger (as applicable); multi-stream alignment under the same reference.
• Record fields: lock_state (locked/holdover/unlocked), loss_of_lock_count + duration, offset_stats (min/max/mean/P95), drift_stats relative to the chosen reference.
• Pass criteria fields: lock_state stays locked for the defined window; any loss-of-lock is timestamped, reason-coded, and recoverable.

• Input mode: repeat A/B/C tests while temperature and EMI conditions are varied in controlled steps.
• Record fields: temp_sensors (T-retimer/T-fpga-bank/T-ddr/T-switch), thermal_throttle_enter/exit (if implemented), crc/retrain/downshift counters, drops (by reason).
• Pass criteria fields: controlled degradation only (throttle is logged), no silent failure signatures.

• Power-on self-check: read versions, serial, key sensors (temperature) and verify counters reset/roll correctly.
• Link train: confirm target negotiated speed/width; log training status.
• Short error-health run: 1–2 min framed stream; require crc_burst_count and retrain_count within the defined limit.
• DMA path check: run a fixed-size transfer; require no descriptor_starvation and stable queue depth.
• Basic genlock I/O check: verify lock_state is readable and loss-of-lock counter increments on forced unlock.
• Store pass/fail fields: timestamp + operator ID + link_mode + key counters + reason codes for any failure.