How it works (3–5 steps)

1. DMA & queues: descriptors in host memory define what to transmit/receive and where to place data.
2. Pipeline: parsing, classification, steering (RSS/flow rules) and optional offloads are applied.
3. RDMA (if used): verbs/QP/CQ scheduling maps requests to DMA operations with tight latency targets.
4. Line side: MAC/PCS/PMA drive SerDes lanes; PAM4 DSP and FEC protect the link margin.
5. Telemetry: counters and sensors (FEC/BER, module DDM, temperature, power) explain “why” performance changes.

First principles: what to check first (before brand/model)

• Throughput vs packet rate: Gbps describes bandwidth; Mpps predicts small-packet behavior and CPU pressure.
• Tail latency: p50 can look good while p99/p999 fails under microbursts, queue backpressure, or rare retries.
• RDMA stability (if required): “supported” is not enough—validate behavior under congestion and loss sensitivity.
• Observability: FEC corrected/uncorrected, BER proxies, module DDM/CMIS, and event logs must exist and be readable.

Three domains (who owns what)

• Host / PCIe domain: DMA submissions, queue doorbells, MSI-X interrupts, and (light) IOMMU/ATS effects on addressability and completion flow.
• NIC core domain: MAC + packet pipeline, flow steering, offload engines, and scheduling that turns descriptors into predictable packet movement.
• Line side domain: PCS/PMA and SerDes lanes connected to QSFP/OSFP modules and the fabric, with link training and error protection (FEC).

PAM4 vs NRZ: what changes in practice

• PAM4: smaller voltage spacing per level → higher sensitivity to noise, jitter, and non-ideal channel response.
• NRZ: larger margin → fewer “near-edge” states where corrected errors climb while the link stays up.
• Operational signal: rising FEC corrected without immediate uncorrected events often indicates a margin tax that can later convert into tail latency risk.

DSP chain: symptom → module → evidence

• High-frequency loss → CTLE helps restore high-frequency content; lane-to-lane corrected error imbalance is a common hint.
• ISI (intersymbol interference) → FFE/DFE compensate channel memory; training “not converged” or unstable EQ states often appear before hard failures.
• Jitter/phase noise sensitivity → CDR stability matters; errors that correlate with temperature or operating mode suggest clock/jitter stress.
• Residual random errors → FEC absorbs them until uncorrected events start; corrected/uncorrected ratio is a leading indicator.

FEC counters: how to read them without guessing

• Corrected trending up: the link is spending “error budget” to stay stable; margin is shrinking or compensation load is rising.
• Uncorrected events: represent real frame loss at the physical layer; they amplify retries and tail latency, especially for loss-sensitive transports.
• Lane concentration: errors localized to a subset of lanes often indicate a channel/connector/module lane-specific issue rather than a global mode mismatch.

Bring-up checklist (line-side): what to validate first

• Mode agreement: autoneg / capability match, correct lane count and mapping for the selected module mode.
• Training success: link training reaches stable EQ states (TX/RX equalization) and lane deskew completes.
• Polarity/mapping sanity: polarity inversions and lane swaps are tolerated only if mapping is consistent end-to-end.
• Correlations: plot corrected errors against temperature/power and module DDM to identify margin-driven instability.

Why “Gbps is high” but “Mpps collapses”

• Per-packet overhead dominates: each packet consumes descriptor work, queue arbitration, DMA bookkeeping, and completion processing.
• Interrupt/coalescing effects: moderation improves CPU efficiency but can stretch tail latency when bursts arrive or queues back up.
• Pipeline depth and contention: parse/classify/steer stages have finite capacity; microbursts convert into queue backlog and p99 spikes.
• Evidence pattern: wire-rate looks healthy while drops/overruns, queue pressure, or completion stalls increase.

Offload catalog: what it helps, what it can hurt

Metrics hierarchy (procurement + validation)

• Wire rate (Gbps): large-packet ceiling; necessary but not sufficient.
• Mpps (small packets): reveals per-packet cost and queue/interrupt capacity.
• p99 latency: exposes backlog, coalescing, retries, and burst sensitivity.
• CPU cycles saved: quantifies offload value without hiding tail-risk.

RoCEv2 vs InfiniBand: “engineering differences” (not a glossary)

• RoCEv2: runs over Ethernet transport. Practical stability depends on loss/congestion management in the path (mentioned only; no switch details here).
• InfiniBand: fabric semantics are native and consistent for RDMA traffic, with a more unified verbs/QP/CQ operating model end-to-end.
• Implication for p99: RoCE environments often show stronger correlation between congestion signals and tail latency; IB environments still suffer tail growth when retries or resource backlogs occur.

QP/CQ/doorbell/WQE: boundaries that create tail latency

• WQE (work request): the unit of outstanding work. Accumulated outstanding WQEs are the physical form of latency backlog.
• QP (queue pair): ordering and resource window. A blocked QP delays later work even if the link stays up.
• CQ (completion queue): completion aggregation. Completion handling cadence affects tail behavior under bursts.
• Doorbell: submission pressure. Batch submission improves efficiency but can increase waiting time when congestion or retries appear.

Why “a little loss” becomes “huge tail latency”

• Retry/timeout amplification: once retransmit or timeout logic activates, outstanding work waits behind recovery.
• Backlog propagation: queued work accumulates while completions slow down; p50 can look fine while p99/p999 degrade sharply.
• Evidence-first approach: correlate retry/timeout categories with queue backlog, completion error categories, and congestion indicators visible to the NIC/HCA.

What can be proven from the NIC/HCA side (without fabric configuration)

• Congestion signals (category): marks/pause-like indicators and rising queue occupancy patterns that align with latency spikes.
• Retry evidence (category): retransmit and timeout trends that precede uncorrected loss symptoms.
• Completion evidence (category): completion errors or slowed completion cadence that match tail growth.

Queue mental model (what actually runs in parallel)

• RX path: flow classification → RX queue → DMA to host → completion → MSI-X interrupt (or polling).
• TX path: TX queue → descriptors → DMA fetch → scheduling → wire.
• What it controls: wire rate (Gbps) is not enough; Mpps and p99 latency are dominated by queue pressure and completion cadence.

RSS and flow steering: benefits, and why “parallel” can become “noisy”

• RSS benefit: distributes flows across multiple queues to raise parallel throughput and reduce per-core hotspots.
• Hidden cost: cache locality can deteriorate when flows bounce across cores; tail latency rises under bursts.
• Ordering boundary: a single flow should remain on one queue; re-mapping or mixed steering policies can create reordering-like behavior.
• Flow steering role: prioritizes control (stable mapping, predictable p99) over “more randomness.”

SR-IOV (PF/VF): isolation is resource slicing

MSI-X and coalescing: the small-packet vs p99 trade-off knob

• MSI-X: provides multi-vector interrupts so multiple queues can complete independently (reduces single-IRQ bottlenecks).
• Coalescing: batches interrupts to reduce CPU overhead, but can increase tail latency by “holding” completions during bursts.
• Evidence-first rule: interpret p99 spikes together with IRQ rate, queue depth, and completion cadence—not by throughput alone.

What matters from the endpoint perspective

• Link width / speed changes: sudden ceiling shift (step-like throughput drop) without obvious line-side errors.
• Retraining / recovery episodes: short stalls that resemble “micro-disconnects” to upper layers.
• AER categories: error handling can trigger retries or recovery paths; tail latency grows even if p50 remains normal.
• Replay / completion timeout (category): completions slow down, queue pressure increases, and p99 spikes follow.

How to prove “network symptoms” are actually PCIe-driven

• Step 1: check line-side error trends (category). If they remain flat while performance shifts, host-side probability rises.
• Step 2: correlate PCIe link state changes (width/speed/retraining/AER categories) with the time of throughput drops or p99 spikes.
• Step 3: confirm completion stalls and queue pressure increase during those PCIe events.
• Step 4: conclude root layer: completions slowed → DMA progress slowed → queues back up → “network jitter” becomes visible.

Firmware/NVM changes: why versions can shift performance or compatibility (principles)

• Default queue and interrupt behavior can change: different moderation defaults alter Mpps and p99 trade-offs.
• Virtualization resource slicing can change: VF limits, queue allocation, or completion handling policies may differ across versions.
• Error handling paths can change: recovery behavior affects how often and how long “stalls” appear in the field.
• Validation rule: after any version change, re-check the same evidence categories (link state, AER categories, completion cadence, queue pressure).

Internal clock domains (what each domain controls)

• Refclk domain: the external reference feeding internal synthesis. Instability often shows as broad “margin shrink.”
• PLL / conditioning domain: produces internal clocks; poor margin increases BER and pushes FEC corrected upward.
• SerDes / CDR domain: clock recovery and lane timing; sensitive to temperature, supply noise, and channel variability.
• MAC / core domain: packet pipeline timing; interacts with queueing and completion cadence (p99 exposure).
• Timestamp domain (if present): defines where time is captured; cross-domain alignment is a bounded error term.

How jitter becomes field symptoms (evidence-first)

• Margin tightens → lane errors rise → FEC corrected increases (throughput still “looks fine”).
• Worsening margin → FEC uncorrected / symbol errors → drops/timeouts → throughput jitter appears.
• Training sensitivity → repeated training adjustments or retraining episodes, especially near temperature/power edges.
• Practical reading: when line-side module readings are stable but FEC/BER and retraining trends worsen, prioritize clock/power/thermal inside the NIC before blaming the network.

CDR and bring-up stability (why “link up” is not “link stable”)

• CDR role: tracks timing variation and keeps sampling aligned; changing conditions increase lock difficulty.
• What instability looks like: BER drift, higher FEC pressure, occasional lane deskew issues, and sporadic retraining.
• What to correlate: lane error trend + FEC corrected/uncorrected split + retraining events (time aligned with temperature or power changes).

Hardware timestamp boundary (tap point + error components)

Power rails (concept level): how droop turns into link instability

• Common rail categories: core · SerDes · PLL · IO (names vary, roles are consistent).
• Failure chain: rail droop or noise → CDR/SerDes margin tightens → BER and FEC corrected trend upward → p99 spikes follow.
• How to validate: correlate voltage/current trends with FEC/BER and retraining events (time aligned).

Thermal hotspots and throttling: why it “looks like a network problem”

• Hotspots: SerDes edge · ASIC core · PLL area · module cage (thermal coupling matters).
• Throttling signatures: step-like throughput ceilings, rising error trends near a temperature knee, and clustered retraining episodes.
• Evidence-first: when throttling appears, look for synchronized changes in temperature, power, and error counters.

Telemetry categories (what to read, and why it proves the layer)

Optics module (CMIS/DDM) triage: module vs channel vs NIC

• Module-first: module readings drift abnormally and align with errors → prioritize module seating, module thermals, or module health.
• NIC-first: module readings remain stable while NIC FEC/BER/retrain aligns with NIC temperature or rail stress → prioritize NIC thermal/power margin.
• Neither is obvious: module and error trends are clean but performance jitters → return to queue/PCIe evidence chains (previous chapters).