Timing Cards & Modules: Integrated PLL, Cleaner, Fanout & Alarms
← Back to:Reference Oscillators & Timing
A timing card/module is a system-level “time backbone” that turns uncertain time sources into verified, maintainable, multi-domain clocks—delivering controlled jitter, repeatable phase alignment, disciplined holdover, and actionable alarms for production and field operations.
This page explains how to design, integrate, validate, and operate timing cards/modules using measurable checks, acceptance criteria, and selection logic—so timing performance stays predictable across temperature, power events, and failovers.
Definition: What is a Timing Card / Module?
A timing card/module is a deployable clock subsystem that turns one or more time references into controlled clock outputs with repeatable alignment, auditable alarms, and maintainable holdover. It is designed to be integrated, validated, and operated as a system capability (not a single IC).
The system pain it fixes (why it exists)
Impact: downstream lock/quality becomes unpredictable.
Impact: multi-card systems lose deterministic timing.
Impact: field debugging turns into guesswork.
Impact: no stable behavior can be guaranteed during outages.
Timing card vs timing module vs “clock tree board”
Often missing: disciplined holdover + auditable alarms as a closed loop.
Typical fit: constrained space, moderate telemetry, integrated platforms.
Typical fit: systems that require operational SLAs and audit trails.
Typical inputs/outputs (card-level view)
- GNSS / ToD + 1PPS (absolute time anchor)
- PTP hardware-timestamped port (network time feed)
- SyncE recovered clock (transport-grade frequency)
- 1PPS / 10 MHz (lab or system reference)
- Ref clocks (multi-output fanout to endpoints)
- SYSREF / sync pulses (deterministic alignment hooks)
- 1PPS out (system time marker)
- ToD distribution (time-of-day delivery to systems)
When to Use It: Discrete vs Card/Module (Decision Triggers)
Choose a timing card/module when clock alignment, time stability, alarms, and failover must behave like a system-level SLA. If the system can tolerate manual tuning and limited observability, a discrete clock tree can be more cost-effective.
Decision triggers (engineer-first)
- Multi-chassis / multi-card alignment must be repeatable (ps–ns class), including after reboot or reseat.
- Time/clock health must be auditable: alarms, timestamps, counters, and clear state transitions are required for operations.
- Redundancy and failover are required (A/B references, hitless switching goals, defined recovery behavior).
- Production consistency matters: calibration parameters must be fixed, and acceptance tests must be repeatable at scale.
- Alignment need: is the requirement “repeatable after reboot” or “stable over hours across temperature”?
- Observability need: are time events required to be logged with timestamps and state transitions?
- Failover need: is a defined maximum phase transient required during switching?
- Production need: is there a fixed acceptance workflow with stored calibration data and audit trails?
Typical fits (fast sanity check)
Common cost of picking the wrong level
- Field faults become non-reproducible and hard to audit.
- Alignment drifts or resets are difficult to bound.
- Production variability increases without a fixed validation template.
- Unnecessary BOM/power/complexity and longer bring-up time.
- More configuration states to validate and operate.
- The real bottleneck may be elsewhere (layout, power noise, or endpoint constraints).
Internal Architecture: The “Timing Stack” Inside
A timing card/module behaves like a small timing subsystem. The fastest way to understand it is to separate three parallel planes: Clock plane (what time flows through), Control plane (how behavior is configured), and Telemetry plane (what can be measured, alarmed, and audited).
The three-plane mental model (subsystem view)
Five functional bricks (role → interfaces → failure signature)
Interfaces: local osc, (optional) tuning input, temperature sensing.
Signature: temperature-correlated drift, warm-up behavior changes.
Interfaces: reference input, loop profile, lock detect.
Signature: abnormal lock time, phase steps on mode changes.
Interfaces: per-output enable, level select, delay trim.
Signature: one output degrades due to loading/termination mismatch.
Interfaces: taps, thresholds, debounce, event timestamps.
Signature: alarm storms (too sensitive) or missed drift (too loose).
Interfaces: MCU/FPGA, EEPROM, mgmt links, firmware control.
Signature: version drift or config mismatch causing behavior changes.
Inputs & References: Time Sources and Isolation Strategy
Cards/modules rarely rely on a single input. Multiple time sources are classified, then passed through health gates, and finally selected by priority + switching policy. The goal is predictable behavior during degraded inputs, not maximum sensitivity.
Input types (by timing meaning, not by connector)
Health gates (sanity checks that prevent “bad-but-preferred” inputs)
Isolation strategy (minimum set that prevents cross-domain contamination)
Disciplining & Holdover: Control Modes and What “Good” Looks Like
The real value of a timing card/module is not “having clocks,” but bounded time behavior when references degrade or disappear. This section defines control modes as observable states, focuses on logs and measurable curves, and provides a reusable acceptance template (placeholders must be set by system requirements).
Three modes (defined by behavior, not by control theory)
Entry: no valid external reference passes health gates.
Observable: phase error drifts according to local stability.
Log: mode, temp, tune word (if any), drift metrics.
Entry: reference selected + stable window satisfied.
Observable: phase error converges into a stable band.
Log: active_ref, loop profile, lock time, phase/freq error.
Entry: reference fails gates; holdover policy asserted.
Observable: phase error grows within a defined envelope.
Log: last-good ref stats, temp, predicted drift, alarms.
What “good” looks like (curves and signatures)
- Phase error moves into a steady band and stays there.
- No periodic “time bumps” tied to mode updates.
- Recovery does not produce a visible step beyond policy limits.
- Phase error grows predictably (mostly smooth slope).
- Temperature changes shift slope, but remain bounded.
- Alarms reflect genuine degradation, not noise flapping.
- Phase steps during switching or recovery (“time jump”).
- Holdover slope changes abruptly with minor temperature swings.
- Alarm storms caused by missing hysteresis/stable window.
Acceptance templates (placeholders; set by system requirements)
Metric: peak/percentile of phase_error(t).
Pass: |phase_error| ≤ Y within X hours.
Pass: |freq_error| ≤ Z (Z depends on wander budget).
Pass: no phase step > A, alarms clear within B.
Output Clocking: Domains, Alignment, and Distribution Rules
Output clocking is domain management. A timing card/module must feed multiple endpoints while keeping skew, phase continuity, and configuration traceability under control. This section describes a practical approach without relying on interface-specific standards.
Output domains (organized by meaning)
Alignment strategy (repeatable after reboot/reseat)
Termination & levels (card-level rules, no protocol dependence)
Output acceptance templates (placeholders)
Monitoring & Alarms: What to Measure, What to Log, How to Act
A timing card/module is operated as a closed-loop, observable system. Alarms are not “strings”; they are events with context, confidence (debounce/confirm), and a bounded action policy. This section focuses on on-card monitoring points and operational logic (not external NMS).
Alarm classes (organized by impact)
Impact: alignment/SLA risk.
Typical action: degrade or switch (policy-gated).
Impact: increased risk of drift and switching.
Typical action: tighten gates, change profile, prepare failover.
Impact: performance collapse or false positives if unhandled.
Typical action: protective degrade + strong logging.
Event chain (Detection → Debounce → Confirm → Report → Act)
Action policy (bounded, auditable)
Minimum log fields (to reproduce and audit decisions)
Redundancy & Failover: Hitless Switching and Guard Paths
Redundancy is not “having a backup.” It is a measurable switching policy that keeps critical domains stable under reference faults. This section describes on-card A/B references, guard paths, and acceptance templates for hitless behavior (placeholders set by system requirements).
Redundancy targets (what is actually duplicated)
Guard paths (keep the backup “ready and comparable”)
Hitless switching (defined by allowed transients)
System Integration: Power, Thermal, EMC, and Backplane Reality
Timing cards/modules are unusually sensitive to supply noise, return paths, connector/backplane behavior, and thermal gradients. This section focuses on the integration-specific failure modes that typically turn “bench-good” into “system-bad”.
Power: low-noise rails, domain partitioning, filters, and boot sequencing
Thermal: gradients, airflow, and sensor placement
Backplane & chassis: returns, common-mode noise, reflections, and cable length
Integration checklist (risk → quick check → fix → pass)
Quick check: compare jitter/phase_error with management traffic OFF vs ON; log Vrail ripple at the same time.
Fix: separate rails, add targeted filtering, re-route returns to keep digital currents out of analog island.
Pass: incremental jitter/phase_error change ≤ ΔJ / ΔP (placeholders set by system budget).
Quick check: correlate phase_error slope with temp slope and fan/airflow states; look for periodic modulation.
Fix: move oscillator away from hot spots/pulsed airflow; relocate/duplicate sensors near the drift source.
Pass: drift slope vs temperature stays within the holdover envelope (system-defined).
Quick check: measure at port and at endpoint; compare skew/jitter with direct-cable vs backplane path (one-variable change).
Fix: tighten terminations/levels, add common-mode control where required, enforce cable length rules for aligned domains.
Pass: endpoint jitter/skew meets budget with backplane installed (not only on bench).
Validation & Acceptance: Bench Tests That Actually De-risk Deployment
Acceptance should be a repeatable engineering flow: define baselines, lock measurement windows, prove one-variable deltas, and record context. The goal is not “pretty plots” but de-risking deployment by isolating power/backplane/thermal effects and verifying modes (holdover and failover) with auditable evidence.
Test setup rules (so results remain comparable)
Output phase noise / jitter (measure points + windows + baselines)
Phase alignment (multi-channel + cross-card + temperature deltas)
Holdover (loss-of-reference, thermal change, and aging trend)
Failover (transients, alarm correctness, recovery time)
Engineering Checklist: Bring-up → Production → Field
This section turns the timing-card “capabilities” into executable stage-gates. Each gate contains only actions, required evidence, and measurable pass criteria (placeholders such as X/Y/Z/T must be set by the system timing budget and SLA).
Bring-up (Lock → Mode transitions → Output sanity)
- Lock check: verify reference selection, lock indicators, and “healthy” state under nominal input.
- Mode walk: execute Free-run → Discipline → Holdover transitions; record the exact trigger used.
- Output electrical: validate standard + termination at both the port and the endpoint (HCSL/LVDS/LVPECL/LVCMOS).
- Domain mapping: confirm each output domain is routed to the intended consumer (system clock / refclk / sysref / pps / ToD).
- Alarm sanity: inject a controlled fault (reference removed / degraded) and confirm alarm + log closure.
- Config snapshot (register dump / profile ID), firmware version, and build hash.
- Lock timeline and mode transition timestamps (with input ref quality label).
- Endpoint measurement screenshots (level + termination + jitter/phase delta vs baseline).
- Alarm event entries for each injected fault + recovery.
- Lock time ≤ T_lock and remains locked for ≥ T_stable under nominal conditions.
- Mode transitions produce no endpoint faults; phase transient ≤ Δφ_switch.
- Additive jitter at endpoints ≤ ΔJ_budget (window defined by the system spec).
- Alarm injection produces: detect → debounce → confirm → report → action, all within ≤ T_alarm.
Production (Calibration → Sealing → Sampling plan)
- Calibration items: temperature compensation coefficients, frequency offset trim, delay table (channel alignment), holdover model parameters.
- Sealing: write EEPROM/flash, verify CRC/signature, lock critical fields; bind to serial number.
- Golden baseline: compare each unit against a golden reference (relative deltas preferred over absolute).
- Sampling strategy: define lot sampling rate and re-test triggers (process change / firmware change / component swap).
- Calibration record: coefficients + delay table + firmware/build ID.
- Acceptance summary: jitter/phase delta vs golden; holdover short test snapshot.
- Configuration hash + EEPROM CRC report (pass/fail).
- Config sealing succeeds (CRC/signature valid) with version match.
- Relative deltas vs golden: jitter ≤ ΔJ_golden, skew ≤ ΔSkew_golden.
- Lot statistics meet thresholds: out-of-family rate ≤ R_oof, drift shift ≤ ΔDrift_lot.
Field (Alarm policy → Log rotation → OTA upgrade + rollback → Drills)
- Alarm policy: define warning/degrade/failover thresholds and the exact action taken for each state.
- Log rotation: set retention, roll size, and “must-keep” fields (timestamp, ref quality, loop mode, phase/freq error, temp, rail status, switch events).
- Remote upgrade: enforce pre/post checks; require rollback trigger conditions and a validated recovery path.
- Periodic drills: scheduled failover drill, holdover drill, and alarm chain drill (black-box success criteria).
- Drill reports: trigger → detected → action → recovered timeline.
- Upgrade reports: pre-check snapshot, post-check snapshot, and rollback record (if used).
- Degrade/failover logs with root-cause tags (ref degraded, temp excursion, rail anomaly).
- Alarm-to-action latency ≤ T_action; false-trigger rate ≤ R_false.
- Failover drill: phase transient ≤ Δφ_hitless; service impact = “none” by system definition.
- Holdover drill: phase error envelope ≤ E_holdover(t) over X hours.
- Rollback completes within ≤ T_rollback and restores last-known-good timing profile.
Applications & IC Selection Notes (Card-Level Selection Logic)
This is not a shopping list. It is a card-level selection method: required capabilities → system constraints → spec-writing rules. The material numbers below are starting points for datasheet lookup and lab validation; package, lifecycle, and availability must be verified.
A) Selection dimensions (capabilities to specify)
- Inputs: GNSS / PTP (hardware timestamp) / SyncE recovered clock / 1PPS / 10 MHz; multi-source arbitration needed or not.
- Outputs: count + standards (HCSL/LVDS/LVPECL/LVCMOS) + special domains (SYSREF / PPS / ToD).
- Alignment: on-card channel skew budget and cross-card phase alignment target (ps–ns class).
- Holdover: define an error envelope over time E_holdover(t) (phase vs time), not a single number.
- Alarms: required signals/telemetry (GPIO/I²C/host) and a graded policy (warning/degrade/failover).
- Redundancy: A/B ref, hitless definition (Δφ_hitless, Δf_hitless), and drill requirements.
B) System constraints (what silently breaks timing)
- Thermal reality: airflow stability, gradients across OCXO/TCXO/MEMS area, and sensor placement for control decisions.
- Power noise: rail noise density, load steps, and isolation between analog timing island vs digital control plane.
- Backplane/cabling: reflections, common-mode injection, and ground potential differences across chassis.
- Operations: remote-only vs local serviceability, allowed downtime for updates, required log retention and audit trail.
C) Risk notes (how to write specs that are testable)
- Typical vs worst-case: require worst-case across temperature, rails, and chosen reference inputs; typ-only specs are not deployable.
- Bind every number to a window: RMS jitter must state integration limits; phase/ToD must state averaging time and measurement method.
- Budget alignment: card targets must roll up to a system budget (converter SNR, SerDes tolerance, network SLA).
- Auditability: every “degrade/failover” decision must be provable by logs (fields + timestamps + thresholds).
- Lifecycle reality: check PCNs, NRND/obsolete status, and second-source plan for long-life programs.
D) Reference material numbers (starting points only)
Grouped by function blocks commonly found in timing cards/modules. Verify package suffix, grade, lifecycle, and timing performance in the intended measurement window.
AD9545(ADI) — clock synchronizer / DPLL platform :contentReference[oaicite:13]{index=13}ZL30772(Microchip) — packet/SyncE DPLL class device :contentReference[oaicite:14]{index=14}
Si5345(Skyworks/Silicon Labs line) — jitter attenuator family :contentReference[oaicite:15]{index=15}LMK04828(TI) — jitter cleaner + distribution class device :contentReference[oaicite:16]{index=16}AD9528(ADI) — JESD clock generator class device :contentReference[oaicite:17]{index=17}HMC7044(ADI) — dual-loop jitter attenuator class device :contentReference[oaicite:18]{index=18}
ADCLK948(ADI) — low-jitter fanout buffer family :contentReference[oaicite:19]{index=19}LMK00334(TI) — clock buffer / level translator class device :contentReference[oaicite:20]{index=20}
ADM7150(ADI) — ultralow-noise LDO class device :contentReference[oaicite:21]{index=21}
ADuM1250(ADI) — I²C isolator class device :contentReference[oaicite:22]{index=22}TMP117(TI) — digital temperature sensor class device :contentReference[oaicite:23]{index=23}
ZED-F9T(u-blox) — timing GNSS module :contentReference[oaicite:24]{index=24}LEA-M8T(u-blox) — timing GNSS module family :contentReference[oaicite:25]{index=25}mosaic-T(Septentrio) — GNSS timing receiver module :contentReference[oaicite:26]{index=26}LC29H(Quectel) — dual-band GNSS module series :contentReference[oaicite:27]{index=27}
Recommended topics you might also need
Request a Quote
FAQs (Troubleshooting Only) + JSON-LD
These FAQs only close long-tail troubleshooting within the timing card/module boundary. Each answer is a data-driven 4-line checklist with measurable probes and pass criteria placeholders (X/Y/Z/T) that must be set by the system timing budget and SLA.
GNSS says “locked” but 1PPS phase still slowly drifts—first log which two counters?
Likely cause: GNSS lock indicates tracking, but timing quality is degraded so the disciplining integrator accumulates slow phase error.
Quick check: Trend phase_error_slope (ps/s) and freq_steer_word (or equivalent DAC/FCW) over ≥ T hours while recording ref_quality.
Fix: Tighten reference validation to reject “noisy-lock” and/or increase averaging/hysteresis; if GNSS receiver is marginal, validate with a timing-grade module (e.g., u-blox ZED-F9T / LEA-M8T) before changing loop targets.
Pass criteria: Over T hours the absolute phase drift rate stays ≤ X ps/s and the steer word remains within ±Y% of its nominal range without repeated ref-quality drops.
Holdover is fine at room temp but fails across temperature—what trend plot reveals it fastest?
Likely cause: Holdover model is under-calibrated versus temperature (or sensor placement misses the actual oscillator gradient), so prediction error spikes during thermal transitions.
Quick check: Plot phase_error(t) together with temp_gradient = temp_osc - temp_board and holdover_residual during a controlled temp sweep.
Fix: Re-run temperature calibration and update coefficients/EEPROM; if the platform requires stronger holdover, validate DPLL/holdover devices (e.g., ADI AD9545 or Microchip ZL30772) with correct sensor placement and airflow constraints.
Pass criteria: Across the specified temperature range, holdover phase error remains inside the envelope E_holdover(t) ≤ X for at least T hours after reference loss.
Cleaner output jitter is great, yet downstream FPGA occasionally loses lock—probe what at the connector?
Likely cause: The endpoint is failing on electrical integrity (swing/common-mode/termination/reflections) even though the source jitter is low.
Quick check: At the card connector measure differential swing, common-mode level, and reflection/ringing (overshoot/undershoot) with the intended termination populated at the endpoint.
Fix: Correct the output standard and termination, reduce stub length/return discontinuities, and if loading is heavy use a dedicated fanout/buffer stage (e.g., ADI ADCLK948 or TI LMK00334) per domain.
Pass criteria: FPGA lock drop count equals 0 over T hours and connector waveform meets limits (e.g., overshoot/undershoot ≤ X mV and stable common-mode within ±Y mV).
After failover, alignment is off by a fixed offset—what does that imply about delay table vs phase trim?
Likely cause: A fixed post-switch offset typically indicates an unaccounted fixed path latency (delay table mismatch) rather than random phase noise or lock instability.
Quick check: Compare delay_table_id and phase_trim_value pre/post failover and confirm the measured offset is constant (±X ps) across repeated switches.
Fix: Calibrate and store separate delay tables for main/backup paths (and for each output domain) and ensure the switch sequence applies the correct table before declaring “in-service.”
Pass criteria: After any failover event, residual fixed offset ≤ X ps and channel-to-channel skew remains within the budget ≤ Y ps without manual re-trim.
Periodic “time bump” every N minutes—how to tell disciplining step vs software timestamp jump?
Likely cause: The bump is either a deliberate phase step from disciplining policy or a discontinuity introduced by the timestamp/ToD distribution path.
Quick check: Correlate the bump timestamps with phase_step_event_count/discipline_step_log and the host/ToD event log; if only the host log jumps, the source is software.
Fix: If it is disciplining, switch to continuous steering or reduce step magnitude and increase smoothing; if it is software, enforce monotonic timestamp handling and audit the ToD update transaction.
Pass criteria: No phase step exceeds X ps in magnitude and ToD/timestamps remain monotonic with max discontinuity ≤ Y ns over T hours.
PTP input looks stable but card switches ref anyway—what health gate threshold is likely too tight?
Likely cause: Health gating is rejecting PTP on transient metrics (delay variation, offset spikes, or missing-stamp bursts) due to insufficient debounce/hysteresis.
Quick check: Inspect the last 60–300 s before switch: switch_reason, ptp_offset_peak, and missing_stamp_count versus the configured thresholds.
Fix: Add hysteresis and increase confirmation window for PTP degrade, and align thresholds to the system wander budget rather than instant jitter snapshots.
Pass criteria: With stable PTP, ref switching does not occur for ≥ T days and any switch is preceded by metrics exceeding thresholds continuously for ≥ X seconds.
Why does enabling SSC reduce EMI but break one output domain—what compatibility check first?
Likely cause: The affected endpoint PLL/CDR does not tolerate the applied spread depth/rate, even if other domains remain fine.
Quick check: Verify SSC is enabled on the failing domain only, then measure modulation depth (ppm) and modulation rate at that output and compare to the endpoint tolerance spec.
Fix: Disable SSC on sensitive domains while keeping it on EMI-critical ones, or route the sensitive domain through a non-spread path (typical clock-tree uses jitter attenuators like Si5345-class or conditioners like LMK04828-class with per-domain policy).
Pass criteria: EMI peak reduction meets target while the sensitive endpoint shows 0 lock-loss events over T hours and phase/frequency excursions remain ≤ X/Y.
Multi-output skew is good at boot but degrades over hours—what thermal gradient check?
Likely cause: Channel delay elements and routing experience drift under thermal gradients, so skew slowly walks even if the source remains locked.
Quick check: Log per-channel skew_error alongside temp_osc and temp_board, then compute correlation with temp_gradient.
Fix: Improve airflow/heat spreading, relocate/duplicate sensors, and enable periodic phase re-trim if supported (ensure trims are logged and bounded).
Pass criteria: Over T hours and across operating temperatures, skew drift stays ≤ X ps (p-p) and does not correlate strongly with temperature (|r| ≤ Y).
Alarm storms appear during power events—what to filter vs what must be immediate?
Likely cause: A rail transient triggers many dependent alarms simultaneously, and missing policy separation causes repeated debounce/retry loops.
Quick check: Align timestamps of rail_uv/ov_event (or brownout) with the alarm burst rate (alarms/min) and verify whether resets coincide with switch_event.
Fix: Debounce and rate-limit “secondary” alarms during known power-sequencing windows, but keep “hard” timing integrity alarms (loss-of-lock, missing pulse) immediate with clear single-shot actions.
Pass criteria: During power events, alarm rate ≤ X alarms/min with no repeated oscillation, and critical alarms still assert within ≤ Y ms when truly violated.
One channel shows higher jitter than others—how to isolate fanout loading/termination issue quickly?
Likely cause: The “bad” channel is seeing different loading/termination or crosstalk, increasing deterministic jitter and edge distortion.
Quick check: Swap endpoint loads between two outputs and see whether the higher jitter follows the load, and measure connector reflections (ringing amplitude) on the affected path.
Fix: Normalize termination and loading, reduce stubs, and use a robust per-output buffer if needed (e.g., ADCLK948 / LMK00334 class fanout) to isolate domains.
Pass criteria: Channel-to-channel RMS jitter delta ≤ X fs (in the defined integration window) and reflection/ringing at the connector is ≤ Y mV (p-p).
Phase monitor shows noise but system works—what measurement bandwidth/window mistake is common?
Likely cause: The monitor is integrating the wrong band/timebase (mixing jitter with wander or using inconsistent averaging), producing “noise” that is not relevant to the system budget.
Quick check: Record the analyzer integration limits (f1..f2) and averaging time, then re-run using the exact window defined in acceptance (same reference path and trigger).
Fix: Standardize a single measurement recipe (window + averaging + reference) and validate it against a known-good baseline trace before concluding a hardware issue.
Pass criteria: With the correct window, measured RMS jitter/phase stats fall within the system budget ≤ X and correlate with observable system behavior (no false-fail alerts).
Firmware update changed timing behavior—what “golden log snapshot” should you compare?
Likely cause: Default profiles or calibration mappings changed (loop bandwidth, thresholds, delay tables), shifting behavior even if hardware is unchanged.
Quick check: Compare a golden snapshot set: fw_version, profile_id, config_hash, plus loop_mode, ref_quality, and summary stats of phase_error/freq_error under the same input conditions.
Fix: Restore the prior timing profile, migrate EEPROM calibration fields explicitly, and re-run a short acceptance suite; if the design uses DPLL/cleaner blocks, validate config equivalence for devices like AD9545, ZL30772, Si5345, LMK04828 class parts.
Pass criteria: Post-update deltas versus golden remain within limits (jitter ≤ X, skew ≤ Y, holdover envelope unchanged) and no new unexpected switch events occur over T hours.