Market Data Feeds That Don't Go Dark: A Three-Tier Fallback Design

A trading system is only as good as the prices it sees. Strategies get the attention, but in our experience more live incidents start in the data layer than in the logic: a terminal that stopped publishing at 3am, a vendor API that returned yesterday's bar with today's timestamp, a symbol that was renamed by a broker. This piece describes the market data feed architecture we run for our own scanners and analytics. It uses three sources in priority order, detects staleness explicitly, tags every bar with where it came from and repairs gaps automatically. None of it is exotic. All of it is the difference between a system that degrades quietly and one that goes dark.

The problem with a single feed

Single-source designs fail in predictable ways:

• Silent stalls. The connection stays open, no error is raised, and no new data arrives. Anything polling the latest bar keeps reading the last good value and assumes the market is flat.
• Partial outages. One symbol or asset class stops updating while the rest continue.
• Bad data. Duplicated bars, out-of-order timestamps, zero or negative prices after a rollover, or a bar that is later revised.
• Upstream changes. A broker renames a symbol, changes contract specifications or shifts server time.

Redundancy helps only if the system can tell which source is healthy right now and record which source produced each piece of data. Otherwise you have two broken feeds instead of one.

The three tiers

Why the broker terminal is primary

This is the decision people question most, since a commercial vendor usually has better uptime than a desktop terminal. The reason is consistency. Our strategies are backtested and executed against a specific broker's price history. Contract-for-difference prices, synthetic indices and spot metals differ between brokers in spread, session boundaries and sometimes bar construction. If the live scanner reads vendor prices while the backtest used broker prices, the live signals are generated on data the strategy was never tested on. Small differences at bar boundaries are enough to flip a breakout or a threshold cross.

So the primary is a small publisher running inside the terminal. It reads completed bars for a configured symbol list, stamps them, and pushes them to the database. The scanners then read from the database, never directly from any one source.

Secondary and tertiary

The secondary is a server-side process that connects to a broker API from a container. It produces the same bar schema and writes to the same table. Its job is to keep data flowing if the terminal machine reboots, loses connectivity or freezes.

The tertiary is a commercial data vendor. It covers broadly the same instruments for liquid markets but not broker-specific synthetic products. It is the last resort for live data and the first choice for historical gap repair where the broker sources have nothing.

Architecture

The flow is simple:

1. Each publisher (terminal, server, vendor poller) writes bars into one bars table keyed on (symbol, timeframe, bar_open_time).
2. Each publisher also writes a heartbeat row every 30 seconds, whether or not new bars exist.
3. A supervisor process reads heartbeats and bar freshness, decides which source is active for each symbol group and records that decision.
4. Scanners and dashboards read only from the bars table and the supervisor's status, never from publishers directly.
5. A backfill worker scans for gaps on a schedule and fills them from the highest-priority source that has the data.

Decoupling consumers from producers through the database is the key move. Scanners do not know or care which source is live. Failover is a change in which publisher is permitted to write, not a change in every consumer.

Heartbeats and staleness detection

An open socket is not evidence of health. We track two separate signals per source:

• Heartbeat age: time since the publisher last reported it was alive. This catches crashed processes and dead hosts.
• Data age: time since the newest bar's open time, compared with what the market calendar says it should be. This catches the silent stall, where the process is alive but no data is moving.

Data age needs a market calendar. A one-minute bar feed for a stock index CFD is not stale on Saturday. A crypto feed is. A gold feed has a daily break. The threshold for each symbol group is a multiple of its bar interval during open sessions, and suspended outside them.

The supervisor logic, simplified:

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

PRIORITY = ["terminal", "server", "vendor"]
HEARTBEAT_MAX_AGE = timedelta(seconds=90)


@dataclass
class SourceHealth:
    source: str
    last_heartbeat: datetime | None
    last_bar_open: datetime | None


def is_healthy(h: SourceHealth, now: datetime, bar_interval: timedelta,
               market_open: bool, stale_multiple: int = 3) -> bool:
    if h.last_heartbeat is None or now - h.last_heartbeat > HEARTBEAT_MAX_AGE:
        return False
    if not market_open:
        return True  # no data expected, heartbeat alone is enough
    if h.last_bar_open is None:
        return False
    # The newest completed bar should open no more than N intervals ago
    return now - h.last_bar_open <= bar_interval * (stale_multiple + 1)


def choose_source(health: dict[str, SourceHealth], current: str,
                  bar_interval: timedelta, market_open: bool) -> str | None:
    now = datetime.now(timezone.utc)
    for source in PRIORITY:
        h = health.get(source)
        if h and is_healthy(h, now, bar_interval, market_open):
            if source != current:
                alert(f"data source change: {current} -> {source}")
            return source
    alert("all data sources unhealthy", severity="critical")
    return None

Two details matter in practice. First, failback to a higher-priority source should be deliberate: require the primary to be healthy for several consecutive checks before switching back, so a flapping terminal does not cause the active source to oscillate. Second, when no source is healthy, the supervisor returns nothing and scanners stop generating signals. Trading on stale data is worse than not trading.

Source tagging per bar

Every bar row carries a source column, plus ingested_at (server time when written). This is cheap and pays for itself many times over:

• When a live signal looks wrong, you can see immediately whether it fired on terminal, server or vendor data.
• Performance attribution can exclude or flag periods that ran on fallback data.
• Data quality reports per source show which publishers drift, duplicate or lag.

When a higher-priority source later supplies a bar that a lower-priority source already wrote, the higher-priority version wins and the source tag updates. The rule is explicit: a write may replace an existing bar only if its source has equal or higher priority.

Idempotent upserts

Publishers retry. Networks duplicate. Terminals restart and resend their last N bars. Every write must therefore be safe to repeat. In Postgres this is a conditional upsert:

INSERT INTO bars (symbol, timeframe, bar_open_time, open, high, low, close,
                  volume, source, source_priority, ingested_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, now())
ON CONFLICT (symbol, timeframe, bar_open_time) DO UPDATE
SET open = EXCLUDED.open, high = EXCLUDED.high, low = EXCLUDED.low,
    close = EXCLUDED.close, volume = EXCLUDED.volume,
    source = EXCLUDED.source, source_priority = EXCLUDED.source_priority,
    ingested_at = EXCLUDED.ingested_at
WHERE bars.source_priority >= EXCLUDED.source_priority;

Lower source_priority numbers mean higher priority (terminal is 1). The WHERE clause stops a vendor write from overwriting a terminal bar, while still allowing the terminal to correct its own bar if it publishes a revision. Only write completed bars; the forming bar belongs in a separate, overwritable live table, or every consumer needs to know which row is unfinished.

Gap detection and backfill

Even with failover, gaps appear: a source was healthy by heartbeat but missed a few bars, or all sources were down for a period. A backfill worker runs on a schedule and:

1. Builds the expected bar timestamps for each symbol from the market calendar.
2. Compares them with stored bars to find missing ranges.
3. Requests each range from sources in priority order, stopping at the first that returns complete data.
4. Writes through the same idempotent upsert, so a backfill can never downgrade an existing bar.
5. Records unfillable gaps in a table rather than silently interpolating.

We do not forward-fill prices to hide gaps. A missing bar is a fact the strategy and the analyst should see.

Clock skew and time zones

Most MT5 broker servers run on a server time that is not UTC, commonly UTC+2 or UTC+3 depending on daylight saving, and terminal bar timestamps are in that server time. Mixing them naively with a UTC vendor feed shifts every bar by two or three hours, which is invisible in a chart and fatal in a join.

The rules we follow:

• Convert to UTC at the publisher, before anything is written. Store the original server offset alongside if you need to reconstruct broker-time bars.
• Derive the broker offset from a known reference rather than hardcoding it, and recheck it around daylight saving changes in both the broker's and your own jurisdiction.
• Synchronise every host with NTP and alert if the host clock drifts beyond a second or two. Heartbeat ages are meaningless if the clocks disagree.
• Daily bars are the hardest case. A broker that closes its day at 17:00 New York time produces daily bars that do not match a vendor's midnight UTC daily bar. Store intraday bars and build higher timeframes yourself with one explicit session definition.

Symbol mapping across brokers

The same instrument has different names everywhere: XAUUSD, XAUUSD.r, GOLD, XAUUSDm, a vendor-specific ticker. Symbol suffixes, contract sizes and digit precision differ by broker and sometimes by account type.

Keep a mapping table with one canonical internal symbol and a row per source carrying that source's ticker, price multiplier, digits and session template. Publishers translate to canonical symbols before writing. Never let consumers see broker-specific names.

Synthetic indices

Some brokers offer synthetic or proprietary instruments that exist only on their own servers. No commercial vendor carries them and no other broker prices them the same way. For these, the tertiary tier does not exist. The mapping table should mark such symbols as single-source or dual-source, and the supervisor should treat loss of the broker sources as a hard stop for those instruments rather than an invitation to substitute a lookalike.

Alerting

Alerts need to be few enough that people read them. We alert on:

• Source change: any failover or failback, with symbol group, from and to.
• All sources down for any open market: critical.
• Staleness without failover: data age breaching threshold while heartbeats look healthy, which usually means a publisher bug.
• Unfillable gaps after backfill.
• Clock drift beyond tolerance on any host.
• Cross-source divergence: where two sources both publish a symbol, compare closes and alert when the difference exceeds a configured number of ticks for several consecutive bars. This catches mapping and multiplier errors early.

Alerts go to a messaging channel the team actually watches, with enough context in the message to act without opening a dashboard.

Key takeaways

• Put the source your strategies were tested on as primary, even if it is less robust, and make redundancy cover its weaknesses.
• Decouple consumers from producers. Scanners read a single table, not a feed.
• Track heartbeat age and data age separately, with thresholds driven by a market calendar.
• Tag every bar with its source and only let equal or higher priority sources overwrite.
• Make every write an idempotent upsert, and backfill gaps through the same path.
• Normalise time to UTC and symbols to canonical names at the publisher.
• Stop signal generation when no healthy source exists.

How we apply this

This design runs underneath our own scanners and analytics, operated around the clock. It came from incidents, not theory: each component above exists because its absence once cost us a missed or false signal. We build client data pipelines on the same principles, using Postgres, containerised publishers and broker and vendor integrations we have shipped in production.

If your trading or analytics stack relies on one feed and hope, see our data engineering services or browse the systems we have built and operate.

This article is for information and education only and is not investment advice. See our risk disclaimer.