How to backtest on a survivorship-free universe

The short way

Two datasets, not one. The first is price history for every stock that traded, including the ones that no longer do. The second is who belonged to the universe on each date — usually an index, because "the S&P 500 in 2008" is the universe most tests mean. A vendor that sells the first without the second gives you a survivorship-free haystack and no way to know which straws were eligible; the reverse gives you a list of names with no prices behind half of them. Why the second dataset costs money at all is in the guide to historical index constituents.

The cheapest complete version on a Windows machine is Norgate Data at Platinum ($630 a year for US stocks), which is the first tier with delisted securities and constituent history, feeding the free Zipline engine through Norgate's own zipline-norgatedata package. The bundle is the whole universe, living and dead; membership is a filter applied per date inside the pipeline, not a list fixed at the start:

# C:\Users\<you>\.zipline\extension.py  (Windows, where the Norgate updater runs)
from zipline_norgatedata import register_norgatedata_equities_bundle

register_norgatedata_equities_bundle(
    bundlename='norgatedata-sp500',
    symbol_list=['$SPXTR'],                  # total-return index, for the benchmark
    watchlists=['S&P 500 Current & Past'],   # every member since 1990, dead ones included
    start_session='1990-01-01',
)

# in the algorithm
from zipline.pipeline import Pipeline
from zipline_norgatedata.pipelines import NorgateDataIndexConstituent

def make_pipeline():
    member = NorgateDataIndexConstituent('S&P 500')
    return Pipeline(columns={'member': member}, screen=member)

zipline ingest -b norgatedata-sp500 builds it. Norgate's own bundle example puts the S&P 500 current-and-past list at around 1,800 securities, against the roughly 500 you would get from today's list, and every one of the extra 1,300 is a name a naive backtest could never have bought.

On a Mac, or without wanting to run a data pipeline at all, QuantConnect's cloud does the equivalent with nothing to install: its US equity security master carries delistings, mergers and ticker changes from January 1998, and LEAN maps each security to the same identity whatever its ticker was that year.

What the options are

A data subscription feeding a desktop engine. Norgate Data has plugins of its own for AmiBroker, WealthLab and Zipline, and RealTest reads it through that product's own integration. Its constituent history is exposed as a 1-or-0 time series per security per index, which is the right shape: the engine asks "was this a member today" on every bar instead of consulting a list. Silver and Gold, the two cheaper tiers, carry currently listed securities only, so the thing the product is bought for starts at Platinum. The updater is a Windows application that must be running for any of the plugins to answer.

An engine that ships its own universe. WealthLab includes Wealth-Data in the subscription: end-of-day bars for the members of a handful of US and German indices from about 1995, delisted ones included, with dynamic DataSets that offer each stock as a candidate only while it was in the index. It is the least assembly of anything here, and the narrowest — a few indices, daily bars, and a membership rule you do not write yourself.

A hosted platform with the data mounted. QuantConnect runs backtests against a security master of roughly 27,500 US equities from 1998, with delistings, mergers and ticker changes handled by the engine. On the final trading day a delisted stock raises a warning event; afterwards LEAN liquidates whatever is still held. Historical index membership is not a licensed constituent table here: the closest thing is its ETF constituents dataset, built from issuers' published holdings from June 2009. QuantRocket is the self-hosted version of the same idea — US stocks and ETFs from 2007 with delisted names, included with a paid licence — and sells S&P 500 additions and removals as part of the separately priced Sharadar fundamentals bundle.

Raw files you wire up yourself. FirstRate Data sells minute bars outright, with delisted tickers suffixed -DELISTED so a recycled symbol cannot collide with its predecessor. There is no membership data, so the universe definition is yours to source, and any engine will do — Backtrader and vectorbt both take whatever pandas data you hand them and ask no questions about where it came from, which is the problem as much as the convenience.

Where this breaks

A ticker is not an identity. Exchanges reassign symbols, so a table keyed on ticker splices an old company and a new one into a single price series and the join never complains. Every product above answers this differently, and the answer is worth knowing before you merge in anything else. Norgate gives a delisted security a suffix of the year and month it last traded — Enron sits in its database as ENRNQ-200411 — and gives every security an unchanging numeric assetid; a ticker change moves the whole history under the new symbol and the old one disappears from the database. WealthLab's delisted symbols carry the last trading date and a letter for the company's fate, as in EYE.20050527.A. QuantConnect keys everything to its own security identifier and fires a symbol-changed event, cancelling any open order on the old ticker. The safe rule is to join your own data — fundamentals, a signal file, anything — on the vendor's permanent identifier and never on the ticker; the reasons identifiers exist at all are in security identifiers.

Membership has two dates. An index change is announced on one day and takes effect on another, and a membership series stamped at the announcement lets the backtest buy a week early — the point-in-time problem in its most literal form. A vendor's 1-or-0 series should switch on the effective date; a list you assembled from press releases or a Wikipedia page history probably does not. The same goes for exits. WealthLab closes a position on the day its company leaves the index. Zipline does nothing of the kind: a pipeline screen only stops the name appearing in tomorrow's output, and Norgate's own worked example has to sell anything no longer in the index in a loop you write. Whether leaving the index is a sell signal is a decision about the test, and it should be one you made.

Adjusted prices are prices nobody traded at. Norgate's Python package defaults to total-return adjustment, and so does its Zipline bundle; QuantConnect's default mode adjusts for splits and dividends using the entire split and dividend history. That is right for measuring returns and wrong for any rule that reads a price level. "Skip stocks under $5" run on a series that has had twenty years of dividends subtracted from it excludes names that traded well above $5 at the time, and a per-share commission applied to an adjusted price charges for a share count nobody traded. Keep an unadjusted close beside the adjusted one — Norgate exposes one as a pipeline factor, QuantConnect has a Raw mode — and use it for anything that means a price rather than a return. Why two vendors' adjusted closes disagree in the first place is in why adjusted close differs.

The last print is not the last cash. Delisted history ends at the last exchange trade, and an engine that auto-closes at that price has booked a sale at a number the holder may never have seen — a merger settled in cash or stock after the last trade, a bankruptcy where the shares went on trading over the counter. Norgate is explicit that a move to OTC is not a delisting in its database: the history carries on under the OTC symbol until the stock stops trading altogether, and a Major Exchange Listed series marks which days were which. That is correct, and it means the exchange history and the OTC history meet somewhere you should look at. For a strategy that holds distressed names, the delisting exit is the single number most worth checking by hand.

"Survivorship-free" is a range, not a checkbox. FirstRate Data and Kibot both include delisted names and both say the set is incomplete — FirstRate adds that it thins the further back it goes; QuantConnect's master starts in 1998 and excludes OTC stocks; ETF holdings as a membership proxy start in June 2009 and were only monthly until January 2015, so a month of membership changes can pass unseen. None of these is hidden. Each one caps how far back the test can honestly start, and survivorship bias creeps back in wherever the data thins out.

If you outgrow this

When the fundamentals have to be point-in-time too. Norgate's fundamentals are a current snapshot with no history, so a test that ranks on earnings or book value is using this year's figures for every year. That needs fundamentals dated by when each figure became public rather than by the period it describes — ask a vendor which date its rows carry before buying — and the same identifier discipline, because the fundamentals file and the price file must agree on who a company is.

When daily bars stop being enough. The membership problem does not change at minute resolution; the data bill does. AlgoSeek leases US equity minute and tick history from 2007 with delisted names, point-in-time index membership and a security master that follows a ticker through mergers, from $1,500 a month for a single equity dataset on a twelve-month minimum. Storing it is a separate problem, covered in storing minute bars locally.

When the test is right and the result is too good. Survivorship is one leak; there are others that survive a clean universe untouched — fills at prices the book could not have given (slippage) and parameters chosen with the whole history in view (walk-forward is the usual check). A clean universe is necessary and does not make a test honest by itself.

The rest of the category is backtesting frameworks, and the data side is in market data APIs.