Plugin

US rates curve

US Treasury yields from 3 months to 30 years, the real 10 year yield and the fed funds rate, with curve spreads and breakeven inflation, from the Treasury and the New York Fed.

Data sourceNetwork: home.treasury.gov, markets.newyorkfed.orgDesktop onlyv1.0.0

Get early access Plugin API documentation

About

US rates move the dollar, gold and equity indices. The slope of the curve (10 years minus 2 years) has turned negative before most US recessions, and many traders read gold against the real yield. This plugin brings the curve into a backtest as daily feeds.

Before a run, it downloads the Treasury's daily par yield curve, real yield curve and bill rates, and the effective federal funds rate from the New York Fed, and writes one feed per series, named after the FRED series of the same data: us_dff (effective federal funds rate), us_dtb3 (13 week bill), us_dgs3mo, us_dgs2, us_dgs10, us_dgs30 (par yields), us_dfii10 (10 year real yield, TIPS), and three computed here: us_t10y2y (10 years minus 2 years), us_t10y3m (10 years minus 3 months) and us_t10yie (10 year breakeven inflation, nominal minus real). All in percent.

The data comes from the publishers themselves, both public and keyless.

Configuration

Field	Type	Default	Description
`cache_days`Refetch after (days)	number	`1`	Days before the plugin downloads the current year and the fed funds rate again. Within that time a run reads the cached feeds and needs no network.

Use it in a strategy

pythonbars = ctx.bars(1)
if bars:
    slope = ctx.data("us_t10y2y", bars[-1].ts)  # 10 year minus 2 year yield, in points
    if slope is not None and slope < 0:
        ctx.signal("EXIT")  # stand aside while the curve is inverted

The feeds are refreshed before a run whose strategy reads ctx.data, or when "Fetch external data (news and plugin feeds)" is ticked. A run on the server never fetches.

What it provides

`us_dgs3mo, us_dgs2, us_dgs10, us_dgs30`Feed	Treasury par yields for 3 months, 2, 10 and 30 years in percent, from 1990 (the 30 year has no values from February 2002 to February 2006, when it was not issued).
`us_dtb3`Feed	The 13 week Treasury bill rate on a bank discount basis in percent, from 2002, as the Treasury publishes it.
`us_dfii10`Feed	The 10 year real yield (Treasury real yield curve, from TIPS) in percent, from 2003.
`us_dff`Feed	The effective federal funds rate of the New York Fed in percent, from July 2000.
`us_t10y2y, us_t10y3m, us_t10yie`Feed	Computed on days both legs were published: 10 year minus 2 year yield, 10 year minus 3 month yield, and 10 year breakeven inflation (nominal minus real), in percentage points.

Data source and terms

Publisher	U.S. Department of the Treasury, daily Treasury par yield curve, real yield curve and bill rates
Terms	Read the terms
API key	None needed
Attribution	Source: U.S. Department of the Treasury
Rate limit	One small file per year and series type, about 90 for the whole history, paced a second apart; once the history is cached a refresh asks for the current year only (3 files).

Publisher	Federal Reserve Bank of New York, Effective Federal Funds Rate (EFFR)
Terms	Read the terms
API key	None needed
Attribution	The EFFR is subject to the Terms of Use posted at newyorkfed.org.
Rate limit	One download of about 1.5 MB per refresh; automated access is allowed as long as it does not disrupt the site.

Hosts it may reach: home.treasury.gov, markets.newyorkfed.org

When values are known

The Treasury publishes a day's yields that evening and the New York Fed a day's EFFR at about 9:00 the next business morning. Each value is stored two US business days after the day it describes (federal holidays, Good Friday and one-off closures skipped) and carried forward over weekends and holidays, for at most 10 days. A bar on Monday reads Thursday's yields; Friday's arrive on Tuesday.

Limits

Daily closing values only: an intraday strategy sees rates that are two business days old, never the move of the day.
The fed funds rate starts in July 2000 (the New York Fed's own history); the bill rate starts in 2002, the real yield in 2003, the par yields in 1990.
us_dtb3 is the Treasury's own 13 week bill rate; FRED's DTB3 comes from the Fed's H.15 and can differ by a few basis points on some days.
The first fetch needs about 90 small files, and the Treasury sometimes takes 20 seconds per file. A refresh fetches the newest years first for at most 90 seconds and writes what it has; the older years follow on the next runs, so the early history can be missing at first. Finished years stay in the data cache, and the data stays on your machine.

Preview

json{
  "2024-05-07": 0.31,
  "2024-05-08": 0.28,
  "2024-05-09": 0.28,
  "2024-05-10": -0.04,
  "_meta": {
    "delay_rule": "business_days:2 (US holidays)",
    "source": "Made-up sample, not vendor data",
    "terms": "https://home.treasury.gov/subfooter/privacy-policy",
    "units": "percent"
  }
}

A made-up sample that shows the shape of the output. Not vendor data and not a result.

The code

main.py, the file the app installs.

python"""US rates curve: a data source on the plugin kit.

Before a run it downloads US Treasury yields from the Treasury's daily interest
rate files and the effective federal funds rate from the New York Fed, and
writes one feed per series, named after the FRED series of the same data:

    us_dff      effective federal funds rate (New York Fed, from July 2000)
    us_dtb3     13 week Treasury bill, bank discount rate
    us_dgs3mo   3 month Treasury par yield
    us_dgs2     2 year Treasury par yield
    us_dgs10    10 year Treasury par yield
    us_dgs30    30 year Treasury par yield (none from 2002 to 2006)
    us_dfii10   10 year real yield (TIPS, from 2003)
    us_t10y2y   10 year minus 2 year, computed here
    us_t10y3m   10 year minus 3 month, computed here
    us_t10yie   10 year breakeven inflation: nominal minus real, computed here

All in percent. A strategy reads them by day:

    def on_market(self, ctx):
        bars = ctx.bars(1)
        if not bars:
            return
        slope = ctx.data("us_t10y2y", bars[-1].ts)
        if slope is not None and slope < 0:
            ctx.signal("EXIT")

When a value is known: the Treasury publishes a day's yields that evening and
the New York Fed a day's EFFR the next business morning, so each value is
stored two US business days after the day it describes and carried over
weekends and holidays, at most 10 days.

The Treasury serves one file per year and series type, about 90 for the whole
history, and some days answers each one slowly (20 seconds was seen). So a
refresh fetches the newest years first and stops after FETCH_BUDGET_SECONDS;
the feeds are written with what is there, and the older years follow on the
next runs. Each finished year is kept in the data cache, so once the history
is complete a refresh asks only for the current year. A failed request stops
the refresh and leaves the feeds as they were.
"""

import json
import time
from datetime import date, timedelta

from orionquant_engine.plugins.kit import dates, feeds, net

HOSTS = ("home.treasury.gov", "markets.newyorkfed.org")
TREASURY = "https://home.treasury.gov/resource-center/data-chart-center/interest-rates/daily-treasury-rates.csv/{year}/all"
NYFED_EFFR = "https://markets.newyorkfed.org/api/rates/unsecured/effr/last/0.json"

#: Treasury file type: (first year it has, {column: series}).
CURVES = {
    "daily_treasury_yield_curve": (1990, {"3 Mo": "DGS3MO", "2 Yr": "DGS2", "10 Yr": "DGS10", "30 Yr": "DGS30"}),
    "daily_treasury_bill_rates": (2002, {"13 WEEKS BANK DISCOUNT": "DTB3"}),
    "daily_treasury_real_yield_curve": (2003, {"10 YR": "DFII10"}),
}
#: Computed series: (minuend, subtrahend), on days both were published.
SPREADS = {"T10Y2Y": ("DGS10", "DGS2"), "T10Y3M": ("DGS10", "DGS3MO"), "T10YIE": ("DGS10", "DFII10")}

DELAY_RULE = "business_days:2"
MAX_GAP_DAYS = 10
#: Finished years of the Treasury files, kept in the data cache (a relative
#: path lands there) and outside the feed folder, so ctx.data never reads them.
CACHE_DIR = "plugin_cache/us_rates_curve"
#: A year counts as finished once fetched a week into the next one, after any
#: late correction of its last days.
SETTLE_DAYS = 7
#: How long one refresh may spend downloading Treasury year files before it
#: writes what it has and leaves the rest for the next run.
FETCH_BUDGET_SECONDS = 90
#: Days the US bond market or the Fed closed outside the holiday rules.
ONE_OFF_CLOSURES = (
    "2001-09-11", "2001-09-12", "2004-06-11", "2007-01-02", "2012-10-30",
    "2018-12-05", "2025-01-09",
)

_API = {"api": None}


def _log(message):
    if _API["api"] is not None:
        _API["api"].log(message)


def _config(key, default):
    api = _API["api"]
    return api.get_config(key, default) if api is not None else default


def feed_name(series):
    return "us_" + series.lower()


def _sources():
    """The two sources as docs.json writes them down (name, terms, attribution)."""
    try:
        with open(__plugin_dir__ + "/docs.json", encoding="utf-8") as fh:  # noqa: F821  # set by the loader
            return json.load(fh)["sources"]
    except (OSError, ValueError, KeyError, NameError):
        return []


def us_holidays(years):
    """US federal holidays, kept generously: the day itself and the day it is
    observed, Juneteenth from 2021, Good Friday (the bond market closes), and
    the one-off closures. An extra holiday only makes a value known a day
    later, never earlier."""
    out = set()
    for y in years:
        fixed = [date(y, 1, 1), date(y, 7, 4), date(y, 11, 11), date(y, 12, 25)]
        if y >= 2021:
            fixed.append(date(y, 6, 19))
        for d in fixed:
            out.update((d, dates.observed(d)))
        out.update((
            dates.nth_weekday(y, 1, dates.MON, 3), dates.nth_weekday(y, 2, dates.MON, 3),
            dates.nth_weekday(y, 5, dates.MON, -1), dates.nth_weekday(y, 9, dates.MON, 1),
            dates.nth_weekday(y, 10, dates.MON, 2), dates.nth_weekday(y, 11, dates.THU, 4),
            dates.easter(y) - timedelta(days=2),
        ))
    out.update(date.fromisoformat(d) for d in ONE_OFF_CLOSURES)
    return out


def parse_treasury(rows, columns):
    """One Treasury year file to {day: {series: value}}. Dates are MM/DD/YYYY;
    an empty cell is a day without that maturity."""
    out = {}
    for row in rows:
        stamp = (row.get("Date") or "").strip()
        try:
            month, day, year = (int(x) for x in stamp.split("/"))
            key = date(year, month, day).isoformat()
        except ValueError:
            continue
        for column, series in columns.items():
            raw = (row.get(column) or "").strip()
            try:
                out.setdefault(key, {})[series] = float(raw)
            except ValueError:
                continue
    return {k: v for k, v in out.items() if v}


def parse_effr(payload):
    """The New York Fed's JSON to {day: rate}."""
    out = {}
    for item in (payload or {}).get("refRates", []):
        try:
            out[feeds.day_key(item["effectiveDate"])] = float(item["percentRate"])
        except (KeyError, TypeError, ValueError):
            continue
    return out


def _cache_path(kind, year):
    return "{}/{}_{}.json".format(CACHE_DIR, kind, year)


def _read_cache(kind, year):
    try:
        with open(_cache_path(kind, year), encoding="utf-8") as fh:
            data = json.load(fh)
        return data if isinstance(data, dict) and isinstance(data.get("rows"), dict) else None
    except (OSError, ValueError):
        return None


def cached_rows(kind, year, today, max_age_days):
    """{day: {series: value}} of one Treasury year file from the cache, when
    that year was fetched after it had settled or fetched recently enough.
    None when it has to be downloaded."""
    cached = _read_cache(kind, year)
    if cached is None:
        return None
    fetched = cached.get("fetched", "")
    settled = date(year + 1, 1, 1) + timedelta(days=SETTLE_DAYS)
    recent = (today - timedelta(days=max_age_days)).isoformat()
    return cached["rows"] if fetched >= settled.isoformat() or fetched > recent else None


def download_year(kind, year, today):
    """Download one Treasury year file and keep it in the cache. Raises NetError."""
    rows = net.get_csv(TREASURY.format(year=year), hosts=HOSTS, timeout=60.0, delay=1.0,
                       params={"type": kind, "field_tdr_date_value": year, "_format": "csv"})
    parsed = parse_treasury(rows, CURVES[kind][1])
    with open(_cache_path(kind, year), "w", encoding="utf-8") as fh:
        json.dump({"fetched": today.isoformat(), "rows": parsed}, fh)
    return parsed


def collect(today, max_age_days):
    """Every series observed, by day, and whether every year was there.
    Newest years first, within the time budget. None when a download failed."""
    try:
        effr = parse_effr(net.get_json(NYFED_EFFR, hosts=HOSTS, timeout=60.0))
    except net.NetError as exc:
        _log("fetch failed, the cached feeds stay: {}".format(exc))
        return None, False
    observed = {"DFF": effr}
    started = time.monotonic()
    complete = True
    for year in range(today.year, min(first for first, _ in CURVES.values()) - 1, -1):
        for kind, (first, _columns) in CURVES.items():
            if year < first:
                continue
            rows = cached_rows(kind, year, today, max_age_days)
            if rows is None:
                if time.monotonic() - started > FETCH_BUDGET_SECONDS:
                    complete = False
                    continue
                try:
                    rows = download_year(kind, year, today)
                except net.NetError as exc:
                    _log("fetch failed, the cached feeds stay: {}".format(exc))
                    return None, False
            for day, values in rows.items():
                for series, value in values.items():
                    observed.setdefault(series, {})[day] = value
    for name, (a, b) in SPREADS.items():
        left, right = observed.get(a, {}), observed.get(b, {})
        observed[name] = {d: round(left[d] - right[d], 2) for d in left if d in right}
    return observed, complete


def data_fetch(context):
    """Lifecycle hook: refresh the feeds before the backtest runs."""
    cache_days = float(_config("cache_days", 1))
    meta = feeds.read_meta(feed_name("DGS10")) or {}
    if meta.get("complete") and feeds.fresh(feed_name("DGS10"), cache_days * 86400):
        return
    today = feeds.as_date(feeds.today())
    observed, complete = collect(today, cache_days)
    if observed is None:
        return
    if not observed.get("DGS10") or not observed.get("DFF"):
        _log("the Treasury or the New York Fed returned no values, the cached feeds stay")
        return
    sources = _sources()
    treasury = sources[0] if sources else {}
    nyfed = sources[1] if len(sources) > 1 else {}
    holidays = us_holidays(range(1989, today.year + 2))
    for series in sorted(observed):
        rows = observed[series]
        if not rows:
            continue
        src = nyfed if series == "DFF" else treasury
        known = feeds.shift(rows, DELAY_RULE, holidays=holidays)
        feeds.write_feed(feed_name(series), feeds.fill_forward(known, until=today, max_gap_days=MAX_GAP_DAYS),
                         meta={
                             "source": src.get("name", "US Treasury and Federal Reserve Bank of New York"),
                             "terms": src.get("terms", "see the plugin's docs"),
                             "attribution": src.get("attribution"),
                             "delay_rule": DELAY_RULE + " (US holidays)",
                             "series": series,
                             "units": "percent",
                             "complete": complete,
                         })
    if complete:
        _log("cached {} US rate series".format(len(observed)))
    else:
        _log("cached the newest years; older Treasury years follow on the next runs")


def register(api):
    _API["api"] = api
    api.register_data_fetch(data_fetch)