Plugin

Binance spot bars

Downloads closed Binance spot bars from M1 to D1 for crypto pairs like BTCUSDT, and writes daily close feeds for pairs a strategy wants to watch.

Data sourceNetwork: data-api.binance.visionDesktop onlyv1.0.0

Get early access Plugin API documentation

About

A download source for crypto pairs. In the Data view, choose Binance spot as the source, type a Binance spot symbol (BTCUSDT, ETHUSDT) and a timeframe from M1 to D1. The plugin pages through Binance's public market data host 1,000 bars at a time, half a second apart, and the engine stores the bars like any other download.

Only closed bars are kept: the bar still forming at the moment of the download is left out, so a backtest never sees a candle that was not finished. Times are UTC, each bar under its open time.

The plugin also writes the daily close of the pairs in Daily close feeds as binance_<pair>_close (BTCUSDT by default), for strategies that look at another market, for example a filter on Bitcoin while trading something else.

Configuration

Field	Type	Default	Description
`cache_days`Refetch close feeds after (days)	number	`1`	Days before the close feeds are downloaded again. Within that time a run reads the cached feeds and needs no network.
`close_feeds`Daily close feeds (pairs, comma separated)	string	`"BTCUSDT"`	Pairs whose daily close is written as a feed before a run, separated by commas, for example BTCUSDT, ETHUSDT. Empty writes none. The bar download does not use this setting.

Use it in a strategy

pythonbars = ctx.bars(1)
if bars:
    btc = ctx.data("binance_btcusdt_close", bars[-1].ts)
    if btc is not None and btc < 20000:
        ctx.signal("EXIT")

For the bars themselves no code is needed: download them in the Data view with Binance spot as the source and run a backtest on the pair. The close feeds are refreshed before a run whose strategy reads ctx.data, or when "Fetch external data (news and plugin feeds)" is ticked. A run on the server never fetches.

What it provides

`binance_spot`Bars	The download source: closed OHLCV bars in UTC for M1, M5, M15, M30, H1, H4 and D1. Volume is in the base asset (BTC for BTCUSDT).
`binance_<pair>_close`Feed	The daily close of each pair in Daily close feeds, a number, under the day after the UTC day it closes.

Data source and terms

Publisher	Binance spot market data (data-api.binance.vision, /api/v3/klines)
Terms	Read the terms
API key	None needed
Attribution	Market data: Binance spot
Rate limit	Up to 1,000 bars per request, half a second apart: a year of M1 bars is about 530 requests and takes some minutes. Binance answers 429 or 418 when a machine asks too much.

Hosts it may reach: data-api.binance.vision

When values are known

A bar is complete at its close, and only closed bars are stored, each under its open time as every bar in the engine. A daily close feed value is stored under the next UTC day, when the day's bar has closed.

Limits

Binance refuses some countries with HTTP 451 (unavailable for legal reasons). The download then stops with a message saying so; the plugin cannot work around it.
Only pairs trading today can be chosen; pairs Binance delisted are missing, so a test over "all coins" leans towards the survivors.
One exchange's prices and volume, not a market wide index. Spot only, no futures and no funding.
A pair the engine does not know, such as BTCUSDT, needs its asset class set to crypto or a custom symbol before a run. The data stays on your machine and is for your own use under Binance's terms.

Preview

json[
  {
    "close": 100,
    "high": 101.5,
    "low": 97.75,
    "open": 99,
    "ts": "2024-03-01T00:00:00Z",
    "volume": 12.5
  },
  {
    "close": 102.5,
    "high": 104,
    "low": 98.75,
    "open": 100,
    "ts": "2024-03-02T00:00:00Z",
    "volume": 12.5
  }
]

A made-up sample that shows the shape of the output. Not vendor data and not a result.

The code

main.py, the file the app installs.

python"""Binance spot bars: a download source for crypto pairs, plus optional daily close feeds.

Registered as the data source "binance_spot". In the Data view, pick it as the
source, type a Binance spot symbol (BTCUSDT, ETHUSDT) and a timeframe from M1
to D1, and the engine stores the bars like any other download. A backtest then
runs on them.

Only closed bars are returned: the bar still forming when you download is
left out, so a backtest never trades on a candle that was not finished. Bars
are fetched 1,000 at a time from Binance's public market data host
(data-api.binance.vision, which serves market data only and needs no key),
half a second apart.

Some countries are refused by Binance with HTTP 451 ("unavailable for legal
reasons"). The plugin then stops with a message saying so; it cannot work
around it.

With `close_feeds` set (for example "BTCUSDT, ETHUSDT"), it also writes the
daily close of each pair as a feed before a run, for strategies that look at
another market: `binance_btcusdt_close`. A day's close is known when the day
ends, so it is stored under the next UTC day.

    def on_market(self, ctx):
        bars = ctx.bars(1)
        if not bars:
            return
        btc = ctx.data("binance_btcusdt_close", bars[-1].ts)
        if btc is not None and btc < 20000:
            ctx.signal("EXIT")
"""

import re
from datetime import datetime, timezone

from orionquant_engine.plugins.kit import feeds, net

HOSTS = ("data-api.binance.vision",)
KLINES = "https://data-api.binance.vision/api/v3/klines"
SOURCE_ID = "binance_spot"

#: Binance answers at most 1,000 klines per request.
PAGE = 1000

#: The engine's timeframes to Binance kline intervals. S5 has no kline.
INTERVALS = {"M1": "1m", "M5": "5m", "M15": "15m", "M30": "30m", "H1": "1h", "H4": "4h", "D1": "1d"}

#: Binance spot opened in July 2017; the close feeds start there.
FIRST_DAY = "2017-07-01"
DELAY_RULE = "next_day"
MAX_GAP_DAYS = 7

# The terms link is listed in docs.json, so the template check knows it is never fetched.
TERMS = "https://www.binance.com/en/terms"
ATTRIBUTION = "Market data: Binance spot"

_API = {"api": None}


def _log(message):
    if _API["api"] is not None:
        _API["api"].log(message)


def _config(key, default):
    api = _API["api"]
    return api.get_config(key, default) if api is not None else default


def binance_symbol(symbol):
    """BTCUSDT from btcusdt, BTC/USDT or BTC-USDT. ValueError for anything that
    cannot be a Binance spot symbol (letters and digits, 5 to 20 of them)."""
    flat = re.sub(r"[/\-_ .]", "", str(symbol or "")).upper()
    if not re.fullmatch(r"[A-Z0-9]{5,20}", flat):
        raise ValueError("{!r} is not a Binance spot symbol like BTCUSDT".format(symbol))
    return flat


def _ms(day):
    d = datetime.fromisoformat(str(day)[:10]).replace(tzinfo=timezone.utc)
    return int(d.timestamp() * 1000)


def _now_ms():
    return int(datetime.now(timezone.utc).timestamp() * 1000)


def _explain(exc, symbol):
    """A NetError as a message for the person downloading."""
    if exc.status == 451:
        return ("Binance refuses requests from your location (HTTP 451, unavailable for legal reasons). "
                "This source cannot be used from here.")
    if exc.status == 400:
        return "Binance answered HTTP 400: {} is probably not a Binance spot symbol (BTCUSDT, ETHUSDT).".format(symbol)
    if exc.status in (418, 429):
        return "Binance is rate limiting this machine (HTTP {}); try again in a few minutes.".format(exc.status)
    return "Binance could not be reached: {}".format(exc)


def fetch_klines(symbol, interval, start_ms, end_ms=None):
    """Every kline opening from start_ms to end_ms (both in ms, inclusive; no
    end means up to the latest), page by page, oldest first. Each page starts
    one millisecond after the last open time of the one before. Raises
    NetError as the kit does."""
    rows = []
    cursor = start_ms
    pages = 0
    while end_ms is None or cursor <= end_ms:
        page = net.get_json(KLINES, hosts=HOSTS, params={
            "symbol": symbol, "interval": interval,
            "startTime": cursor, "endTime": end_ms, "limit": PAGE,
        })
        if not isinstance(page, list):
            raise net.NetError("network", "Binance answered with something that is not a list of klines")
        pages += 1
        rows += page
        if len(page) < PAGE:
            break
        cursor = int(page[-1][0]) + 1
        if pages % 50 == 0:
            _log("{} {}: {} bars so far".format(symbol, interval, len(rows)))
    return rows


def to_bars(rows, now_ms):
    """Klines to engine bar dicts, closed bars only, one per open time.

    A kline is [open time, open, high, low, close, volume, close time, ...]
    with prices as text. The close time is the last millisecond of the bar, so
    a bar is closed when its close time is before now.
    """
    bars = {}
    for row in rows:
        try:
            opened, close_time = int(row[0]), int(row[6])
            o, h, low, c, v = (float(x) for x in row[1:6])
        except (TypeError, ValueError, IndexError):
            continue
        if close_time >= now_ms:
            continue
        ts = datetime.fromtimestamp(opened / 1000.0, tz=timezone.utc)
        bars[opened] = {
            "ts": ts.isoformat().replace("+00:00", "Z"),
            "open": o, "high": h, "low": low, "close": c, "volume": v,
        }
    return [bars[k] for k in sorted(bars)]


def download(symbol, timeframe, start, end):
    """The data source contract: bars for symbol from start to end (UTC days,
    both included), as dicts with ts, open, high, low, close, volume."""
    pair = binance_symbol(symbol)
    tf = str(timeframe or "").strip().upper()
    if tf not in INTERVALS:
        raise ValueError("Binance serves {}; not {}".format(", ".join(INTERVALS), timeframe))
    interval = INTERVALS[tf]
    start_ms = _ms(start)
    end_ms = _ms(end) + 86_400_000 - 1
    if end_ms < start_ms:
        raise ValueError("the end date {} is before the start date {}".format(end, start))
    try:
        rows = fetch_klines(pair, interval, start_ms, end_ms)
    except net.NetError as exc:
        raise ValueError(_explain(exc, pair)) from None
    bars = to_bars(rows, _now_ms())
    _log("downloaded {} closed {} bars for {}".format(len(bars), tf, pair))
    return bars


# --- the optional daily close feeds ---------------------------------------------------


def close_feed_name(pair):
    return "binance_{}_close".format(pair.lower())


def wanted_pairs(setting):
    """The pairs in the close_feeds setting; a bad one is logged and skipped."""
    out = []
    for token in re.split(r"[,\s]+", str(setting or "").strip()):
        if not token:
            continue
        try:
            out.append(binance_symbol(token))
        except ValueError as exc:
            _log(str(exc))
    return list(dict.fromkeys(out))


def daily_closes(rows, now_ms):
    """Daily klines to {UTC day the bar describes: close}, closed days only."""
    return {bar["ts"][:10]: bar["close"] for bar in to_bars(rows, now_ms)}


def data_fetch(context):
    """Lifecycle hook: refresh the daily close feeds named in close_feeds."""
    pairs = wanted_pairs(_config("close_feeds", ""))
    if not pairs:
        return
    max_age = float(_config("cache_days", 1)) * 86400
    for pair in pairs:
        name = close_feed_name(pair)
        if feeds.fresh(name, max_age):
            continue
        try:
            rows = fetch_klines(pair, "1d", _ms(FIRST_DAY))
        except net.NetError as exc:
            _log("fetch failed for {}, the cached feed stays: {}".format(pair, _explain(exc, pair)))
            continue
        closes = daily_closes(rows, _now_ms())
        if not closes:
            _log("Binance returned no daily bars for {}, the cached feed stays".format(pair))
            continue
        known = feeds.shift(closes, DELAY_RULE)
        feeds.write_feed(
            name,
            feeds.fill_forward(known, until=feeds.today(), max_gap_days=MAX_GAP_DAYS),
            meta={
                "source": "Binance spot, daily close of {} (UTC days)".format(pair),
                "terms": TERMS,
                "attribution": ATTRIBUTION,
                "delay_rule": DELAY_RULE,
                "symbol": pair,
            },
        )
        _log("cached {} daily closes for {}".format(len(closes), pair))


def register(api):
    _API["api"] = api
    api.register_data_source(SOURCE_ID, "Binance spot", download)
    api.register_data_fetch(data_fetch)