Plugin

Notebook JSON export

Writes a whole run as one versioned, documented JSON file plus a trades CSV and a JSON Schema, ready for pandas or any notebook.

ExporterNo networkDesktop onlyv1.0.0

Get early access Plugin API documentation

About

For your own analysis in Jupyter, pandas or any other tool: the run as one JSON file with what ran (instrument, timeframe, period, seed, currency, engine), its configuration, the statistics, the equity curve, every trade with every field the engine recorded, and the monthly returns. The trades also come as a CSV, and the file's JSON Schema comes along, so the export explains itself wherever it is copied.

The file says "format": "orionquant-notebook" and "formatVersion": 1. Version 1 only ever gains fields; a change of meaning would be version 2, so code written against version 1 keeps working. Keys are camelCase as in the engine's result, times are ISO 8601 in UTC, money is in run.resultCurrency.

In pandas:

pythonimport json
import pandas as pd

with open("notebook_EURUSD_42.json", encoding="utf-8") as fh:
    nb = json.load(fh)
trades = pd.DataFrame(nb["trades"])
equity = pd.DataFrame(nb["equity"]).assign(ts=lambda d: pd.to_datetime(d["ts"])).set_index("ts")
trades_csv = pd.read_csv("notebook_EURUSD_42_trades.csv", parse_dates=["entryTs", "exitTs"])

Configuration

Field	Type	Default	Description
`equity`Equity points	select	`"all"`	all: one equity point per bar. daily: the last point of each UTC day, which keeps the file small for a long run on a short timeframe.Options: all, daily
`fileName`File name	string	`"notebook"`	The start of the file names; the instrument and the run's seed are added.

Use it in a strategy

Not called from a strategy: after a run, choose it in the result's export menu. Exports are part of the plans that include CSV export. The pandas lines above read the files it writes.

What it provides

`notebook_<instrument>_<seed>.json`File	The run: run, config, stats, costsUsed, equitySampling, equity, trades and monthlyReturns, as the schema describes.
`notebook_<instrument>_<seed>_trades.csv`File	One row per trade, columns named as in the JSON (id, symbol, direction, entryTs, exitTs, prices, lots, P&L, costs, exitReason, mae, mfe and more).
`notebook_json_export.schema.json`File	The JSON Schema (draft 2020-12) of the JSON file, with a description of every field that needs one.

Limits

The strategy's source code is left out of the configuration (strategy.code and strategy.files), so a shared file does not share your strategy.
The price bars and indicator series are not included; a long run on a short timeframe still gives a large file, which equity daily makes smaller.
A number that is not finite (an infinite profit factor, say) is written as null, because JSON has no Infinity.

Preview

json{
  "$schema": "notebook_json_export.schema.json",
  "config": {
    "accountCurrency": "EUR",
    "endDate": "2024-03-13",
    "instrument": "EURUSD",
    "seed": 42,
    "startDate": "2024-03-12",
    "startingBalance": 10000,
    "strategy": {
      "params": {
        "lookback": 20
      },
      "providerId": "python",
      "strategyId": "my_breakout"
    },
    "timeframe": "H1"
  },
  "costsUsed": {
    "commissionPerLot": 7,
    "source": "flat"
  },
  "equity": [
    {
      "balance": 10000,
      "drawdownPct": 0,
      "equity": 10000,
      "ts": "2024-03-12T08:00:00Z"
    },
    {
      "balance": 10176.5,
      "drawdownPct": 0,
      "equity": 10176.5,
      "ts": "2024-03-12T14:00:00Z"
    }
  ],
  "equitySampling": "all",
  "format": "orionquant-notebook",
  "formatVersion": 1,
  "monthlyReturns": [
    {
      "month": 3,
      "returnPct": 1.765,
      "year": 2024
    }
  ],
  "run": {
    "accountCurrency": "EUR",
    "accuracyTierUsed": "M1_PROXY",
    "barsProcessed": 2,
    "dataSourceUsed": "csv",
    "endDate": "2024-03-13",
    "engineUsed": "python",
    "engineVersion": "0.1.6",
    "instrument": "EURUSD",
    "protocolVersion": "2.18.0",
    "resultCurrency": "USD",
    "ruined": false,
    "ruinedAt": null,
    "seed": 42,
    "spreadModel": "history",
    "startDate": "2024-03-12",
    "startingBalance": 10000,
    "timeframe": "H1"
  },
  "stats": {
    "finalEquity": 10176.5,
    "maxDrawdown": 42.5,
    "maxDrawdownPct": 0.42,
    "netPnl": 176.5,
    "numTrades": 1,
    "profitFactor": null,
    "returnPct": 1.765,
    "sharpe": 0.12,
    "winRate": 100
  },
  "trades": [
    {
      "barsHeld": 6,
      "commission": 3.5,
      "direction": "LONG",
      "entryPrice": 1.0925,
      "entryTs": "2024-03-12T08:15:00Z",
      "exitPrice": 1.0961,
      "exitReason": "SIGNAL",
      "exitTs": "2024-03-12T13:45:00Z",
      "grossPnl": 180,
      "id": 1,
      "mae": -42.5,
      "mfe": 195,
      "netPnl": 176.5,
      "quantityLots": 0.5,
      "stopPrice": 1.0905,
      "swap": 0,
      "symbol": "EURUSD",
      "targetPrice": null
    }
  ]
}

A made-up sample that shows the shape of the output. Not vendor data and not a result.

The code

main.py, the file the app installs.

python"""Notebook JSON export: a whole run as one documented JSON file, for pandas.

Three files go out together:

    <name>.json                         the run: what ran, its configuration,
                                        statistics, equity curve, every trade
                                        and the monthly returns
    <name>_trades.csv                   the trades again, one row each, for a
                                        quick pd.read_csv
    notebook_json_export.schema.json    the JSON Schema of the first file

The JSON carries `"format": "orionquant-notebook"` and `"formatVersion": 1`.
Version 1 only ever gains fields; a field that changed meaning would make it
version 2, so a notebook written against version 1 keeps reading every
version 1 file. Keys are camelCase as the engine's result has them, times are
ISO 8601 in UTC, and money is in `run.resultCurrency`.

Two things are left out on purpose: the strategy's source code (a file meant
for a notebook gets passed around, and the code is yours), and any number that
is not finite (JSON has no Infinity, and pandas would choke on it): such a
number is written as null.
"""

import csv
import io
import json
import math
import re

_API_REF = {"api": None}

FORMAT = "orionquant-notebook"
FORMAT_VERSION = 1
SCHEMA_FILE = "notebook_json_export.schema.json"

#: The run facts copied from the result's top level into "run".
_RUN_FIELDS = ("protocolVersion", "engineVersion", "engineUsed", "dataSourceUsed", "accuracyTierUsed",
               "spreadModel", "barsProcessed", "ruined", "ruinedAt")

#: The trade columns of the CSV, in order: the scalar fields of a trade, named
#: as in the JSON. A field a result does not carry is an empty cell.
TRADE_COLUMNS = [
    "id", "symbol", "direction", "entryTs", "exitTs", "entryPrice", "exitPrice", "quantityLots",
    "grossPnl", "commission", "swap", "netPnl", "exitReason", "barsHeld", "stopPrice", "targetPrice",
    "mae", "mfe", "entryOrder", "requestedLots", "entryTouchSize", "exceededTouch", "bookShort",
]


def _setting(key, default):
    api = _API_REF["api"]
    return api.get_config(key, default) if api else default


def _finite(value):
    """`value` with every NaN or infinity replaced by None, at any depth."""
    if isinstance(value, float) and not math.isfinite(value):
        return None
    if isinstance(value, dict):
        return {k: _finite(v) for k, v in value.items()}
    if isinstance(value, list):
        return [_finite(v) for v in value]
    return value


def _daily(equity):
    """The last equity point of each UTC day, in order."""
    last = {}
    for point in equity:
        last[str(point.get("ts", ""))[:10]] = point
    return [last[day] for day in sorted(last)]


def notebook(result, *, sampling="all"):
    """The JSON document of one run, as a dict."""
    config = dict(result.get("config") or {})
    strategy = dict(config.get("strategy") or {})
    strategy.pop("code", None)
    strategy.pop("files", None)
    config["strategy"] = strategy

    run = {
        "instrument": config.get("instrument", ""),
        "timeframe": config.get("timeframe", ""),
        "startDate": config.get("startDate", ""),
        "endDate": config.get("endDate", ""),
        "seed": result.get("seed", config.get("seed", 0)),
        "resultCurrency": result.get("resultCurrency", ""),
        "accountCurrency": config.get("accountCurrency"),
        "startingBalance": config.get("startingBalance"),
    }
    for key in _RUN_FIELDS:
        run[key] = result.get(key)

    equity = [{k: p.get(k) for k in ("ts", "equity", "balance", "drawdownPct")} for p in result.get("equity") or []]
    if sampling == "daily":
        equity = _daily(equity)

    doc = {
        "$schema": SCHEMA_FILE,
        "format": FORMAT,
        "formatVersion": FORMAT_VERSION,
        "run": run,
        "config": config,
        "stats": result.get("stats") or {},
        "costsUsed": result.get("costsUsed"),
        "equitySampling": sampling,
        "equity": equity,
        "trades": list(result.get("trades") or []),
        "monthlyReturns": list((result.get("analysis") or {}).get("monthlyReturns") or []),
    }
    return _finite(doc)


def trades_csv(trades):
    """The trades as CSV text: one row each, the columns of TRADE_COLUMNS."""
    buf = io.StringIO()
    writer = csv.writer(buf, lineterminator="\n")
    writer.writerow(TRADE_COLUMNS)
    for trade in trades:
        writer.writerow(["" if trade.get(c) is None else trade.get(c) for c in TRADE_COLUMNS])
    return buf.getvalue()


def _file_part(text):
    """A piece of a file name: letters, digits, dash and underscore only."""
    return re.sub(r"[^A-Za-z0-9_-]+", "_", str(text)).strip("_") or "run"


def export(result):
    sampling = "daily" if _setting("equity", "all") == "daily" else "all"
    doc = notebook(result, sampling=sampling)
    config = result.get("config") or {}
    base = "{}_{}_{}".format(_file_part(_setting("fileName", "notebook")),
                             _file_part(config.get("instrument", "")), _file_part(result.get("seed", 0)))

    # The schema ships in this plugin's folder; it goes out next to the data
    # so the export explains itself wherever it is copied.
    with open(__plugin_dir__ + "/" + SCHEMA_FILE, encoding="utf-8") as fh:  # noqa: F821  set by the loader
        schema_text = fh.read()

    # Relative paths land in the export's own folder (or the data cache).
    # Three files, so the function returns nothing and the export hands back the folder.
    with open(base + ".json", "w", encoding="utf-8", newline="\n") as fh:
        json.dump(doc, fh, indent=1, ensure_ascii=False, allow_nan=False)
        fh.write("\n")
    with open(base + "_trades.csv", "w", encoding="utf-8", newline="") as fh:
        fh.write(trades_csv(doc["trades"]))
    with open(SCHEMA_FILE, "w", encoding="utf-8", newline="\n") as fh:
        fh.write(schema_text)
    api = _API_REF["api"]
    if api:
        api.log("wrote {0}.json, {0}_trades.csv and {1} ({2} trades, {3} equity points)".format(
            base, SCHEMA_FILE, len(doc["trades"]), len(doc["equity"])))
    return None


def register(api):
    _API_REF["api"] = api
    api.register_exporter("notebook_json", "Notebook JSON", export)