Skip to content

Latest commit

 

History

History
174 lines (123 loc) · 8.7 KB

README.md

File metadata and controls

174 lines (123 loc) · 8.7 KB

dbn-go - Golang bindings to DBN

Latest Tag Go Reference Code Of Conduct

Golang tooling for Databento's APIs and DBN format

This repository contains Golang bindings to Databento's file format Databento Binary Encoding (DBN), Historical API, and Live API. It also includes tools to interact with these services.

NOTE: This library is not affiliated with Databento. Please be careful with commands which incur billing. We are not responsible for any charges you incur.

Library Usage

To use this library, import the following package, optionally with Databento Historical or Live support:

import (
    dbn "github.com/NimbleMarkets/dbn-go"
    dbn_hist "github.com/NimbleMarkets/dbn-go/hist"
    dbn_live "github.com/NimbleMarkets/dbn-go/live"
)

Most dbn-go types and enums parallel Databento's libraries. Available messages types are:

Reading DBN Files

If you want to read a homogeneous array of DBN records from a file, use the dbn.ReadDBNToSlice generic function. We include an io.Reader wrapper, [dbn.MakeCompressedReader]https://pkg.go.dev/github.com/NimbleMarkets/dbn-go#MakeCompressedReader), that automatically handles zstd-named files. The generic argument dicates which message type to read.

file, closer, _ := dbn.MakeCompressedReader("ohlcv-1s.dbn.zstd", false)
defer closer.Close()
records, metadata, err := dbn.ReadDBNToSlice[dbn.OhlcvMsg](file)

Alternatively, you can use the DBNScanner to read records one-by-one. Each record can be handled directly, or automatically dispatched to the callback method of a struct that implements the dbn.Visitor interface.

dbnFile, _ := os.Open("ohlcv-1s.dbn")
defer dbnFile.Close()

dbnScanner := dbn.NewDbnScanner(dbnFile)
metadata, err := dbnScanner.Metadata()
if err != nil {
	return fmt.Errorf("scanner failed to read metadata: %w", err)
}
for dbnScanner.Next() {
    header := dbnScanner.GetLastHeader()
    fmt.Printf("rtype: %s  ts: %d\n", header.RType, header.TsEvent)

    // you could get the raw bytes and size:
    lastRawByteArray := dbnScanner.GetLastRecord()
    lastSize := dbnScanner.GetLastSize()

    // you probably want to use this generic helper to crack the message:
    ohlcv, err := dbn.DbnScannerDecode[dbn.OhlcvMsg](dbnScanner)

    // or if you are handing multiple message types, dispatch a Visitor:
    err = dbnScanner.Visit(visitor)
}
if err := dbnScanner.Error(); err != nil && err != io.EOF {
    return fmt.Errorf("scanner error: %w", err)
}

Reading JSON Files

If you already have DBN-based JSON text files, you can use the generic dbn.ReadJsonToSlice or dbn.JsonScanner to read them in as dbn-go structs. Similar to the raw DBN, you can handle records manually or use the dbn.Visitor interface.

jsonFile, _ := os.Open("ohlcv-1s.dbn.json")
defer jsonFile.Close()
records, metadata, err := dbn.ReadJsonToSlice[dbn.OhlcvMsg](jsonFile)
jsonFile, _ := os.Open("ohlcv-1s.dbn.json")
defer jsonFile.Close()
jsonScanner := dbn.NewJsonScanner(jsonFile)
for jsonScanner.Next() {
    if err := jsonScanner.Visit(visitor); err != nil {
        return fmt.Errorf("visitor err: %w", err)
    }
}
if err := jsonScanner.Error(); err != nil {
    fmt.Errorf("scanner err: %w", err)
}

Many of the dbn-go structs are annotated with json tags to facilitate JSON serialization and deserialization using json.Marshal and json.Unmarshal. That said, dbn-go uses valyala/fastjson and hand-written extraction code.

Historical API

Support for the Databento Historical API is available in the /hist folder. Every API method is a function that takes an API key and arguments and returns a response struct and error.

databentoApiKey := os.Getenv("DATABENTO_API_KEY")
schemas, err := dbn_hist.ListSchemas(databentoApiKey, "DBEQ.BASIC")

The source for dbn-go-hist illustrates using this dbn_hist module.

Live API

Support for the Databento Live API is available in the /live folder.

The general model is:

  1. Use dbn_live.NewLiveClient to create a LiveClient based on a LiveConfig and connect to a DBN gateway
  2. client.Authenticate with the gateway.
  3. client.Subscribe to a stream using one or many SubscriptionRequestMsg.
  4. Begin the stream with client.Start.
  5. Read the stream using client.GetDbnScanner.

The source for dbn-go-live illustrates using this dbn_live module.

Tools

We include some tools to make our lives easier. Installation instructions

Open Collaboration

We welcome contributions and feedback. Please adhere to our Code of Conduct when engaging our community.

License

Released under the Apache License, version 2.0, see LICENSE.txt.

Portions adapted from databento/dbn databendo/databento-rs under the same Apache license.

Copyright (c) 2024-2025 Neomantra Corp.

Made with ❤️ and 🔥 by the team behind Nimble.Markets.