Historical order book reconstruction API for crypto markets

jshore · Post by **jshore** » Tue Sep 03, 2019 10:40 am

It might be more useful to provide normalized L3 data and/or L2 as opposed to the, sometimes very funky, json formats these exchanges use. There are a few reasons why this might be a good idea:

- many of the exchanges randomly drop trades (so u would need to do a rest call to backfill missing trades)

- your connection will drop at random or periodically, so you need to have multiple connections, ideally from different data centers; gluing this back together into a time-based sequence is easier to do with some sort of record format.

- finally, would save consumers of your data a lot of time in parsing the disparate formats.

Perhaps best to provide both, while knowing that the raw feed will be periodically disrupted and *will* miss trades. I have connected to most of the above and more and we see missing trades in the feed (when this can be detected by a sequential trade ID) multiple times per day.

tardis_thad · Post by **tardis_thad** » Tue Sep 03, 2019 12:18 pm

Agree, in fact it's something I'm working on currently (normalization of L2 and trade data). It will be available as separate API endpoint and via official API clients.

Do you have specific exchanges in mind when it comes to missing trade data via WebSocket connection?

nikol · Post by **nikol** » Thu Sep 05, 2019 7:41 pm

@tardis_thad:

Tip: there is certain interest in historical funding within perpetuals. You could imply it from your data and offer to the market.

tardis_thad · Post by **tardis_thad** » Fri Sep 06, 2019 6:32 am

Indeed, I already store and provide funding info for perpetuals (from exchanges WS feeds), but as you mentioned that could also be done by recalculating it based on exchange formula and historical market data (which exchange uses to create index)

svisstack · Post by **svisstack** » Fri Sep 20, 2019 5:59 pm

>> Thanks, I'm already familiar with CoinAPI, it's a fine service and some good points in that thread, although I'm pretty sure they do not provide non aggregated historical order book data and it's only 20 top bids/asks if I understand correctly their docs hence not something I was after as wanted to have a way to reconstruct full market state in any given moment in the past tick by tick.

CoinAPI here;-) We have the full order book in realtime in API product and historically in the flat files, the order book is tick by tick or as granular as exchange delivers and not limited by the number of levels.

Flat files are available via separate website: https://www.cryptotick.com/

Listing: https://www.cryptotick.com/Shop/Files?path=datatype-exchange-date-symbol/limitbook_full

Sample: https://s3.amazonaws.com/coinapi-samples/limitbook_full/20171229/27606-BITSTAMP_SPOT_BTC_EUR.csv.gz

Docs: https://www.cryptotick.com/Faq

tardis_thad · Post by **tardis_thad** » Wed Sep 25, 2019 6:54 am

@svisstack sure, I was talking about CoinAPI which provides historical top 20 bids/asks via REST API if I understand the docs correctly and since it's an API it's more comparable to tardis.dev API. Wasn't trying to confuse anyone, apologies if that was your impression. Your service have historical data that goes further back in history which may be important for some.

radikal · Post by **radikal** » Mon Oct 07, 2019 2:10 pm

I'm pretty impressed. I use CoinAPI for a few pieces of my crypto infra (exchanges we don't connect and record natively) but can definitely see using Tardis to fill gaps, replace missing data, etc. Access to historical funding data is a big win potentially as I don't know have that, for example.

Point being, keep it up! You'll likely have our fund as a customer =)

tardis_thad · Post by **tardis_thad** » Wed Nov 20, 2019 1:34 pm

Thanks for the feedback and sorry for late response - been busy adding new exchanges, data normalization, writing docs, 'full access' data pricing plans etc.

CSV data download should also be available soon.

Let me know if you'd have any questions or more feedback.

Thanks!