ZFS/dragonfly vs mongo

prikolno · Post by **prikolno** » Sat Nov 10, 2018 1:59 am

Another thought that crossed my mind: if you're offering a tech product that has multiple end users, you may need a multi-tenant architecture and a way to load balance among those heterogenous users.

You rarely have to deal with that at a trading firm. At a smaller scale, a trading firm is really just 1 team. Even if you have multiple silos across a trading firm, the major resources you'll have to load balance are just your build servers and simulation cluster, which can be done quite easily within your build/CI application and cluster middleware or orchestration service. Colo servers and trading engine instances can be added like a single-tenant configuration, you just give a couple of boxes per venue per team.

Multi-tenancy is messy if your storage layer consists of multiple, separate file systems that you're stacking to increase capacity at each site. It is easier to think of it in terms of performance (IOPS/transaction latency) of your database cluster and how to manage/throttle query traffic from your users to keep within that capacity. Throttling file system operations can be done too obviously, just lower level.

Maggette · Post by **Maggette** » Sat Nov 10, 2018 4:35 pm

"Another thought that crossed my mind: if you're offering a tech product that has multiple end users, you may need a multi-tenant architecture and a way to load balance among those heterogenous users.

You rarely have to deal with that at a trading firm. At a smaller scale, a trading firm is really just 1 team. Even if you have multiple silos across a trading firm, the major resources you'll have to load balance are just your build servers and simulation cluster, which can be done quite easily within your build/CI application and cluster middleware or orchestration service. Colo servers and trading engine instances can be added like a single-tenant configuration, you just give a couple of boxes per venue per team.

Multi-tenancy is messy if your storage layer consists of multiple, separate file systems that you're stacking to increase capacity at each site. It is easier to think of it in terms of performance (IOPS/transaction latency) of your database cluster and how to manage/throttle query traffic from your users to keep within that capacity. Throttling file system operations can be done too obviously, just lower level.

"

IMHO there is a well proven stack with several architecture options for that kind of problem outside of finance. Message based, immutable data, micro service and "reactive manifesto" are the buzz words to google. I made some pleasent experiences with quite large and highly concurrent applications with the classical appache jungle here....Vert.x and Akka

prikolno · Post by **prikolno** » Sat Nov 10, 2018 9:26 pm

Agreed.

Rashomon · Post by **Rashomon** » Thu Dec 06, 2018 9:13 pm

EspressoLover, I’m still thinking about this question (now more in relation to stable tools: some filesystem, postgres, redis, or …… I not …… a combination thereof).

POSIX has a join function and awk, or C, can be used to write a better or different one. Can’t your naming problem be solved just by ln -s’ing your files in 2 or 3 different ways?

I’m almost ready to hack inodes as well, even if it’s just writing some C functions, to keep from having to use a database. It seems like two major reasons to use a DB are (1) server/client model and (2) handle the distributed end for you. (1) seems like a waste of time relative to how a standard OS would handle it, and as you mention in ((2) flash memory is now so big that this seems like a waste

radikal · Post by **radikal** » Fri Feb 08, 2019 2:31 pm

I'm being tasked with helping out with the data pipeline for US single name equity options; I'm a bit of a grey beard on such things and leaning towards standard h5 flat files with whatever cool-kid compression is on fad these days.

Am I going to kill myself with trying to mmap /[date]/[symbol]? May have to resort to /[date:hour]/[symbol]?

rickyvic · Post by **rickyvic** » Mon Apr 01, 2019 1:21 am

if it ain't broke don't fix it. H5

Regarding ahl they used to have files to store data, they abandoned that idea, tested a bunch of vendors with high frequency data, turned into nightmare.

Some teams used oracle sql db, bcause they needed relational db.

Then some people started to work with orderbook data, not sure what happened with that.

Mongo db came when they went from R to python.