Jd database?

[prikolno]

@Grisha No problem. kdb's architecture is probably lower level than what you're thinking. A kdb system is really a group of q processes that you coordinate. The q processes are like lego blocks. It is entirely up to you to architect your own strategy how to partition a table, point one or more q processes to serve those partitions, and how to load balance your client requests across those q processes. It also gives you low level messaging facilities to communicate both synchronously and asynchronously between these processes. So your answer will depend on your setup:



A single q process can be multithreaded by spawning secondary threads from a main thread. Each secondary thread maintains a copy of its own local variables and has access to global variables in the main thread's scope, since they share the same address space. Since queries are processed sychronously on the main thread and there are low level language constraints only allowing the main thread to mutate the global variables, by construction the language prevents race conditions and guarantees write consistency for a single q process.



When you have multiple q processes servicing the same data for clients, there are no concurrency primitives to synchronize I/O between those q processes - it behaves just like ordinary file I/O. This leaves it entirely up to you how to route queries to those q processes to avoid consistency issues, and there's different ways for doing this. (TLDR: The conventional design essentially creates partition-level locks at the load balancer level.) This is really rarely a problem for financial data though, most of the data is WORM and you almost never need to write to the same table partition on multiple clients. In any case, you can see the trade-offs made to achieve flat file-like performance.



Going back to your question, you can intuit from what I've described about the typical kdb load balancing strategy that the performance deteriorates as there is greater write contention, as you would expect of most databases.

[prikolno]

@procrastinatus Yes I think you get the gist of my proposal, glad you like it.



I find it's the intermediate domain that separates the top tier firms from the mid tier. Every top 50 HFT firm or quant manager whose name or abbreviation matches the regex /[A-Z]{3}/ has both a Spark/Hadoop/Databricks installation and a developer writing some one-shot .py scripts for single machine. But the top 5 are just a bit better in the intermediate domain.

[svisstack]

"This will survive up till 1 rack of servers"



@prikolno: don't see that, why?



Not excited about the outsourcing regression or messing with my threads to the database guys as I will certainly want to have control over all fine details of it. Also, it's easy to move the computation to where the data is these days.



Another risk for kdb, apart from no expertise in the job market, is k/q development linked heavily to Arthur consciousness? That things usually grow in the open-source environment. Even Microsoft bend the knee on this matter.

[Maggette]

"That things usually grow in the open-source environment."



Does it? I am still sitting on the fence on that.



I contributed to Flink and Spark and I can say it isn't a good sign when an idiot like myself is able to push his nonsense into products:).



I think people sometimes are not really aware that a lot of the "open source" stuff is driven by heavy company investment that vanishes at the whim of some managers.



Apache PIG anyone?



Flink is basically the guys from DataArtisans and Alibaba....and Alibaba recently bought DataArtisans. Kafka is more or less Confluent.



That said, I am pretty confident you can deliver the stuff kdb+ does for you on your own for less money.

[svisstack]

@Maggette: yes they do for sure and if you can push and merge to the master without any approvals the code must be low quality and mess; if you care about the quality you are not letting this happen even in the small closed dev team.



Actually, spark looks like a mess with a lot of workaround's patched/added later (student started it so it's also supporting that hypothesis), but technology is mature enough for sure and a lot of players are on board so I don't see that happening (ref-to/lack of contributions), but maybe I'm wrong and you can paint some color or 30k foot view here.

[Maggette]

I am still too young to be able to paint the 30k foot picture.



It's just that I had several projects in big and medium sized companies that were heavily reliant on open source and that I consider a failure (for example 15 scrum teams + a big DevOps and some practically unmaintainable micro service architecture on kubernetis ...they will never get rid of the 15 teams)



On the other hand I know to well the stories that are told by grizzled veterans. The company licensed some expensive shit...and then a hundred consultants came to configure, trouble shoot and "integrate" the "wunderwaffe".



Maybe the grass is just always greener on the other side of the fence.

[svisstack]

@Maggette: I meant the spark development and direction, as you told that contributed.



Success/failure is all about the people and open-source/prop-software don't play any role here as you should consider software only as a tool to accomplish some goal. Software is selected by the people in the first place so in both cases not enough "IQ" on the driver seat.

[jslade]

> I was sort of hoping J/Jd might be vaguely similar - stable, performant, productive despite the small community.



Jd isn't real Erlang like; nobody runs the telephone system on jd. It's low risk for me; I already know it, develop my novel algorithms in J, it serves my needs well, and I know I can get the help I need from the people who wrote it.



APL family is one of those things; if you spend a couple years with it, you really can develop things much faster than in other frameworks, languages, even without the wide variety of libraries available most people are used to. The problem is you really have to spend a couple of years (or a chunk of a year full time) on learning it. Hobby that turned into useful tool for business.

[prikolno]

@svisstack If it helps qualify the statement, when I say a naive file system, I mean something that comes with a single namespace, and doesn't require you to install expensive hardware like an Infiniband director switch for clustering; write your own custom drivers; design your own tiering code; or use an unofficial clustering extension. Also I keep in mind that single rack setups are usually less dense, e.g. in Equinix these would usually be on a 4.5 kVA circuit.



A naive file system can most definitely support 1 rack of servers in my experience. You can do it with DDN, Vast, Weka, Isilon with little fan fare, and the cost for commercial solutions is becoming more competitive over the years with QLC. You can still beat throughput benchmarks on a third of the EC2 instance types with an open source file system with a single head node, although I've never seen it saturate line rate with a single head node. (Usually the limiting variable is that the access pattern resembles random IO when you have that many clients hitting it at once.)



Some of your views seem localized to a different search space.



I don't feel Spark's community problems are explained by its origin. How about other student projects like xgboost? Likewise, year after year my juniors seem to win the ACM Sigmod and are immediately productive when they join the workforce, and I'd much trust these students to write better DBMS code than myself or half of the frameworks namedropped here.



And I've (fortunately) never encountered a serious bug in a linear algebra library. Even if they exist, production systems usually have trivial domain knowledge safeguards that catch these. On the other hand, I've seen plenty across the kernel, the compilers, Boost, Intel official publications, and so on, with very nuanced and unpredictable behavior that make them potentially more dangerous and harder to catch. Under a consistent world view, you're more justified in refusing to use any of these, but mere mortals like me probably can't get much work done adopting this stance.

[svisstack]

@prikolno: makes sense, but I think you are conservative with the estimates.



We currently run 1 dense rack (2x16A) in one location on a similar setup and I can see this could be doubled (2x dense rack), This is backed by the other 2 racks with 1x CX4 and 1x VNX. The answer highly depends on I/O requirements.