Any advancement on compiled vector oriented languages

[chiral3]

By that definition I've been guilty of being an idiot due to choices in codebases. After time I've had issues integrating modules, scalability, extensibility, latency, etc. with very large production codebases that were great initially but evolved into something miserable. Today I have a hodgepodge, but it's a smaller hodgepodge than earlier in my career that is architected much better and better integrated with my data (this latter part is largely python based). This isn't an issue for a small operation or a single strategy but for something larger, higher frequency, cross asset, and vertically integrated it's a matter of life and death.

[EspressoLover]

I really wouldn't enjoy being stuck in a python codebase at that scale either. In software everybody's situation is different in weird, subtle and unpredictable ways. Who knows what kind of idiosyncratic constraints someone's under.



But in general, there's very little reason in this day and age to be locked into any one specific language or framework. In 2018, there's tons of fantastic tooling and infrastructure that makes refactoring into services simple, safe and straightforward. It's almost a stereotype of Silicon Valley that a company gets launched with a half-baked Rails app then later down the road gets refactored into a polyglot service-oriented architecture.



I'm not even a fan of super-isolated, single-function microservices. But if someone's tired of their aging, ill-suited codebase, then there's no reason they're permanently stuck with it. In the aforementioned python codebase from hell, wait till you're banging your head against the wall working on some sub-component where python's a really bad fit. Re-write it in Visual HaskellLang.js instead of hacking on the pre-existing python. Replace the python module with a dumb client which wraps the API calls to REST, RPC or message queues depending on what works best. Then just roll the existing app and the service together in Kubernetes.



Repeat until it's no longer frustrating working on the original python app. Or until it's been refactored out of existence. Besides migrating to containers, which you probably should be if you haven't anyways, there's no overhead to this approach. The dev just spends his time refactoring instead of polishing a turd. Ops just deploys containers instead of artifacts. End-users see the same frontend and APIs. The primary reason this strategy might now apply is if the original codebase is so tightly coupled that there's no clean way to slice it into sub-components. In which case there's major issues with the underlying software engineering. Switching languages would just be re-arranging deck chairs on the Titanic.



This definitely doesn't apply if you're doing anything HFT or latency-critical. Anything that's ever called in the hot-path needs to live on a pet (not cattle) server, run bare metal, avoid IPC, and have a meticulously tuned API. But let's be realistic, well less than 1% of the code written on Wall Street will ever be invoked in this context.

[ax]

a downside of python at scale is dynamic data types and everything is runtime. to the original question python has this http://numba.pydata.org/

[EspressoLover]

But Julia's also dynamically typed. Even though they tend to be correlated, we should remember that typing and interpreted/compiled are two different things.



Going back to the original motivation for the question, there's an inherent tradeoff. Static typing is good for large, complex, long-lived systems with a lot of interconnected pieces where runtime failure is expensive. But dynamic typing makes research easier by facilitating rapid prototyping and flexible coding standards.



If the goal is to unify research and production code, then you inevitably have to come land somewhere on this spectrum. But in its defense, python3's typing module does provide a pretty nice system in this context. It makes it pretty easy to barf out untyped research code, then gradually annotate with it with typing as it gets promoted along to full production. (Though to be fair my understanding is that Julia takes a similar approach.)



Otherwise the alternative is to go with the flow of research and production having in many ways fundamentally different technological needs. Often the best tech stack for one will have major drawbacks for the other. But if that's your philosophy, you're going to have to accept the inevitable burden of re-writes when logic graduates into production. Either way there's no silver bullet, and there's always some bullshit that comes with the research/production dichotomy.

[chiral3]

My core is still c++ (and cuda). Millennials hate c++. Too low level for them

[bullero]

Echoing EspressoLover - use the right tool for the job. Even if you can ride a motor scooter from Boston to Bogotá that does not mean you should.



Your python and such are mainly for scientific computing. How I see them is that they allow you to do prototyping and write proof of concepts very fast. Yes you can write a trading system using them but no you shouldn't. It is so easy to paint oneself into a corner when you can just throw together some code without static type setting. Later, things get messy when a lot of people are making small modifications to the code which imposes no type checking until runtime. Also, it puts higher demand for other people to study the code and try to interpret what is going on. Yes you can leave comments or write documentation but they are usually infested with lies damn lies. When you enforce static type checking its very clear what the writer means. So, help the readers of the code please. Also, compilers check your silly mistakes which generally is a good thing.

[TonyC]

> My core is still c++ (and cuda)



thats why you roll your own APL to Cuda cross compiler, so you don't have to learn too much new stuff that's not as intuitive or as slick as the old stuff you learned in 7th grade

[indiosmo]

Another vote for using the appropriate tools.



I use python for stuff like processing end of day reports and other non-critical systems and it's great, allows for quick and easy changes.



For anything in the order path nothing beats the safety of the type system, along with good tests and sanitizers.



Also if you can use it, modern C++ with modern libraries like boost range, nlohmann json, fmt and spdlog can make a lot of previously laborious tasks easier.

[rickyvic]

@fdaxhunter I know why, it is because it is a mess to debug and a lot of libraries you need to know how they work inside to avoid bugs, then versions and compatibility is another issue.

I live a much nicer life using matlab, which is faster by orders of magnitude if you write code correctly thanks to jit, install it and it runs, you can even compile it.



Just it is not a commonly used programming language and you cannot hire coders, a least not easily. Almost not a programming language IMO, but it works.



For latency sensitive work you need to port it to low level languages unless you are suicidal, some use functional compiled languages, but I cant get my head around it, at least not without wasting days.

[prikolno]

Echo EL +1. FWIW it's actually a lot easier to debug Python than MATLAB, for various reasons that I'll try to walk through with real world examples.



1. Significantly better testing frameworks, e.g. property-based testing.

Think of a simple test case for an order book: if you insert 2 bids priced p1 > p2, then `getBidPrice()` should return p1.



What you really prefer is to have some guarantees that for all numeric price types p1 > p2, this holds.



In MATLAB, you just test some sensible numbers like p1=1 and p2=0 and in the ideal world, you've also built some pseudo-inductive reasoning with other test cases like the base case that the book accepts zeroes as prices (since the instrument may be a spread) and the case that ordering is preserved for negative prices. In many situations it's not so obvious and you just manually pick a few 'privileged' values like -1, 0, 1, type max, type min, and hope that you've covered a good part of your domain.



With a testing library like QuickCheck (Haskell) or hypothesis (Python), the test cases are randomly generated to match your distribution or type properties so you don't have to try exhaust the cases manually, you just need to specify that p1, p2 are numeric types and the library builds the test cases for you and even caches where the cases have failed in the past or narrows down the simplest case where your test fails.



It's also sensible to have microbenchmarking built into the testing framework, which modern testing frameworks like pytest and criterion (Haskell) provide. You've already gone through all that effort to mock a bunch of order objects, why not also benchmark how fast your book inserts each order?



2. Strong typing.

Back to the order book example, you really want to limit your function to accept arguments of an integer type or fixed decimal price type that you wrote. Why? In MATLAB, your order book functions will work even if someone passed in orders with floating point (or even singleton array) prices because it's weakly typed. Inevitably, some junior guy on your team will write a lot of code assuming this invariance since the test cases will still pass. Then later someone joins your team and wants to trade 2 year treasuries or JPY and now you have a cascade of code to clean up.



3. Static type hints and optional typing.

bullero's post brings me to the incredible feat of engineering on Jukka's PhD thesis and Lisa Roach's part that have made it possible to resolve many of the static typing complaints for using Python in production. Many of the points in his post don't apply after Python 3.5 (PEP 484/526). Instagram, Amazon, Dropbox all have near 100% test coverage with static type hints as you should if you're running a big firm like FDAX's friend.



As with many dynamically-typed languages, often you debug a runtime error on a function `foo` involving variable `var` by tracing back to previous call sites with `foo` or `var`. It's painful to do this with grep/ag and much easier if your code base has static type hints.



4. Better scoping, no massive, polluted global namespace.

Self-explanatory.