Market making models

[Maggette]

At first you should read doomanx signature:)



A more constructive remark: what would your agent "play against"?



IMHO you would need a kind of realistic simulation of an LOB market and all its effects. This is a hard thing to do (impossible if you ask me). And if you had it, you could probably monte carlo tree search the problem.



You could go down the rabbit hole and try training a GAN to replicate the LOB, and than run RF against that Big Smile



Disclaimer: I don't know much about market making.

[prikolno]

If you're looking for something easy, I think reverse K-S is better than RL and optimal control for MM.



@Maggette No, you typically wouldn't bootstrap with simulated data from a multi-agent setting. You'd just use production data.



The difficult part about RL for MM is that there's a lot of plumbing that I don't think any academic paper will discuss, without which you're likely better off reformulating this into a setup which you can attack with classical literature on optimal control like I described. Label resolution is one of many that's easier to explain: Let's say your label resolves on uptick/downtick, then on each uptick/downtick you have a cascade of model updates. Not easy to make it run in amortized constant time with typical patterns like speculative LUT (described in Matt's post) because there's an extra dimension in the number of feature vector observations since last uptick/downtick.



I don't have MS lawyers preventing me from explaining all the plumbing, but I think a more fundamental issue is that you should condition on the problem not condition on the tool. Otherwise you'll just find simple lookup table will beat whatever sophisticated tricks you use, which is quite typical of problems that you might throw RL and approximate DP at. There's many parts to MM that you might address with RL, not just the prediction alpha. In some cases, you'd solve it in a way that the prediction alpha comes for "free", in some cases the optimal policy comes for "free".



@EL: No problem. Likewise, your comments are always appreciated.

[gaj]

> You'd just use production data.



You mean you let the RL learner modify the trading parameters and influence your actual trades? Sounds a bit dangerous, and doesn't seem like you would get enough data.

[prikolno]

>Sounds a bit dangerous



Why would it be dangerous? When you A/B test a parameter by hand you'd generally let it push 10s to 100s of bps ADV to see how it works right? In many cases that's the only way to know. A lot more dangerous to me than letting a learner do it in a principled way.



You would absolutely have enough data, you just need to reduce the state representations to control the rate of convergence. In typical ML applications you would learn the reduced state representations with a NN, but for MM there's very effective heuristics.

[gaj]

Interesting. Didn’t know that people actually train RL in production. Like Maggette, I thought you’d normally train it on a simulator.



> In some cases, you'd solve it in a way that the prediction alpha comes for "free", in some cases the optimal policy comes for "free".



Do you mind elaborating on this part? This sounds extremely interesting, but I don’t quite get how you get either the alpha or the optimal policy for free.

[doomanx]

In all seriousness, I don't think market making with RL is a good idea. As Maggette alluded to, building the interaction with the 'environment' is like perfectly simulating the market reaction to any order, which is difficult to do. As prikolno said, what you can do is use your current parameter settings as a initialisation point and use a RL type method to update your current policy with production data.



I have had success applying it in optimal execution situations. There is a lot of literature on the cost of execution of different types of orders. You might be interested in https://arxiv.org/pdf/1403.2229.pdf for some ideas. There is also earlier work by Kearns where they introduce this idea as well as RL for optimal routing between different venues.

[bullero]

Most of the models do not deal with the queue position of your limit order. They suggest a quoting strategy which is not realistic.



A filled limit order may be regarded as an option to make the spread and the option value depends on the current and anticipated queue configuration(s). Essentially, it is better to be filled when there is more stuff sitting behind you.



Now, someone here said HJB does not incorporate the queue position. This is not an entirely accurate statement. HJB is only a tool to describe the evolution of an abstract "value function" given your state and given that you do not touch the environment. Now, is it possible to take into account the queue position in the HJB? The answer is yes, but you have to make certain technical changes to the statement of the problem. You need to transform the problem into an optimal stopping problem which relies on the queue position, queue imbalance, order flow imbalance etc.. Also, you need to transform the way you treat time in the model. In the existing academic literature you get optimal spreads around some reference price for some fixed point in time which essentially means that you do not take into account book changes. Instead, what you need to do is to define time as time until next event (marketable order arrival, limit order addition or limit order cancellation). After all these rather large modifications you get a set of HJB-QVIs that is applicable to one book configuration. There are quite a lot of combinations of these... You solve all of them numerically over a large grid of possible parameters, save them for later use and load them into memory when you do trading. The solution is a binary mapping from "market state" to decisions to cancel, insert or do nothing (conditional on your state). The math of this stuff is quite trivial, the most challenging part of this is the management of your computational resources. Why? Your HJBs will be quite jumpy between different book configurations and this will creep into your finite difference scheme. Why is this a problem? It is a problem because your optimal decisions will depend on the relative value of the value function in different states: should I cancel now or roll a dice and do nothing (in which case I might get the option to the spread + expected rebate). Now, if there is numerical instability in the HJBs you tend to get oscillation in your decisions if the oscillation in HJB-QVIs hide the option value + expected rebate that is baked in the value function. So, not only are you solving over a huge space of possible book states but you are also solving over a large inventory time grid.







edit: typos etc.

[nikol]

@bullero



perhaps, you suggest to solve BHJ-problem with Bellman recursive equation on top of Markov discrete process, where decision policy (immediate order execution strategy) is a solution. Since HFT is discrete, it is even more precise way of solving the problem.

[bullero]

@nikol,

[Jurassic]

are we talking about mming on exchanges, because i didnt think the banks had anywhere near the sophiscation of hjb