Queue Position

[Lebowski]

Hi Folks.



There are scattered references to this topic in passing throughout this phorum, but I was wondering if anyone would be willing to discuss estimating queue position for aggregated limit order books such as CME. I have found a paper from my school (unfortunately only the slideshow is available publicly I believe) as well as this SO reply.



My understanding of the problem is as follows: We know the queue position at the time we enter the limit order, which also represents an upper bound on our queue position. That being said, we cannot ascertain if cancellations are coming from in front of or behind us in the queue because we are only given the size and number of orders. We can subtract all the trades at this price level from our upper bound to give us a pessimistic estimate of our queue position. I believe this is the approach that TT uses.



My concern with the upper bound approach is that, although it will tell us when we certainly would get filled, it doesn't tell us when we *may* get filled. For example: suppose I'm backtesting a system which is trying to capture the spread. I will obviously need resting limit orders on the bid and ask. To make the spread, I need to get filled on both orders before the queue behind me on the side I get filled on first is exhausted. Because I'm looking for the probability of getting filled on both before the first queue is exhausted and not just the probability of getting filled on a single order, the possibility of having better queue position than I thought could potentially put me in trades where the probability of completing the second trade before the queue behind me depletes is too low to make money on.



My first thought was that we could assume any order with Alpha > 0 wouldn't be cancelled, and thus any cancellations would have to come from orders above that level, but since I wouldn't cancel an order that had positive alpha, neither would anyone in front of me in the queue, and thus, I will have the exact same estimate of queue position as when I just subtracted trades at the price level from my upper bound.



Now, we can also take the upper bound logic the other way: suppose all cancellations happened in front of us in the queue. In this case, we could subtract all these orders and any trades that happen from our position in the queue to get a lower bound. I don't really know how well this approach would work in practice...I feel like it could give me negative minimum queue position a lot. The necessity behind the minimum bound on our queue position would be considering the worst case scenario: i.e. we are as close as possible to being filled on the first queue and as far from possible on the second queue.



These are my thoughts on the topic so far, if anyone would like to correct me or add to any of this, please do.



tl;dr: How do you estimate queue position for aggregated limit order books?



Thanks.

[Baltazar]

What you consider alpha could not be alpha for someone else:

say a guy as a bid on the first level, it seems a nice order, one to keep.

But maybe he got filled on a similar product and does not need that bid anymore and will cancel it.



I think just use random attribution of cancels with less probability for the best prices.

Ideally you would then need to run your strategy and see how good your estimate is.



You could test it on an exchange where you know your queue position to see how wrong it is (eurex for example) but that would be making the assumption that the cancelling pattern is the same on both exchange.



Note that if someone increases the volume at a given price he will loose priority (acting like a cancel/replace) whereas if he reduces volume it will not (just like an amend)

[Lebowski]

Thanks @baltazar



I had definitely considered the possibility that me and another guy in the queue would have different ideas about what orders had alpha on them, but I had been considering different reasons such as asymmetric information and perhaps something like the guy behind me having a lower cost structure and thus is able to get the same trade done for substantially less. Great point about the possibility of him getting filled elsewhere and no longer wanting to offer liquidity with me. Maybe this assumption of orders with alpha staying in the book isn't as good as I thought.



I think you're right that I'll inevitably need to make some assumptions about the distributions of cancellations. I have been wrestling with ways to avoid the unfortunate reality that I'll probably have to have some live order fills to compare to my estimates of queue position to come up with anything verifiable for months. As an undergrad, the idea of "paying the market" potentially a few thousand bucks to gather a large enough sample of fills is a bit tough, but I think ultimately that may be the only way. I had considered trying to get my hands on someone else's fill data, but that's a bit obtrusive. I was thinking I'd use some kind of naive market making strategy for liquid futures with relatively low tick sizes like NQ to get a sample and just cap my losses at some given number. I'll test in more markets/market conditions if I scale up, but since I probably won't have the speed to get in on the level price gapped from as quickly, I'll probably try to conservatively layer the book for queue position. Therefore, my naive strategy will be looking for mean reversion, so I'll probably only deploy it in low vol situations for now. Not having colo at the moment also adds to the uncertainty, because no one would pay for colo unless it improved your queue position (ignoring factors like uptime, redundancy, etc. that colo provides).



By the way, I've been restricting my research to CME for practical reasons, but I've been interested in looking at EUREX as well. I was unaware that they provided queue position. Thanks for that info. Do they provide the trader with queue position through their execution platform or is it possible to extract that information from historical data like with an ITCH feed? I'm looking at flat market depth tick data files, so I think my data provider may have thrown that information away in an attempt to keep a uniform dataset across all exchanges. It seems like their EUREX Order by Order offering may have what it takes to reconstruct queue position anyway. As such, I've contacted a EUREX rep to ask about their historical data. Thanks for the lead on them providing queue position!



I have also been wresting with the issue of where in the queue a partial cancellation came from. Partial cancellations are tough because not only do you not know where in the queue they come from, but they also invalidate (to some degree) the assumption that you can match uniquely sized orders entering the queue with when they exit. That was pretty much my first idea on how to handle this position estimation issue



Thanks again for your help.

[Baltazar]

I think they don't provide queue position but full descriptions of orders resting at each levels.

Using this you can reconstruct queue position much more easily.



Plus keep in mind, some people don't want you to know your queue position: you place an order after someone is already there, but then they reduce/increase the volume very quickly in order to be behind you without you knowing.

I don't want to make accusation about why they do that but that's something I see regularly in the data.

[Lebowski]

Full description of resting orders is better for me. Thanks again for the info.





I know you said you don't want to speculate as to why they do this, so I'll respect that, but I find it very interesting also. Maybe they construe your order as the tip of an iceberg? I really can't imagine why they'd do this, but I'm curious. I feel like by joining the queue behind you, they're just increasing the chances of the queue behind you holding up long enough for you to get filled on the other side, but I have no idea why this would be a good trade for them. Maybe they're baiting you into thinking you'd get filled on both sides, but if you cancel or get filled, they're next in line on the trade they just tried to bait, so that might not be the case.



I've actually found that it's better in many situations when you *think* an iceberg is occurring to try to trade against it than to trade with it. I've never done this in practice, nor do I have a comprehensive idea of exactly how to classify an iceberg other than an order that continues to refresh, but that's the very limited intuition I've gained from looking at what I believe to be icebergs in my tick data.

[Baltazar]

Mecanical market, a member of this board, got a post on that on his blog:

https://mechanicalmarkets.wordpress.com/2015/04/30/market-data-patterns-order-anticipation-and-an-example-trading-strategy/

[ax]

you could keep track of queue position on a firm wide basis. so if traderA cancels his order it can get reallocated internally to someone else.

[Lebowski]

@baltazar, thanks for the link, I've read his blog in passing, it's very good. I believe my comment on trading against icebergs was more in reference to the "snap back" effect he mentions. Although this is specific to NASDAQ, it's very helpful nonetheless. Also, in reference to the to phenomenon of people adding on after you place your order, check out this post from him:

https://mechanicalmarkets.wordpress.com/2015/04/24/sarao-spoofing-allegations-and-queue-position/



@ax thanks for your reply. I can't say I understand how that would help us ascertain queue position. The fact I am a one man firm may preclude me from using this approach as well.

[EspressoLover]

> you could keep track of queue position on a firm wide basis. so if traderA cancels his order it can get reallocated internally to someone else.



This is a good idea, but traderB has to be aware that she is, to a certain extent, consuming traderA's toxicity.

[HockeyPlayer]

Be aware the CME's market data aggregates some updates together. You could be watching a price level that has 20 contracts in 11 orders and get an update telling you it is now 21 contracts in 11 orders.



Try using the data on trades to validate your forecast as to the makeup of orders at a given price level.



If you aren't working with the raw price feed data you are at a disadvantage.