What stack are you using for Logs/Monitor/Alerts

[prikolno]

Glad it was of use, I thought it was too verbose.



>On modern versions of the linux kernel, clock_gettime() syscalls never leave userland thanks to vDSO.



Yes, sorry, I use that phrase as a force of habit. What I mean is when you want to offload making the system call or issuing rdtsc(p) to the host CPU. This is less of a matter of latency but more of a matter of accuracy and determinism.

[nikol]

@prikolno



Shame that you have cut that valuable rant. Apart from qualitative stuff which I stored into my memory, could you reiterate the list of components?



Here is summary of the above to make your life easier.



Logging: log4net, Elastic Search, RabbitMQ

Metrix: Graphite, InfluxDB (+Telegraf), Prometheus

Monitoring: Grafana, Prometheus

Alerts: Grafana, PagerDuty





====

From my side, you guys look like Titans, so, perhaps, there is no interest in my DIY projects, but anyway, my two cents.



Having spend years in risk management in office at home I developed some "slow" algo-trading platforms first in Java and now in python, with

Logging: flat files

Metrix: flat files + zipping (tried HDF5, very good, but did not find a will to force myself to continue developing infrastructure around, focusing on strategies)

Monitoring: not yet... choosing. I tend to think about something like plotly. In Java it was Swing+JFreeCharts.

Alert: 10 years ago it was simple check of PnL consistency between my accounts and exchange. If anything wrong - Loud Alert in the house (my wife was complaining) and simple SMS. Now we will do the same.





A lot of thanks to all of you! Valuable read.

[prikolno]

nikol: Logging, metrics and monitoring are mostly just composed of UI, messaging, serialization and persistence, all of which can be homebrewed. The external dependencies here could be any native GUI frameworks (GTK, Qt, Cocoa), web-based UI libraries, or the inbetween (Electron), a 3rd party brokered or brokerless messaging layer (RabbitMQ, Solace, Redis), any serialization format (protobuf, bson, msgpack, hdf5), and/or any DBMS (mongo, postgres, redis, snowflake).

[Maggette]

"Fill rates, exposure, rejects etc. If you have a strategy that tries to hit an order that you speculate will appear before any of your own fills get acked or any market data event is broadcast, then rejects are fairly important. "



Also thanks for the valuable input.



These are "features", like in "what variable am I tracking?". My question was more geared towards "given that I decided which variable represents the state of my system, how or when do I throw alerts?"





A simple threshold based on risk management or technical considerations? Some control chart thing like the CUSUM algo I mentioned? Do I throw warnings? How to calibrate to enhance precission and recall?



Monitoring is a exercise in state detection or classification. So which approach is used to detect something fishy?

Thx

[nikol]

As an alert system I ended up with Telegram bot which allows to send push notification.

It seems to be quite straightforward. Downside is that I must have internet connection on my device.

This can be solved by Telegram API (more complicated) which can duplicate the message by sending SMS.

[svisstack]

@EspressoLover: Understand your reasoning, but elastic deliver more value when you are using structured logging, then you know from where it came and you can tag and later search the logs by any property. It's also easy to set up it in the cloud right now, so there are no longer much maintenance regarding upgrading and managing the stack (ml, kibana, elastic, etc). We are using it in that setup, pushing both to the std*, file and elasticsearch database.

[quantie]

@Maggette I like cusum tests as they are intuitive recently saw this paper.timeseries change point techniques

[Maggette]

Excellent. Thx.