optimally storing time series in an object oriented language

frstwrldprblm · Post by **frstwrldprblm** » Mon Mar 28, 2011 3:39 am

just throwing it out there to get a feel for how people go about doing this. i do not store data locally but rather retrieve from web/service. when the data is brought into local memory i used to just index by location (1, 2, 3...etc), but over time realized that this is beyond sub-optimal.

I have come to heavily rely on the hash table object (java hashmap) to easily loop through tickers:

for(String i: hashmap.keySet()){} //etc..

or through dates

for(double i: hashmap.keySet()){} //assuming YYYYMMDD

good thing about using hashmaps etc is to guarantee dates are synchronized cross tickers where actual location in array (although should be identical) is not assumed by code.

is this a wide practice? is there a better way?

thanks

FDAXHunter · Post by **FDAXHunter** » Mon Mar 28, 2011 7:35 am

Depends on the nature of your time series. If it is daily data for a bunch of stocks accessed sporadically, some sort of dictionary can work and hash tables will do the job. In that usage scenario, pretty much anything will work and I would roll with whatever is easiest for you to use.

Keep in mind that wrapping things in an object invariably leads to more overhead because of accessor functions. This may work if you need to access the time series once or sporadically, but it certainly won't if you have large time series more than say, half a million entries, that are accessed repeatedly.

In such a case, a raw array still works best and it's most useful to have all the alignment already done server-side.

If your time series are sparse (i.e. small timesteps but most entries are empty. This can happen because you are aligning an infrequently traded instrument to some other time series) it may be beneficial to use an object wrapper again so that you are able to avoid storing the empty elements.

Bottom line is: really depends on your usage scenario.

jslade · Post by **jslade** » Tue Mar 29, 2011 2:45 am

For daily series, why not go all matlab and stick them in a matrix indexed by time, use a hash table for the subsettings by ticker/filter and take windows. Ram's cheap; matlab style coding works and has no OO overhead.

This guy has some interesting thoughts on storing tick data:

http://www.puppetmastertrading.com/blog/2009/01/06/tick-data-hdf5-part-2/

I had something like it half cooked up using netCDF. Then I tried compiling it 64 bit, and got discouraged. Same ideas kind of work with flat binary files though.

goldorak · Post by **goldorak** » Tue Mar 29, 2011 2:25 pm

We are using the same principle, but in Perl, for most of our applications. This is very useful for time series that do not match exactly for whatever reason.

As FDAX mentioned this is all dependent on the size of your data and the frequency at which you are accessing them. If frequent access is a necessity, you will want to create a function returning an array (or a matrix) once and for all.

frstwrldprblm · Post by **frstwrldprblm** » Tue Mar 29, 2011 2:48 pm

Agreed. On all points offered here.

Yes for data coming from reliable systems hash map obj is a bit overkill, yet for some data that may be sparse coming from different locations it is extremely useful.

I did not think I was doing anything original, but it is also good to know I am not doing anything extremely sub-optimal. Thanks for the input.

jslade: thanks for the link, I will take a look but may be some overkill for what I am trying to do.

Nonius · Post by **Nonius** » Fri Aug 26, 2011 7:21 pm

Anyone know decent C++ Time Series classes? Googling, first link is some Norwegian dude's freeware. Second is some classes that cost 400 euros.

Context is intraday trade, price volume data + some relatively low overhead pattern recognition functions.

Scotty · Post by **Scotty** » Tue Aug 30, 2011 2:32 am

I'm using the pandas library in Python.

http://pandas.sourceforge.net/

redandtheblue · Post by **redandtheblue** » Wed Aug 31, 2011 3:30 am

Pandas is great for python and really worth looking into.

Unfortunately, I (as a former heavy c++ guy) do not know of any.

Polter · Post by **Polter** » Wed Aug 31, 2011 10:12 pm

Unfortunately only of historical interest right now [yeah, I know ${L_SMILIES_SAD}$ ], Boost.Time_series looked promising:

http://boost-sandbox.sourceforge.net/libs/time_series/doc/html/

It has been accepted into Boost:

http://lists.boost.org/Archives/boost/2007/08/126147.php

You can still download it:

http://boost.2283326.n4.nabble.com/Time-Series-lib-td3223924.html

However, the last update I could find (1/19/2011) states:

"It hasn't been maintained for a while, though. You may find that it doesn't work with recent versions of Boost."

http://boost.2283326.n4.nabble.com/Time-Series-lib-td3223924.html

For now I just use a combination of containers (STL and a dense/spare numerical linear algebra library) and implement whatever I need when there's no good library available, but a specialized time-series library with a bunch of time-series-specific functions could be nice...