APL thread

jslade · Post by **jslade** » Sat Sep 08, 2012 2:00 am

One of my two remaining nerve cells fired the other day (I call that synapse, "Earl") and I started thinking about APL. While this language has been mentioned here before, I figured I'd start a thread.

I have a very small amount of experience with the crippled version of Kx systems Q, and thought it was pretty OK. I ain't about to buy a license though.

Purpose: suffering, and possibly doing useful things of interest to people here. Ideally it should be as easy as Q for interfacing to other languages.

Options/drawbacks:

K/Q: I guess I could download Q again. Upside is people actually use it. Downside, the crippleware is useless for real problems.

J: [url=/Show%20Post.aspx?PostIDKey=152835]mentioned here[/url]. Supposedly a K/KDB replacement, though it isn't clear how well it tracks Q. Downside: it's slow.

A+: Upsides: free, and used by MS. Downsides, being phased out, and rather different language structure from K/Q.

Dyalog APL: http://www.dyalog.com/ used by TonyC, according to the memory banks. Costs a small amount. It's unclear if it has any advantages other than TonyC uses it.

Kona: mentioned by Ralph in [url=/Show%20Post.aspx?PostIDKey=158366]another thread.[/url] Seems to be a free version of K, and work could possibly be recycled into Q. Unknown how fast it runs. Seems to be really new.

I'm leaning towards Kona, since the knowledge might be useful elsewhere, and nobody has told me if it's insanely slow. I'll probably fiddle with it to see if it compares with Q in simple benchmarks.

It's likely I won't get much farther than some code haikus, but you never know: maybe someone else is interested in such things.

doctorwes · Post by **doctorwes** » Sat Sep 08, 2012 2:32 am

Also consider NARS2000: http://www.nars2000.org/

It's classic APL plus some interesting extensions, and recently added support for native files. Also free - GPL, in fact. Don't know about performance.

Unicomp still sells an APL buckling spring keyboard.

TonyC · Post by **TonyC** » Sat Sep 08, 2012 6:05 am

> Dyalog APL: http://www.dyalog.com/ used by TonyC, according to the memory banks. ...

> ... It's unclear if it has any advantages other than TonyC uses it.

I would've thougt that'd been enough. I like Dyalog a lot, costs $1000/yr for commercial, $50 for students

APL was what I learned first, wayback in 7th grade, writing out APL in a spiral notebook so as to make te most of my twice weekly 15 minute sessions on the 2741 ibm golfball terminal

. I'm used to it. Its what I can code in best... the learning curve is a little steep

Don't know why you think J is slow, it hasn't been that way in my experience. Has a snappy new db that's supposed to be as fast as Q. All open source and free, no special symbols (which by the way, at least to my lexdyxic eyes, isn't an advantage. I find the APL special glyphs easier to recognize than J's ascii dyads)

Remember that "Q ain't K". Q is a superset of SQL written in K. Agree that $100,000/yr license is espensive

Closest "main stream" languages are probably matlab and R.

If you have an example of what you're thinkin' application wise I might be able to offer more specific insights

jslade · Post by **jslade** » Sat Sep 08, 2012 7:42 am

Dang, that was quick:

> APL was what I learned first, wayback in 7th grade,

I remember this: FWIIW, I'm pretty good chums with a cousin of the "Rose" in the book you used. I often hear stories about him doing weird things (not just weird APL things, but those too).

The above-linked thread had a second-hand claim that J isn't as fast/good as K/Q. From digging a little further maybe the differences are in the lack of chunking in the back end DB and lack of fancy-pants joins.

I dug into Q a bit two years ago, chasing a contract; worked some of the problems listed here. The exercise convinced me that it's pretty neat (like lisp in many ways), but I didn't follow up on the project. I was under the impression that there was also a bit of syntax sugar to Q that made it more legible to mere mortals, but you could drop down to K brevity if you wanted to.

At the low expectation end of things, I'd be doing this for the same reason I'm learning Anglo-Saxon. Best case, a mmap TSDB, coupled with a language I can do math in is all kinds of useful. If it can do stream processing too, that would be great. I find all SQL-like things to be loathsome, and the weird "big data" solutions that Google types have come up with also don't fit time series very well. HDF kind of works, but lots of reinvention of the wheel to make it work the way I want it to.

I guess either way, J is probably where it's at for me. I just downloaded and fiddled with Kona, and it's very cute, but also pretty basic.

Edit add: aw shucks, there is another one, which is wordy and does bigdata;

Nial

signalseeker · Post by **signalseeker** » Sat Sep 08, 2012 3:45 pm

I went this route earlier this year and unfortunately there is no clear answer.

KDB+/Q is definitely the most fully featured product in terms of what you need but unless you can pony up for license(s), it is not worth it.

Kona looked interesting but it still needs a lot of work before it gets anywhere near kdb, both in terms of features and performance.

The rest of the APL languages besides J are neat but not having real database semantics was a show stopper for me.

J did not give me a clear signal and if they have a brand new db, I am tempted to take a look at it again.

In the end I decided to stick with numpy, pandas, and HDF5. numpy and pandas let you do vector programming at the highest level of performance with pandas offering table like sematics and HDF is a decent data store. It doesn't come close to KDB+ in terms of purity of design and elegance but you definitely don't pay for performance.

And my heart still yearns for a clojure approach in this problem space.

jslade · Post by **jslade** » Sun Sep 09, 2012 1:21 am

I'm pretty sure I could do the Clojure thing *if* I had the mmap storage piece. Esper already exists. HDF5 might do it, but I'm a bit distrustful of how it was implemented in Java/Clojure, and super reluctant to involve myself in anything relating to the JNI, which will probably be necessary. Looked at bit at hector/cassandra; meh, wrong priorities.

I went through the trouble of pitching Kx on quoting me a cheap personal/developer's license. Maybe something will come of it. I like it; I just don't have $100k/year to spend on it. I could hire someone to write one at that rate, and I'd have something to sell afterwords.

Kona is a neat piece of software; it's small enough to take in at a glance. Make a great embedded language somewhere.

I've got J running. I have no idea how to plug JDB into it (there is some kind of weird CPAN thing), but when I figure it out, I'll be sure to let you know. Potential upside to J; it has some kind of plotting gizmo and UI builder built in. I think the most recent K/Q doesn't have this.

Strange · Post by **Strange** » Sun Sep 09, 2012 3:03 am

Does HDF5 actually do some sort of smart data manipulation as KDB+ and Q would?

jslade · Post by **jslade** » Sun Sep 09, 2012 3:19 am

HDF5 does some of the work (compression, data integrity, useful types, distributing it across file systems), but you have to tell it to be clever.

With a lisp, you could write macros to do most of the work if you have some useful elementary particles. With something else ... I dunno, go look at how the Pytables thing works (it's HDF5 based). It's not trivial, as there is no query language.

Another approach outlining the basic idea:

http://www.puppetmastertrading.com/blog/2008/08/22/billions-and-billions/

http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/

http://www.puppetmastertrading.com/blog/2009/01/06/tick-data-hdf5-part-2/

FWIIW, netCDF with HDF5 does a lot of work for you, with a performance hit on writes, but it still ain't easy.

Strange · Post by **Strange** » Sun Sep 09, 2012 3:34 am

I don't know what is the right solution for me. On one side, I do have a lot of option quotes, on the other side, my scale is nowhere near the scale of tick databases. Maybe a relational database or some sort of cheapo column-driven DB is the right solution for me, but so far i am selecting it blindly, since in my prior life all these decisions were done for me.

jslade · Post by **jslade** » Sun Sep 09, 2012 3:54 am

Having done some of the work on HDF5+lisp, I can tell you: dealing with an option chain would be ... extremely unpleasant.

Don't your data vendor sell a SQL DB? If you're not dealing with gigantic data sets, you can get acceptable performance out of Postgres or even shitty-pants MySQL. Presumably the vendor will index it and store some queries for speed. Squirting it in and out of R can be a pain, and is much slower than it should be (the dates cause a huge hit in runtime), but it works.

I'm visiting APL-land in hopes of discovering new coding Judo, and ramping up my productivity, beyond the TSDB problem. The "Rose" whose book taught TonyC how to write code that looks like line noise; I've seen the things he's done in minuscule amounts of time. J is pretty astounding as well, after about 45 minutes of use.