APL thread

lmog · Post by **lmog** » Sun Sep 09, 2012 7:12 am

A bit of an off topic, this thread reminded me of a performance attribution app I wrote in KDB/Q...it took a while to get my head thinking in Q but by the end of it, I felt I really had perhaps gotten a glimpse into Iverson's head...

Apparently this documentary is still under active development (a bunch of APL related blog posts too):

http://www.aprogramminglanguage.com/

signalseeker · Post by **signalseeker** » Mon Sep 10, 2012 1:53 am

jslade, please let us know how J pans out.

jslade · Post by **jslade** » Mon Sep 10, 2012 4:53 am

TLDR Day+1.5 update (for my own self anyway): I'm running through some demos now. This language is pretty cool; very "metal" in the same way Lush was. The IDE is pretty complete; project manager, debugger, CPAN like thingee, offline hyperspec, some kind of autocomplete, etc. It does this neat thing where you can run it in your browser, which is useful for running the wiki examples. I'd say the environment itself is extremely well thought out, which is very encouraging. Most such things are obviously retarded in some way; I haven't found a flaw in this one yet.

Available code of interest to NPeeps:

1) plot -excellent plotting gizmos. Here is an example from the plot demo; note that unlike R, you can use a dropdown menu to dump the plot. [url=/User%20Files/2442/plot.pdf]Attached File: plot.pdf[/url]

2) Optimization: differential/GA; simplex, nedler-mead -nothing special, but "the basics."

3) General math: fftw, lapack+superset/alternative matrix lib, splines

4) Stats: normal distribution coverage (CDF, PDF, etc). Chi-sq CDF, an incomplete set of functions for Poisson & Binomial. Histograms, combinatorics, mean, std dev & etc. A gizmo for talking to R is where important things will happen. It would be awfully nice to have a regression gizmo and some robust filters that work on bigdata. I guess it would make an OK first project.

5) Reading data in: CSV gizmo, arbitrary delineations, excel parser, json, major databases, regex, various native parser paradigms semi-documented on the wiki.

6) Datetime type (add on package; hopefully the database supports it in a clever way); does milliseconds. Pretty rudimentary: it's not the R timeDate package.

7) Gizmo for decompiling the impenetrable syntax into something resembling english (haven't tried it yet, but I plan on using this)

JDB: performance "OK" so far. It's definitely a columnar database, which is awesome. The query language is the familiar stuff. It does do auto chunking; I'm at double the 8G physical memory on my laptop, and it's still doing the right thing. I'll have to set the chunking parameter to be a larger number (it's presently low) to see how this effects performance; presently, it takes only a second or so to do complex multi-column queries plus math. I need to make the data more like 100G and see what happens, but I need a faster way to populate the DB (I'm using a demo script which might not be optimal). Note, I'm not using a SSD yet, and it's a laptop. There is a schema grapher and browser; I haven't made them work yet. The memory management on J/JDB is insanely great. While SQL sits there and hogs memory even when it isn't doing much, you hardly notice anything happening when running a giant query in JDB. Obvious downside here: if there is HDF/Q style online compression happening, I can't see it. Maybe something can be turned on. Or maybe I can put it on a filesystem where it won't matter.

Legibility: I can write simple things with difficulty and understand what I wrote, but most other folks code is read-only so far. Some write in an idiom that can be understood; the guru code, though, looks like line noise. Decent result this early in the game. Upside; you can do very complex things in one line of code. I found a bunch of neural nets in an old paper on J; took up one page. GA; half a page.

Language: It's weirder than Q, but the same basic idea; sorta ugly-lispy with right to left execution instead of s-exp parens. There are various FFI's I haven't yet looked at. Since APL comes from the dark ages before they invented file systems, there's going to be a lack of primitives for simple stuff (like in the case of Common Lisp). I mean, there are add on packages for dealing with the concept of "directories" and csv files. It is very handy for our kinds of problems, though, despite the lack of existing code.

SS: if there's anything you want me to try to take JDB through some paces, let me know. Check basecamp.

TonyC · Post by **TonyC** » Mon Sep 10, 2012 7:11 am

"programming in the twenty-first century" http://prog21.dadgum.com/48.html

The" World's Most Mind-Bending Language Has the Best Development Environment"

TonyC · Post by **TonyC** » Mon Sep 10, 2012 7:31 am

Pretty much all matix algebra is built into J and APL... Frinstance, in APL, the single non ascii character "domino" (Domino matrix) when called monadicaly returns the inverse of a matrix, when called dyadicaly (vector Domino matrix) it returns the regression coefficient(s) where vector are independent observations and the columns of matrix are the dependent observation [EDIT, OOPS, OBVIOUSLY I GOT THAT BACKWARDS, vector are the dependent observations & cols of matrix are the independent Observations]

I think the J equivalent of the single APL non ascii character Domino is the two character ascii dyad " . % "

look at inner product, outer product, and the J equivalent of Domino for some cool algos

TonyC · Post by **TonyC** » Mon Sep 10, 2012 7:38 am

Also, check out the "J-ottings" column in the "vector" journal of British Computing Society... vector.org.uk

braincat · Post by **braincat** » Mon Sep 10, 2012 4:42 pm

I have played with HDF5 and pretty much replicated the scheme that was described in puppetmaster articles for tick storage, using HDF5 packet tables. The whole thing is quite bare bones and a lot of additional programming is needed to make it useful (e.g. after implementing a bidirectional iterator, stl algorithm could be used for binary search, etc.) Some things in HDF5 are counter intuitive, for example, I discovered that opening a database handle read only and then trying to write to it or to open nonexistent packet table corrupts the database for me, so read-only open cannot be relied upon to protect against accidental writes.

And the database is corrupted if a writing process is killed before the db is closed, but that is hardly hdf5 specific, and not a good idea to capture live ticks directly into the database anyway.

jslade · Post by **jslade** » Wed Sep 12, 2012 6:55 am

Thanks for those links and pointers, TonyC. FWIIW, matrix inverse is '%.' in J land. Piece of cake. I'll have to compare the performance with the lapack libs to see what we're up against here. I'd like to encapsulate things into some kind of object/container with helpful things like p-values and a plot method. It will take me a bit before I figure that out. No clue what the "J way" to do that would be. In Lush it was via the object system.

Thanks for the comment, braincat. I didn't get as far as you did, but I could see I'd need to be compiling query variations fairly often. FWIIW, you might consider using some of the netCDF4 infrastructure. There is a way to dump streaming data without ruining your day by preallocating a table (you could grow dynamically, but that's super slow), then adding it to the other stuff later.

FWIIW: further update on J fiddlings: it talks to R pretty easily in the J->R direction using RServe. I'd prefer to call J from R with preloaded functions (aka, use J/JDB as a data gizmo from R), but I'll need to code that up. Managed to persist data in JDB.

TonyC · Post by **TonyC** » Thu Sep 13, 2012 10:04 am

some stat & simplex stuff in J

http://webdocs.cs.ualberta.ca/~smillie/Jpage/Jpage.html

jslade · Post by **jslade** » Thu Sep 13, 2012 10:45 pm

Thanks TonyC. A lot of those examples have made it into the CPAN thing that comes with J.

This one has some different stuff:

http://www.astro.umd.edu/~jph/J_page.html

Going through some tutorials this afternoon...