Any advancement on compiled vector oriented languages

[doomanx]

In the end tools like pandas/numpy aren't designed to be used for *massive* datasets. If you get to a size where these operations become painful it's a sign you might need to switch to doing things in memory or distribute, for example by using Dask https://pandas.pydata.org/docs/user_guide/scale.html.

[Maggette]

@kloc

You are probably right. If I think about it, I actually convert a lot via ".values" while using pandas (probably most of the time when slicing) and hence probably numpy than pandas. In addition I use dask when the data set is large and fall back to the "for looping hell" using numba when my code is too slow. Probably that's why I never felt pandas was slow because I never pushed it to the max.



@doomax I made very nice experiences with dask so far. The PySpark API of Spark is alright as well and I use it in several projects. But dask feels easier to use on a single machine.



and by the way: your signature is awesome!!

[sharpe_machine]

Getting back to Julia... What are some examples which motivate one to switch from Python (Numpy/Pandas, sometimes Numba) to Julia? Assuming Python is being used to create features from raw data and then feed it into a "black box" algorithm written in C++/Rust (already top performance)

[prikolno]

>What are some examples which motivate one to switch from Python (Numpy/Pandas, sometimes Numba) to Julia?



Only one I can think of is native support for multiple dispatch, which can be really nice for building features.

[sharpe_machine]

Does not Numba's @jit without signature implement sort of multiple dispatch in Python? It starts with some "default" signature and then re-compiles a function given a new unknown signature which then stored for future calls. Not sure how dispatch is implemented in Julia though.

[smt1]

I don't have much experience with numba so I can't compare its dispatch system relative to julia's. I mostly use cppyy w/ cython&pypy depending on what I'm doing.



I think Julia's most clever design choices are combining multiple dispatch and the macro/metaprogramming system with a python-like syntax.



Like this guy: https://flow.byu.edu/posts/julia-c++ I've run into occasional gotchas that absolutely kill performance in Julia if you're trying to write superelegant terse numerical code that matches C++ performance w/o doing a lot of profiling, but the situation seems to be steadily improving w/ each julia version.



I've also found julia's distributed/parallel support really great, as you'd expect from a modern language that didn't have to deal with legacy cruft.



Julia's multiple dispatch system has a lot of interesting consequences. I think it's tooling of libraries will have a very different graphical structure than something like python.

[rickyvic]

I find the julialang very interesting and python I am hating it more and more over time.

However python has become very powerful for everything except low latency stuff where C and C++ are necessary IMO.



Any way we can use julia to work asynch without having to use dedicated threads?

For research I dont think there is a need to switch to julia but I would be interested in using it in production for real time applications (so no research code rewriting).



research: I am still using matlab and hdf5 (they work reasonably fast), then you need recode to C and frankly it is really time consuming for me.

[Jurassic]

@rickyvic why the new hated of python?

[rickyvic]

I just cant stand the syntax, but more than anything I cant make it work fast in loops hen it is necessary to use them, or better it is a big waste of my time to look for solutions to speed that up.

Then add the versions nightmare and compatibility nightmare and here you go.



But I reckon it has a huge amount of accessible functionality in many fields and probably if you can work Cython or write C code here and there it can be as fast as you want.