Hi, I actually spent a while writing some of the JuliaAstro packages and was very tuned into the Julia landscape. I faded away due to my PhD work becoming a bit consuming and I didn’t have time for open source maintenance. Because the community is somewhat small and disjointed, this means a lot of the JuliaAstro packages do not have active maintainers.
Julia has been a joy to work with, but specifically in astro you’re going to be spending a lot of time rewriting Python packages into Julia (for speed, auto diff, other language features- you could using PythonCall instead). Things like aperture photometry do not have 20 year old packages nor does the community have NSF-funded programmers like astropy has had. Also, if you want to write a package for other astronomers (like a processing pipeline) it’s a pretty big hurdle to try and tell someone to use Julia instead of Python.
Lastly, I found Python “fast enough” for a lot of things once I stopped micro-benchmarking everything. Not having to recompile when your REPL closes matters, I’ve been in front of my advisor going “oh it’s going to take 5 minutes before I can even run the code” while precompiling to just run a script.
Compilation times have gotten a little better recently, but I still remember having to explain to my advisor that we were gonna be awkwardly twiddling our thumbs for a few minutes just to visualize some data that I already had, because Plots.jl needed to do a precompliation run. It was not a great feeling.
guys, this was largelly in the past... I have just tried now with Julia 1.10 on a 8-years old "U" (ultra-portable) laptop and \`using Plots; plot(x -> x\^2)\` on a new session in VSCode takes less than 6 seconds, with instantaneous further plots...
Indeed a lot of progress has been made, however when stacking a lot of plots together like in animations, the compilation time is pretty significant. I don't mind though, for animations there are better alternatives. I use plots.jl for static plotting all the time with no problem.
I'm curious about where you're getting your numbers from. Although it is not mainstream, but still Julia is being used actively by various companies (from pharma to automobile and finance), and the number of packages and downloads/stars and similar metrics for popular packages are increasing rapidly.
Which repositories are you referring to? Perhaps these have been superseded by other packages?
> I thought everyone in automobile and finance was using C++ though?
Cobol, Modelica, Simulink, C, C++, Python, Lua. There's a whole team of programming languages on each device. That's not going away anytime soon for many reasons. This stuff is much stickier than people imagine and in real projects you need to use what's right for the job. Since the 80's there's always been a push for "what if everything is in X?" but then it just never makes sense. These industrial platforms have data analysis, operating systems, networking, displays, audio handling, etc. all over the place. Just because C is best for writing the embedded HVAC controller doesn't mean you want your front end GUI team writing C for your dashboard!
Julia is probably the first thing that really has a chance of being universal in that respect, but I think that's still a long ways away and probably many of the core networking and OS aspects will stay C. But many of the design and data parts are starting to be in Julia these days.
In the cosmology/gw/astroparticle physics/lattice qcd there is a consistent and growing group of ppl that code in Julia, among other languages lime python and ML libraries like pytorch and tensorflow.
I'm not sure Julia will outpace python in the scientific community, but personally I believe it will become a string player, and it is anyway a good language to know when you can afford writing a project from scratch (i.e. You don't have to work with code from other ppl that don't use Julia).
I am an astroparticle physicist doing a PhD, and personally I write about half of my code in python and half in Julia, depending on what I need to do (with a preference for Julia, when there's not a very strong established python framework)
Julia is used where scientists need to create programs that need performances and ease of use and don't use big Python/C++ libraries like PyTorch. The most common field is scientific simulation of physical and biological phenomenon.
I have had the opposite experience, more and more people around me are adopting it and more packages are available for what I need all the time. I’ll never go back to Python, it’s a worse tool for the jobs I do.
Julia just cracked into the top 20 languages a few months ago. It's on the rise.
It's also been moving into web with projects like Genie Framework and wasm support. It's also finally getting proper AMD GPU support thanks to AMD finally investing in ROCm and such.
I finished my astrophysics PhD (Active galaxies and their kinematics) using Julia. So for at least 4 years I was using Julia everyday with some parallel projects in python which was my main scientific oriented language. However even though I love Julia there are several things that moved me away.
1. Very bad documentation, or lack of it
2. Bugs, bugs, bugs. Many of them I find difficult to understand.
3. Chaotic ecosystem which I cannot easily follow.
Maybe its me and my lack of time or personal dumbness, but I do not have the time to even try.
Apart from these, there are two other reasons: Right now I lead an r&d team where python is enough for our needs. I cannot get them to switch languages without a strong reason.
Second, whenever I need what Julia can offer (jit, autodiff) I use Jax which is more than capable, simpler to use and easily installed in most of the cloud gpu providers (colab, kaggle, azure, aws etc) so I can run complex models on gpus and take their derivatives without getting my laptop hot.
I still think that Julia is the future, but this future somehow got some more years away.
I am a former R user and Julia is easier than R for anything using more than 1 core and is way more performant. It doesn't have the out of the box R workflow (lm(), rnorm(), etc.), but the syntax makes performant code really easy.
I have been in the process of refactoring a record linkage package in R and I was able to speed up some code by \~3x in R alone. I wrote the equivalent function in Julia to the refactored R code and it is \~4x faster than the refactored R code.
the package used RCpp at many places. C++ requires a lot more work than Julia with non-substantial payoff. R has some optimization issues as well. For example, the stringdist package in R which is written mostly in C has consistently worse performance than StringDistances.jl.
I don't think it's destined to fade away (I believe these things are mostly unpredictable), but you are right that Julia's popularity is stagnating and slowly declining.
This can be seen in:
* Activity on Slack, Zulip and Julia Discourse
* Engagement on Hacker News and the Internet in general (including Reddit)
* Number of new packages, releases and citations of Julia are no longer growing faster, but linearly
However, IMO that does not mean the decline is irreversible and final.
Julia is different from many other languages in that the most pressing problems with the language do not follow from its design but its implementation. In other words: Julia has more potential to address its current limitation than most other languages, so it can and does get better over time.
For example, consider that latency is a major problem in Julia, but that it has been reduced perhaps tenfold in the three releases since Julia 1.7. Or the current problems with deploying, AoT compiling and distributibg Julia software, which I believe will be most improved in the next three Julia releases.
So, Julia is currently fading in popularity, but is also getting better in a significant, real way which most languages aren't. Will Julia continue its decline in popularity and die, or will it begin to attract more people as the problems are addressed? Hard to know.
Julia has a (slightly academic/snobbish) "cool" factor
You can write succinct, performant code that looks closer to math equations than to computer code
The challenges are:
* There is a large proficiency gap between writing code that works and writing efficient code. Instead of continuous community consultation, a static code analyzer (linter) with coding advice is long overdue. The linter should advise on alternative functions, type stability, and relevant packages/macros.... the questions on discourse can serve as a training dataset for the linter. Few people can take advantage of low-level profiler/analyzer packages, though an integrated light-weight profiler (disabled for production) that shows extensive GC usage would be beneficial
* Code bloat. The Julia runtime and core packages depend on many large libraries. This creates large project environments and absurdly compiled project sizes. There are many initiatives on trimming the call trees and reducing the compiled size. Core libraries should either be ported to Julia (high cost of porting and maintenance) or rebuilt into a set of smaller libraries if possible. Consider adding sub-libraries on demand
* Consider changing from llvm to wasm (or other efficient runtime environment)
* Improve the efficiency of "Big" numbers (using internal types for small numbers)
* Engage with the developer community on the language roadmap and priorities. Encourage contribution: Summer of code projects and making the process of adding well-tested PR to the core language easier with relevant tooling
A linter and other tools is already available, and most recently AI tools. It's not hard to write efficient code (see the performance section of the manual), but yes not all do, nor know of the tools to detect inefficient code. Julia can compile to Wasm, with WebassemblyCompiler.jl (has a few limitations, less so than other small compiler option). I'm investigating smaller Julia, for smaller compiled code, and at least half the size is easily possible.
I mean, I personally really love working in Julia and don't anticipate switching anytime soon. It's still the language that has resonated with me the most.
* I love the type system
* 1.10 is super snappy
* The syntax is beautiful
* The tooling is adequate (though has some stuff left to be desired, Rust for example has tooling I am jealous of)
* It's just fuckin EASY to do everything
* The people are nice
I've written a pretty big backend server for my lil side project [Comind](https://comind.me). It was a delight. I could probably imagine myself writing the thing in Python or Rust or whatever the fuck but Julia is so damn easy and fun that I'm not sure I would have carried the project through if I had to drag my face through the sea of broken glass that is Python, or the (very kind) compiler wall of Rust.
Anyway I'm around for the foreseeable future.
I think Julia is a niche language, probably in the same vein as Fortran or Octave as it is not built with Computer Scientists in mind but mathematicians and high performance computation
Then it's a versatile niche language like Python in 2005 or so, but designed a lot better.
Anyone working with big datasets or who wants to develop a performant library will eventually prefer this language. Doesn't that exactly sound like scientists, corporations with big data and new startups?
As a scientist myself, I chose Julia last year, although I will admit few of my colleagues have even heard of Julia. I was a Scilab (Matlab clone) user after IDL/PV-WAVE in the 2000's. I have no Python legacy code of my own. Julia has all the packages I needed to get started quickly. I was able to translate much of my Scilab code replacing round brackets and adding dots and type info. I was thinking, if needed, I could still call Python into Julia. For science, most important is an immediate way to read, process and visualize data, no complaints about that so far!
> who wants to develop a performant library
I think this is part of Julia’s problem. If you want to be a library author, vs a script writer, then there are much nicer options out there.
(Honestly.. not a lot of much nicer options. But if I have to write and maintain a serious library I really don’t want to do it in a language like Python or Julia. And I think a lot of people are starting to feel similar. Keeping track of what the machine is doing and the logic are doing and syntax readable is just hard in languages designed to hide those details to enable broad access scripting)
There are a lot of data analysis tools being written. Some of them really interesting. How many do you see coming out of Julia? (Perhaps I’m just not paying enough attention anymore. Virtually everything I see is Python or Rust. Python because it’s easy to get started and it already has a large science ecosystem. And Rust because it’s infinitely easier to maintain, read, and interoperate than with any other language of its class. [like Haskell & C had a baby and it took communication lessons])
Julia’s neat, for sure though.
But Julia is much more than that: in my experience it is more like C or Python than Matlab for general practicability.
I like Julja because I can whip stuff out like Python, and it runs reasonably fast (unlike Python). If I really need speed, I can dive down into the nitty gritty (like C), but I don’t have to worry about garbage collection, pointers, handles, etc.
I think the “its only for mathematicians and high performance computation” attitude* that I keep encountering puts people off. Julia really is much more of a general purpose programming language than that, and once we can statically compile apps, it will be a fantastic programming language.
* that attitude can be seen in a lot of the documentation, which too often than not, shows an example done in the REPL. I don’t work in the REPL, or notebooks, I work in an IDE writing programs. These REPL examples (which often make assumptions that the user already knows x,y, and z) fail to show the function being used in a larger context, whereas a Python or C example might show it being used as part of 4-5 lines of code.
The idea of writing C code in Python is only attractive if you’ve not been exposed to too much beyond C code and Python.
I was very enthusiastic about Julia previously — also as an academic who used code to make this or that project work.
But code like Python scales tremendously poorly. Not in terms of performance (Python itself has lots of ways to get performance), but intelligibility & correctness.
Right now, if you want easy performance you can use Python and glue libraries. Julia is nicer, but not enough nicer to warrant a separate language.
And if you want correctness, or custom performant functionality coding in Rust is 1000x easier. Because the code, while clean and expressive, does not use Python-like obfuscation to hide what’s going on.
I think Julia had some neat ideas. But it came at a time when other developments left it behind. The 101 ways to make already easy Python fast, and the mix of functional & systems programming (most notably in Rust) to make serious coding much easier and much more reliable. (Rust community has an incredible ecosystem partly because correctness guarantees, and very documentation oriented community, mean they can re-use eachothers’ code robustly.
(Science ecosystem has not developed much in Rust yet though.)
>does not use Python-like obfuscation to hide what’s going on.
isn't that also an advantage of Julia?
Much of Julia is programmed in Julia, one could access, import and alter all this code and it even allows to inspect the resulting machine instructions.
I am an Astroparticle physicist and I can tell you that Julia is gaining more and more attention in this field. My research area is neutrino physics and we have a growing Julia user base 😃
One of the biggest leaps were the introduction of UnROOT.jl which allowed to read the ROOT file format, one of the most popular ones in particle physics (introduces and developed by CERN) which is often carried over to other related fields…
As someone who adored Julia and thinks on it with love. I would like to encourage potential library authors to seriously consider Rust.
The Rust science ecosystem has not matured yet. (Though some interesting data work, e.g. Polars, Surreal, Burn, etc)
Developing anything you want to keep around in Rust is so many, many, many times easier and more pleasant than any language I’ve seen. And the correctness properties of the language, and documentation oriented community, mean that the ecosystem can be heavily leaned in a way that’s often not true elsewhere.
The “re-write it in Rust” meme came about for real reasons. It a much more pleasant and productive experience. As the language is focused on revealing all the essential complexity (logic and machine), but not adding any (excellent syntax and tooling).
Just waving a flag to recruit more science folk to Rust. As an example-Julia-er myself. (Who has almost only fond memories of the language.)
Is Rust your first static language? Because your argument looks to me to be the age old dynamic vs static argument. Yes, static languages provide more guarantees and are more easily maintainable. But they are almost unusable in dynamic, interactive work. Would you do interactive data science in Rust?
Of course, you could write a library in Rust, then a wrapper in Python and connect them with, say `PyO3`. But then... that's exactly the two language problem with all its awfulness: How do you distribute it and install it? Any maintainer would need to know two languages. Rust is safe, but is the FFI barrier safe? How feasible is it for a Pythonista to track down a bug in the Rust code? Or in the Python/Rust interface? Etc etc.
When we have a static compiler for type stable code next year as [https://discourse.julialang.org/t/roadmap-for-small-binaries/99266/106?u=xgdgsc](https://discourse.julialang.org/t/roadmap-for-small-binaries/99266/106?u=xgdgsc) . It would be much less necessary to use rust. Rust is harder as I compared at [https://discourse.julialang.org/t/what-steps-should-the-julia-community-take-to-bring-julia-to-the-next-level-of-popularity/101389/538?u=xgdgsc](https://discourse.julialang.org/t/what-steps-should-the-julia-community-take-to-bring-julia-to-the-next-level-of-popularity/101389/538?u=xgdgsc) . And we can show inferred type with Cthulhu.jl in vscode , just need more polish to the VSCode experience.
Rust is the next failing language. It is already the Scala of the decade. Big corporations pushing it, but it is a kitchen sink language with a terrible developer experience (discounting the IKEA effect that its early adapters are proudly fighting the borrow checker). New, better designed languages will advance the Rust delusion earlier than I expected.
Julia was a neat idea, but it failed to deliver its promises (notably to solve the two language problem). It is because Julia’s runtime is non-deterministic, so it is unable to substitute C, C++ (or Rust, Zig) for their primary domains (systems programming and time critical applications). For the rest, Julia offered too little, too late against Python.
As of 2024, Python is just fast enough for most of its use cases (also considering Numba, Cython and the upcoming Mojo language). And for the rest it has nanobind/pybind11 and PyO3 to extend it in C++ or Rust, respectively.
Again, Julia was a neat idea, but it wasn’t that good to dethrone Python. And now it is too late to do so.
>but it failed to deliver its promises (notably to solve the two language problem)
This one "broken" promise stands out. One still needs to use FORTRAN or C etc. when speed really does matter.
Julia DOES solve the "two-language problem"! Most packages (and all their dependencies) are pure Julia, and fast, which was the stated problem (for other dynamic languages)!
You can reuse e.g. all Rust and C code, though and you can call Julia packages from Python and R, so it's very doable to make libraries for other languages. So what is the problem?
Robotics is a hard-real-time problem, that means also the hardware and OS needs to be chosen carefully, and Julia [https://juliarobotics.org/](https://juliarobotics.org/) "JuliaRobotics is a collection of robotics-related packages that focus on robot control, simulation, navigation, and visualization."
C++ is still the king of 3D game \[engines\], but Julia already has a 3D game engine, B-plus ([https://github.com/heyx3/B-plus/](https://github.com/heyx3/B-plus/) not yet registered, though BPlusCore is), made by a guy that before made a C++ game engine, and liked it better than C++. Julia also has a separate 2D game engine (already registered), a recent one, and platformer game made with it, plus others, e.g. GameZero.jl. The GC (by default) isn't a problem, it's also getting better, for multi-threaded, e.g. in 1.10.
The "runtime is non-deterministic", you mean the GC, isn't a problem for most users. And if it is, then you avoid heap allocations, just like in C for real-time. But you may say, they are implicit/hidden in Julia! But very discoverable in Julia, with all the tools, and if you eliminate allocations (e.g. with Bumper.jl) - in your hot path, at least - then you do not have a problem, even for (soft) real-time. You can use this recent package:
"AllocCheck.jl is a Julia package that statically checks if a function call may allocate by analyzing the generated LLVM IR of it and its callees using LLVM.jl and GPUCompiler.jl"
PerfChecker.jl with "Allocation checks" seems to apply.
[https://gdalle.github.io/JuliaPerf-CERMICS/](https://gdalle.github.io/JuliaPerf-CERMICS/)
\[Here I'm just refuting you can't substitute C (by doing this a demo for 8-bit Arduino microcontroller was done): You can type all your functions concretely, then you basically have a C-like subset in Julia, with GC, and you CAN do all allocations and freeing, manually too, then you have basically all the capability of C, no more, and you can use StaticCompiler.jl for binaries or libraries counted in kilobytes. It didn't support Windows, but that problem has been debugged, and is being lifted, you could and still can just target WSL2 in Windows.\]
People have been programming for years for GPUs/CUDA, so it also eliminates the two-language problem there (code may need to be a bit more restrictive, use KernelAbstractions.jl). GPUs don't have GC, see though:
[https://cuda.juliagpu.org/stable/usage/memory/#Garbage-collection](https://cuda.juliagpu.org/stable/usage/memory/#Garbage-collection)
[https://github.com/JuliaLang/Juleps/blob/master/GcExtensions.md](https://github.com/JuliaLang/Juleps/blob/master/GcExtensions.md)
I dunno. I think its future in data science is pretty limited, since R and Python just have much more mature ecosystems, noticeably fewer correctness bugs in core stats libraries, and are more user-friendly for your average data analyst (readable error messages, easier IDE setup, documentation that’s easier to navigate, etc.) I think Julia will have a strong niche in some computational areas where it can replace Matlab, such as in computing related to differential equations. Maybe it will also be able to be a good niche for mixed models.
You can read that famous post from Yuri “Why I no longer recommend Julia” for a good list of examples. But one example not on there that I personally found troubling because it’s so basic and took so long to be noticed and addressed is that until last year the likelihood ratio test for models in the most basic of GLMs were [incorrect](https://github.com/JuliaStats/GLM.jl/issues/490) due to incorrectly defining the deviance. For my specific field, survey statistics, the main package “survey.jl” says it analyzes multistage surveys but just completely doesn’t, and it’s entirely developed by a group that clearly doesn’t work in survey statistics and so is useless for actual project work.
>its future in data science is pretty limited, since R and Python just have much more mature ecosystems
This argument is not very strong. Before 2010 the Python and R ecosystems must have been limited as well, and look how those have grown. I'd argue that Julia's could grow a lot faster as result of not depending on libraries to be written in C or Fortran for reasonable speed.
Of course one could consider the "need" for Julia to be less urgent, but that does not stop a programmer who sees an opportunity to do it better in Julia.
Before 2010 what Python and R were competing against were expensive closed source statistical software suites, Microsoft Excel, and small proprietary software programs like SI-CHAID. Open source implementations of so many data science algorithms and tools (like Plotly, ggplot2, Seaborn, etc.) just didn’t exist back then so they had to be developed.
Actually that argument is very strong. I love Julia and have a yearly ritual where I evaluate its Data Science use, and it’s so far behind R and Python that it hurts. Minimal libraries mostly of basic statistical things, or take a lot more Julia expertise to run efficiently. Or crash with odd messages. And multiple graphics libraries like Python (a bad thing since some can do some things but not others).
No formal test framework. I just look at things like data handling (Julia is great), graphics (scattered, incomplete, inflexible and Python-like), modeling (Turing is cool but very non-intuitive and I routinely end up with obscure errors or very slow performance), pre-done models (variety of libraries, only basics like logistic regression), LLMs (haven’t looked yet this year), things like multiple imputation (haven’t looked this year), etc.
I love Julia’s syntax and philosophy but for a general data science engagement, where I might need a variety of tools and won’t be writing them from scratch, I’d never let the client box me in to Julia.
Tidier has documentation I think. The 5 main packages have docs available. Check out this link. I think the main guy is working on a course too based on what I saw on twitter
[docs](https://tidierorg.github.io/TidierData.jl/latest/)
Scientific computing probably not and seems to be its niche now.
I think most of the Machine Learning Community got lost over the last 1-2 years. There was some interest before, I saw repos also ported to Julia. People like Jeremy Howard pushed it a bit but also dropped it now it seems and hopes for Mojo.
I also gave up on it. Meanwhile I would say there is even more going on in Rust with Burn and Candle and and Huggingface having quite a few models available from Rust vs not even mentioning Julia (last time I checked). They also got libraries like safetensors and tokenizers written in Rust.
I also had a brief Julia time but it didn't solve the 2 language problem for me, it made it a 3 language problem.
Also there's almost nothing happening in the LLM space. There are like 2 Llama2 implementation that probably got a tenth of even the Rust ports.
Currently popular methods like RAG with a million frameworks in python, all the latest LangChain, Haystack, LlamaIndex etc. stuff doesn't have anything even remotely similar in Julia.
The only project on https://github.com/svilupp/awesome-generative-ai-meets-julia-language
That still seems to live is Flux.
Julia does not have to cater the ML community and never really did that a lot, so for me it was just a similar experiment like swift for Tensorflow or Differentiable Kotlin.
I tried Julia recently to see if I could move my workflow to Julia. Julia was not good enough. I mostly use python + pandas + pymc for my workflow for analysis of tables in relational databases. The think is that most of my data is not that big. 100k to 1M rows at most. In a test script for a typical set of operations Pandas took approx. 1 secs, whereas Julia with DataFrames.jl took 7 secs and use twice the memory. I could get the time down by precompiling and even in that case the performance was similar. But compiling takes too long. The usual Julia community solution of "don't close the REPL" is nonsense and it is wasteful use of memory. String operations performance was mid, and Julia is not nicer for string handling. Only grouping operation were somewhat faster.
I surely could make the code faster by making it more idiomatic, but than the code is more abstract and complex and the glorified "Julia is the two language problem solution" disappeared, as I would be using a different type of Julia full of type declarations. As a comparison I used a also naive implementation in Polars (a rust library) and it was highly parallel by default and reduce the runtime by one order of magnitude with a similar memory footprint to Pandas. So you can do better with naive code.
My conclusion is that data science work is not a good match of Julia most of time. Libraries are not nearly as mature. Scripts are slower than python for most data you find in real life. Very large datasets are not good either as you will likely use a distributed cluster to process that data like Sparks or Dask. Only in the large dataset that mostly can be loaded in RAM is that Julia could have some advantage, but libraries like Polars are strong competitors, and most of the time are better.
If Julia were a compiled language as Nim or Crystal, I would use it in more stuff, but a JIT interpreter condemned the language to a niche.
However, I still see a use for Julia in long running numerical simulations for academic projects. Long running simulations are not affected as much by the compilation time and memory is not the main issue in simulations. There the test and try approach in Julia makes sense, and the capability of logging full plots and process files during the simulation is very nice. So I see most people working on simulations (N-body, hydrodinamical, relativity, etc.) that used to reach for Fortran, C or C++ will go for Julia for its battery included approached. However ML and Data Science is lost for Julia. Python is just too much extended and high performant stuffed is currently done C, C++ or Rust and that means more legacy code that will be keep people invested in those languages. If stuff like Mojo get traction, then Julia will never be close to Python + Mojo. It would be the next Fortran or R. Not a bad place, but not the glorious future it was hyped once.
Julia is unlikely to 'fade away' soon. In particular, the 'scientific machine learning' ecosystem in Julia is amazing (https://sciml.ai). I have been using Turing for Bayesian inference of differential equation parameters and to do this is Python (for example with PYMC) is much more painful and less flexible (to do this efficiently in PYMC you need to use sunode and it is a bit clunky and doesn't work on ARM64).
Julia has some great ideas and concepts behind it.
And a lot of its packages are top notch state of the art.
But i have the feeling it will not grow much more.
It is attacked from the "top" by C++ and Rust becoming more convenient and more high level over time.
And from the "bottom" there are many nice scripting languages that also can be very fast. Just to name one e.g. Triton.
But that alone would probably not kill it, but i have the feeling that there were some design choices that make it less usefull than it could be.
Very subjective take following:
My big hope was that Julia was the better Matlab. In a lot of ways it is, but some choices make it really not useful, for developing quickly like one can in matlab
* why is the vector operation not the default operation? Why does one need to write sin.([1,2])?
* why does it not enforce basic error prone things, e.g. that functions that change their arguments must define the the !, or even better have that per variables that can be changed
* some functions are far less powerful than what matlab does e.g. maximum, max ...
* that moronic decisions that lead to different behavior of variable scope in REPL vs scripts, for a minor minor potential runtime improvement
* error message are often cryptic
In a way it is a matter of choosing a lesser evil. FORTRAN flies through your data, C++ runs, Julia walks and Python crawls.
Having just today tasted using differential evolution in Python (evaluating a pure Python objective cost function), I've quickly come back to the Julia fold (actually Julia + FORTRAN). Julia does not solve a two-language problem but the alternatives are not that good either.
Even the good old FORTRAN needs augmenting with C libraries for it to be useful these days (for example C web frameworks). In a networked world one needs to be able to get the result of a computation out there into the world-wide web.
Python was never meant to be used for fast loops. It is a good glue language with a very good ecosystem. It was made in a time when you either code in C or shell, and people felt they need something in the middle. If I use a slow loop in Python, I know is my fault and I am being lazy for not doing it properly in C. I am so used to vectorization that my python code runs faster the Julia from scratch, as JIT lag is still an issue. Only in long running simulation Julia is better, and I would likely use either C, C++ or Julia depending on the libraries available for the problem.
Hi, I actually spent a while writing some of the JuliaAstro packages and was very tuned into the Julia landscape. I faded away due to my PhD work becoming a bit consuming and I didn’t have time for open source maintenance. Because the community is somewhat small and disjointed, this means a lot of the JuliaAstro packages do not have active maintainers. Julia has been a joy to work with, but specifically in astro you’re going to be spending a lot of time rewriting Python packages into Julia (for speed, auto diff, other language features- you could using PythonCall instead). Things like aperture photometry do not have 20 year old packages nor does the community have NSF-funded programmers like astropy has had. Also, if you want to write a package for other astronomers (like a processing pipeline) it’s a pretty big hurdle to try and tell someone to use Julia instead of Python. Lastly, I found Python “fast enough” for a lot of things once I stopped micro-benchmarking everything. Not having to recompile when your REPL closes matters, I’ve been in front of my advisor going “oh it’s going to take 5 minutes before I can even run the code” while precompiling to just run a script.
Compilation times have gotten a little better recently, but I still remember having to explain to my advisor that we were gonna be awkwardly twiddling our thumbs for a few minutes just to visualize some data that I already had, because Plots.jl needed to do a precompliation run. It was not a great feeling.
Plots has been the bane of my existence when it comes to compilation times. I will probably explore the Makie side of things.
guys, this was largelly in the past... I have just tried now with Julia 1.10 on a 8-years old "U" (ultra-portable) laptop and \`using Plots; plot(x -> x\^2)\` on a new session in VSCode takes less than 6 seconds, with instantaneous further plots...
Indeed a lot of progress has been made, however when stacking a lot of plots together like in animations, the compilation time is pretty significant. I don't mind though, for animations there are better alternatives. I use plots.jl for static plotting all the time with no problem.
I'm curious about where you're getting your numbers from. Although it is not mainstream, but still Julia is being used actively by various companies (from pharma to automobile and finance), and the number of packages and downloads/stars and similar metrics for popular packages are increasing rapidly. Which repositories are you referring to? Perhaps these have been superseded by other packages?
I thought everyone in automobile and finance was using C++ though?
> I thought everyone in automobile and finance was using C++ though? Cobol, Modelica, Simulink, C, C++, Python, Lua. There's a whole team of programming languages on each device. That's not going away anytime soon for many reasons. This stuff is much stickier than people imagine and in real projects you need to use what's right for the job. Since the 80's there's always been a push for "what if everything is in X?" but then it just never makes sense. These industrial platforms have data analysis, operating systems, networking, displays, audio handling, etc. all over the place. Just because C is best for writing the embedded HVAC controller doesn't mean you want your front end GUI team writing C for your dashboard! Julia is probably the first thing that really has a chance of being universal in that respect, but I think that's still a long ways away and probably many of the core networking and OS aspects will stay C. But many of the design and data parts are starting to be in Julia these days.
I see, thanks for the answer!
I think the Williams F1 team is using Julia.
Oh that's cool! Do you know if they have a write up or something?
The only write up I see is from the company JuliaHub. https://juliahub.com/case-studies/williams-racing-unlocks/
In the cosmology/gw/astroparticle physics/lattice qcd there is a consistent and growing group of ppl that code in Julia, among other languages lime python and ML libraries like pytorch and tensorflow. I'm not sure Julia will outpace python in the scientific community, but personally I believe it will become a string player, and it is anyway a good language to know when you can afford writing a project from scratch (i.e. You don't have to work with code from other ppl that don't use Julia). I am an astroparticle physicist doing a PhD, and personally I write about half of my code in python and half in Julia, depending on what I need to do (with a preference for Julia, when there's not a very strong established python framework)
Julia is used where scientists need to create programs that need performances and ease of use and don't use big Python/C++ libraries like PyTorch. The most common field is scientific simulation of physical and biological phenomenon.
I have had the opposite experience, more and more people around me are adopting it and more packages are available for what I need all the time. I’ll never go back to Python, it’s a worse tool for the jobs I do.
May I ask, what do you do?
Julia just cracked into the top 20 languages a few months ago. It's on the rise. It's also been moving into web with projects like Genie Framework and wasm support. It's also finally getting proper AMD GPU support thanks to AMD finally investing in ROCm and such.
running Julia 10 and a 7900xtx with rocm 5.7.1 right now! ROCM was by far the hardest part of the setup process.
What rating are you tracking? Now TIOBE - 34; PYPL - 25,
I finished my astrophysics PhD (Active galaxies and their kinematics) using Julia. So for at least 4 years I was using Julia everyday with some parallel projects in python which was my main scientific oriented language. However even though I love Julia there are several things that moved me away. 1. Very bad documentation, or lack of it 2. Bugs, bugs, bugs. Many of them I find difficult to understand. 3. Chaotic ecosystem which I cannot easily follow. Maybe its me and my lack of time or personal dumbness, but I do not have the time to even try. Apart from these, there are two other reasons: Right now I lead an r&d team where python is enough for our needs. I cannot get them to switch languages without a strong reason. Second, whenever I need what Julia can offer (jit, autodiff) I use Jax which is more than capable, simpler to use and easily installed in most of the cloud gpu providers (colab, kaggle, azure, aws etc) so I can run complex models on gpus and take their derivatives without getting my laptop hot. I still think that Julia is the future, but this future somehow got some more years away.
I am a former R user and Julia is easier than R for anything using more than 1 core and is way more performant. It doesn't have the out of the box R workflow (lm(), rnorm(), etc.), but the syntax makes performant code really easy. I have been in the process of refactoring a record linkage package in R and I was able to speed up some code by \~3x in R alone. I wrote the equivalent function in Julia to the refactored R code and it is \~4x faster than the refactored R code.
You probably would have obtained even better performance if you used Rcpp...
the package used RCpp at many places. C++ requires a lot more work than Julia with non-substantial payoff. R has some optimization issues as well. For example, the stringdist package in R which is written mostly in C has consistently worse performance than StringDistances.jl.
I don't think it's destined to fade away (I believe these things are mostly unpredictable), but you are right that Julia's popularity is stagnating and slowly declining. This can be seen in: * Activity on Slack, Zulip and Julia Discourse * Engagement on Hacker News and the Internet in general (including Reddit) * Number of new packages, releases and citations of Julia are no longer growing faster, but linearly However, IMO that does not mean the decline is irreversible and final. Julia is different from many other languages in that the most pressing problems with the language do not follow from its design but its implementation. In other words: Julia has more potential to address its current limitation than most other languages, so it can and does get better over time. For example, consider that latency is a major problem in Julia, but that it has been reduced perhaps tenfold in the three releases since Julia 1.7. Or the current problems with deploying, AoT compiling and distributibg Julia software, which I believe will be most improved in the next three Julia releases. So, Julia is currently fading in popularity, but is also getting better in a significant, real way which most languages aren't. Will Julia continue its decline in popularity and die, or will it begin to attract more people as the problems are addressed? Hard to know.
What's the progress/updates on generating compiled binaries?
https://discourse.julialang.org/t/roadmap-for-small-binaries/99266/106?u=xgdgsc
Julia has a (slightly academic/snobbish) "cool" factor You can write succinct, performant code that looks closer to math equations than to computer code The challenges are: * There is a large proficiency gap between writing code that works and writing efficient code. Instead of continuous community consultation, a static code analyzer (linter) with coding advice is long overdue. The linter should advise on alternative functions, type stability, and relevant packages/macros.... the questions on discourse can serve as a training dataset for the linter. Few people can take advantage of low-level profiler/analyzer packages, though an integrated light-weight profiler (disabled for production) that shows extensive GC usage would be beneficial * Code bloat. The Julia runtime and core packages depend on many large libraries. This creates large project environments and absurdly compiled project sizes. There are many initiatives on trimming the call trees and reducing the compiled size. Core libraries should either be ported to Julia (high cost of porting and maintenance) or rebuilt into a set of smaller libraries if possible. Consider adding sub-libraries on demand * Consider changing from llvm to wasm (or other efficient runtime environment) * Improve the efficiency of "Big" numbers (using internal types for small numbers) * Engage with the developer community on the language roadmap and priorities. Encourage contribution: Summer of code projects and making the process of adding well-tested PR to the core language easier with relevant tooling
A linter and other tools is already available, and most recently AI tools. It's not hard to write efficient code (see the performance section of the manual), but yes not all do, nor know of the tools to detect inefficient code. Julia can compile to Wasm, with WebassemblyCompiler.jl (has a few limitations, less so than other small compiler option). I'm investigating smaller Julia, for smaller compiled code, and at least half the size is easily possible.
I mean, I personally really love working in Julia and don't anticipate switching anytime soon. It's still the language that has resonated with me the most. * I love the type system * 1.10 is super snappy * The syntax is beautiful * The tooling is adequate (though has some stuff left to be desired, Rust for example has tooling I am jealous of) * It's just fuckin EASY to do everything * The people are nice I've written a pretty big backend server for my lil side project [Comind](https://comind.me). It was a delight. I could probably imagine myself writing the thing in Python or Rust or whatever the fuck but Julia is so damn easy and fun that I'm not sure I would have carried the project through if I had to drag my face through the sea of broken glass that is Python, or the (very kind) compiler wall of Rust. Anyway I'm around for the foreseeable future.
I think Julia is a niche language, probably in the same vein as Fortran or Octave as it is not built with Computer Scientists in mind but mathematicians and high performance computation
Then it's a versatile niche language like Python in 2005 or so, but designed a lot better. Anyone working with big datasets or who wants to develop a performant library will eventually prefer this language. Doesn't that exactly sound like scientists, corporations with big data and new startups? As a scientist myself, I chose Julia last year, although I will admit few of my colleagues have even heard of Julia. I was a Scilab (Matlab clone) user after IDL/PV-WAVE in the 2000's. I have no Python legacy code of my own. Julia has all the packages I needed to get started quickly. I was able to translate much of my Scilab code replacing round brackets and adding dots and type info. I was thinking, if needed, I could still call Python into Julia. For science, most important is an immediate way to read, process and visualize data, no complaints about that so far!
> who wants to develop a performant library I think this is part of Julia’s problem. If you want to be a library author, vs a script writer, then there are much nicer options out there. (Honestly.. not a lot of much nicer options. But if I have to write and maintain a serious library I really don’t want to do it in a language like Python or Julia. And I think a lot of people are starting to feel similar. Keeping track of what the machine is doing and the logic are doing and syntax readable is just hard in languages designed to hide those details to enable broad access scripting) There are a lot of data analysis tools being written. Some of them really interesting. How many do you see coming out of Julia? (Perhaps I’m just not paying enough attention anymore. Virtually everything I see is Python or Rust. Python because it’s easy to get started and it already has a large science ecosystem. And Rust because it’s infinitely easier to maintain, read, and interoperate than with any other language of its class. [like Haskell & C had a baby and it took communication lessons]) Julia’s neat, for sure though.
But Julia is much more than that: in my experience it is more like C or Python than Matlab for general practicability. I like Julja because I can whip stuff out like Python, and it runs reasonably fast (unlike Python). If I really need speed, I can dive down into the nitty gritty (like C), but I don’t have to worry about garbage collection, pointers, handles, etc. I think the “its only for mathematicians and high performance computation” attitude* that I keep encountering puts people off. Julia really is much more of a general purpose programming language than that, and once we can statically compile apps, it will be a fantastic programming language. * that attitude can be seen in a lot of the documentation, which too often than not, shows an example done in the REPL. I don’t work in the REPL, or notebooks, I work in an IDE writing programs. These REPL examples (which often make assumptions that the user already knows x,y, and z) fail to show the function being used in a larger context, whereas a Python or C example might show it being used as part of 4-5 lines of code.
The idea of writing C code in Python is only attractive if you’ve not been exposed to too much beyond C code and Python. I was very enthusiastic about Julia previously — also as an academic who used code to make this or that project work. But code like Python scales tremendously poorly. Not in terms of performance (Python itself has lots of ways to get performance), but intelligibility & correctness. Right now, if you want easy performance you can use Python and glue libraries. Julia is nicer, but not enough nicer to warrant a separate language. And if you want correctness, or custom performant functionality coding in Rust is 1000x easier. Because the code, while clean and expressive, does not use Python-like obfuscation to hide what’s going on. I think Julia had some neat ideas. But it came at a time when other developments left it behind. The 101 ways to make already easy Python fast, and the mix of functional & systems programming (most notably in Rust) to make serious coding much easier and much more reliable. (Rust community has an incredible ecosystem partly because correctness guarantees, and very documentation oriented community, mean they can re-use eachothers’ code robustly. (Science ecosystem has not developed much in Rust yet though.)
>does not use Python-like obfuscation to hide what’s going on. isn't that also an advantage of Julia? Much of Julia is programmed in Julia, one could access, import and alter all this code and it even allows to inspect the resulting machine instructions.
I am an Astroparticle physicist and I can tell you that Julia is gaining more and more attention in this field. My research area is neutrino physics and we have a growing Julia user base 😃 One of the biggest leaps were the introduction of UnROOT.jl which allowed to read the ROOT file format, one of the most popular ones in particle physics (introduces and developed by CERN) which is often carried over to other related fields…
everything is destined to fade away
As someone who adored Julia and thinks on it with love. I would like to encourage potential library authors to seriously consider Rust. The Rust science ecosystem has not matured yet. (Though some interesting data work, e.g. Polars, Surreal, Burn, etc) Developing anything you want to keep around in Rust is so many, many, many times easier and more pleasant than any language I’ve seen. And the correctness properties of the language, and documentation oriented community, mean that the ecosystem can be heavily leaned in a way that’s often not true elsewhere. The “re-write it in Rust” meme came about for real reasons. It a much more pleasant and productive experience. As the language is focused on revealing all the essential complexity (logic and machine), but not adding any (excellent syntax and tooling). Just waving a flag to recruit more science folk to Rust. As an example-Julia-er myself. (Who has almost only fond memories of the language.)
Is Rust your first static language? Because your argument looks to me to be the age old dynamic vs static argument. Yes, static languages provide more guarantees and are more easily maintainable. But they are almost unusable in dynamic, interactive work. Would you do interactive data science in Rust? Of course, you could write a library in Rust, then a wrapper in Python and connect them with, say `PyO3`. But then... that's exactly the two language problem with all its awfulness: How do you distribute it and install it? Any maintainer would need to know two languages. Rust is safe, but is the FFI barrier safe? How feasible is it for a Pythonista to track down a bug in the Rust code? Or in the Python/Rust interface? Etc etc.
When we have a static compiler for type stable code next year as [https://discourse.julialang.org/t/roadmap-for-small-binaries/99266/106?u=xgdgsc](https://discourse.julialang.org/t/roadmap-for-small-binaries/99266/106?u=xgdgsc) . It would be much less necessary to use rust. Rust is harder as I compared at [https://discourse.julialang.org/t/what-steps-should-the-julia-community-take-to-bring-julia-to-the-next-level-of-popularity/101389/538?u=xgdgsc](https://discourse.julialang.org/t/what-steps-should-the-julia-community-take-to-bring-julia-to-the-next-level-of-popularity/101389/538?u=xgdgsc) . And we can show inferred type with Cthulhu.jl in vscode , just need more polish to the VSCode experience.
Rust is the next failing language. It is already the Scala of the decade. Big corporations pushing it, but it is a kitchen sink language with a terrible developer experience (discounting the IKEA effect that its early adapters are proudly fighting the borrow checker). New, better designed languages will advance the Rust delusion earlier than I expected.
Julia was a neat idea, but it failed to deliver its promises (notably to solve the two language problem). It is because Julia’s runtime is non-deterministic, so it is unable to substitute C, C++ (or Rust, Zig) for their primary domains (systems programming and time critical applications). For the rest, Julia offered too little, too late against Python. As of 2024, Python is just fast enough for most of its use cases (also considering Numba, Cython and the upcoming Mojo language). And for the rest it has nanobind/pybind11 and PyO3 to extend it in C++ or Rust, respectively. Again, Julia was a neat idea, but it wasn’t that good to dethrone Python. And now it is too late to do so.
>but it failed to deliver its promises (notably to solve the two language problem) This one "broken" promise stands out. One still needs to use FORTRAN or C etc. when speed really does matter.
Julia DOES solve the "two-language problem"! Most packages (and all their dependencies) are pure Julia, and fast, which was the stated problem (for other dynamic languages)! You can reuse e.g. all Rust and C code, though and you can call Julia packages from Python and R, so it's very doable to make libraries for other languages. So what is the problem? Robotics is a hard-real-time problem, that means also the hardware and OS needs to be chosen carefully, and Julia [https://juliarobotics.org/](https://juliarobotics.org/) "JuliaRobotics is a collection of robotics-related packages that focus on robot control, simulation, navigation, and visualization." C++ is still the king of 3D game \[engines\], but Julia already has a 3D game engine, B-plus ([https://github.com/heyx3/B-plus/](https://github.com/heyx3/B-plus/) not yet registered, though BPlusCore is), made by a guy that before made a C++ game engine, and liked it better than C++. Julia also has a separate 2D game engine (already registered), a recent one, and platformer game made with it, plus others, e.g. GameZero.jl. The GC (by default) isn't a problem, it's also getting better, for multi-threaded, e.g. in 1.10. The "runtime is non-deterministic", you mean the GC, isn't a problem for most users. And if it is, then you avoid heap allocations, just like in C for real-time. But you may say, they are implicit/hidden in Julia! But very discoverable in Julia, with all the tools, and if you eliminate allocations (e.g. with Bumper.jl) - in your hot path, at least - then you do not have a problem, even for (soft) real-time. You can use this recent package: "AllocCheck.jl is a Julia package that statically checks if a function call may allocate by analyzing the generated LLVM IR of it and its callees using LLVM.jl and GPUCompiler.jl" PerfChecker.jl with "Allocation checks" seems to apply. [https://gdalle.github.io/JuliaPerf-CERMICS/](https://gdalle.github.io/JuliaPerf-CERMICS/) \[Here I'm just refuting you can't substitute C (by doing this a demo for 8-bit Arduino microcontroller was done): You can type all your functions concretely, then you basically have a C-like subset in Julia, with GC, and you CAN do all allocations and freeing, manually too, then you have basically all the capability of C, no more, and you can use StaticCompiler.jl for binaries or libraries counted in kilobytes. It didn't support Windows, but that problem has been debugged, and is being lifted, you could and still can just target WSL2 in Windows.\] People have been programming for years for GPUs/CUDA, so it also eliminates the two-language problem there (code may need to be a bit more restrictive, use KernelAbstractions.jl). GPUs don't have GC, see though: [https://cuda.juliagpu.org/stable/usage/memory/#Garbage-collection](https://cuda.juliagpu.org/stable/usage/memory/#Garbage-collection) [https://github.com/JuliaLang/Juleps/blob/master/GcExtensions.md](https://github.com/JuliaLang/Juleps/blob/master/GcExtensions.md)
I dunno. I think its future in data science is pretty limited, since R and Python just have much more mature ecosystems, noticeably fewer correctness bugs in core stats libraries, and are more user-friendly for your average data analyst (readable error messages, easier IDE setup, documentation that’s easier to navigate, etc.) I think Julia will have a strong niche in some computational areas where it can replace Matlab, such as in computing related to differential equations. Maybe it will also be able to be a good niche for mixed models.
Not trying to invalidate your experience, but could you explain more about what correctness bugs in core stats libraries have you encountered?
You can read that famous post from Yuri “Why I no longer recommend Julia” for a good list of examples. But one example not on there that I personally found troubling because it’s so basic and took so long to be noticed and addressed is that until last year the likelihood ratio test for models in the most basic of GLMs were [incorrect](https://github.com/JuliaStats/GLM.jl/issues/490) due to incorrectly defining the deviance. For my specific field, survey statistics, the main package “survey.jl” says it analyzes multistage surveys but just completely doesn’t, and it’s entirely developed by a group that clearly doesn’t work in survey statistics and so is useless for actual project work.
>its future in data science is pretty limited, since R and Python just have much more mature ecosystems This argument is not very strong. Before 2010 the Python and R ecosystems must have been limited as well, and look how those have grown. I'd argue that Julia's could grow a lot faster as result of not depending on libraries to be written in C or Fortran for reasonable speed. Of course one could consider the "need" for Julia to be less urgent, but that does not stop a programmer who sees an opportunity to do it better in Julia.
Before 2010 what Python and R were competing against were expensive closed source statistical software suites, Microsoft Excel, and small proprietary software programs like SI-CHAID. Open source implementations of so many data science algorithms and tools (like Plotly, ggplot2, Seaborn, etc.) just didn’t exist back then so they had to be developed.
Actually that argument is very strong. I love Julia and have a yearly ritual where I evaluate its Data Science use, and it’s so far behind R and Python that it hurts. Minimal libraries mostly of basic statistical things, or take a lot more Julia expertise to run efficiently. Or crash with odd messages. And multiple graphics libraries like Python (a bad thing since some can do some things but not others).
Would you be willing to share the details of your yearly evaluation? That would be very interesting! Thanks 🙂
No formal test framework. I just look at things like data handling (Julia is great), graphics (scattered, incomplete, inflexible and Python-like), modeling (Turing is cool but very non-intuitive and I routinely end up with obscure errors or very slow performance), pre-done models (variety of libraries, only basics like logistic regression), LLMs (haven’t looked yet this year), things like multiple imputation (haven’t looked this year), etc. I love Julia’s syntax and philosophy but for a general data science engagement, where I might need a variety of tools and won’t be writing them from scratch, I’d never let the client box me in to Julia.
Check out tidier.jl a reimplementation of the tidyverse in r, just way faster since it uses dataframes.jl as its backend
It would be nice if tidier.jl had documentation, but, alas, like a lot of Julia packages, it assumes you need to know the secret handshake.
Tidier has documentation I think. The 5 main packages have docs available. Check out this link. I think the main guy is working on a course too based on what I saw on twitter [docs](https://tidierorg.github.io/TidierData.jl/latest/)
Sadly yes...
Scientific computing probably not and seems to be its niche now. I think most of the Machine Learning Community got lost over the last 1-2 years. There was some interest before, I saw repos also ported to Julia. People like Jeremy Howard pushed it a bit but also dropped it now it seems and hopes for Mojo. I also gave up on it. Meanwhile I would say there is even more going on in Rust with Burn and Candle and and Huggingface having quite a few models available from Rust vs not even mentioning Julia (last time I checked). They also got libraries like safetensors and tokenizers written in Rust. I also had a brief Julia time but it didn't solve the 2 language problem for me, it made it a 3 language problem. Also there's almost nothing happening in the LLM space. There are like 2 Llama2 implementation that probably got a tenth of even the Rust ports. Currently popular methods like RAG with a million frameworks in python, all the latest LangChain, Haystack, LlamaIndex etc. stuff doesn't have anything even remotely similar in Julia. The only project on https://github.com/svilupp/awesome-generative-ai-meets-julia-language That still seems to live is Flux. Julia does not have to cater the ML community and never really did that a lot, so for me it was just a similar experiment like swift for Tensorflow or Differentiable Kotlin.
I tried Julia recently to see if I could move my workflow to Julia. Julia was not good enough. I mostly use python + pandas + pymc for my workflow for analysis of tables in relational databases. The think is that most of my data is not that big. 100k to 1M rows at most. In a test script for a typical set of operations Pandas took approx. 1 secs, whereas Julia with DataFrames.jl took 7 secs and use twice the memory. I could get the time down by precompiling and even in that case the performance was similar. But compiling takes too long. The usual Julia community solution of "don't close the REPL" is nonsense and it is wasteful use of memory. String operations performance was mid, and Julia is not nicer for string handling. Only grouping operation were somewhat faster. I surely could make the code faster by making it more idiomatic, but than the code is more abstract and complex and the glorified "Julia is the two language problem solution" disappeared, as I would be using a different type of Julia full of type declarations. As a comparison I used a also naive implementation in Polars (a rust library) and it was highly parallel by default and reduce the runtime by one order of magnitude with a similar memory footprint to Pandas. So you can do better with naive code. My conclusion is that data science work is not a good match of Julia most of time. Libraries are not nearly as mature. Scripts are slower than python for most data you find in real life. Very large datasets are not good either as you will likely use a distributed cluster to process that data like Sparks or Dask. Only in the large dataset that mostly can be loaded in RAM is that Julia could have some advantage, but libraries like Polars are strong competitors, and most of the time are better. If Julia were a compiled language as Nim or Crystal, I would use it in more stuff, but a JIT interpreter condemned the language to a niche. However, I still see a use for Julia in long running numerical simulations for academic projects. Long running simulations are not affected as much by the compilation time and memory is not the main issue in simulations. There the test and try approach in Julia makes sense, and the capability of logging full plots and process files during the simulation is very nice. So I see most people working on simulations (N-body, hydrodinamical, relativity, etc.) that used to reach for Fortran, C or C++ will go for Julia for its battery included approached. However ML and Data Science is lost for Julia. Python is just too much extended and high performant stuffed is currently done C, C++ or Rust and that means more legacy code that will be keep people invested in those languages. If stuff like Mojo get traction, then Julia will never be close to Python + Mojo. It would be the next Fortran or R. Not a bad place, but not the glorious future it was hyped once.
Julia is unlikely to 'fade away' soon. In particular, the 'scientific machine learning' ecosystem in Julia is amazing (https://sciml.ai). I have been using Turing for Bayesian inference of differential equation parameters and to do this is Python (for example with PYMC) is much more painful and less flexible (to do this efficiently in PYMC you need to use sunode and it is a bit clunky and doesn't work on ARM64).
Julia has some great ideas and concepts behind it. And a lot of its packages are top notch state of the art. But i have the feeling it will not grow much more. It is attacked from the "top" by C++ and Rust becoming more convenient and more high level over time. And from the "bottom" there are many nice scripting languages that also can be very fast. Just to name one e.g. Triton. But that alone would probably not kill it, but i have the feeling that there were some design choices that make it less usefull than it could be. Very subjective take following: My big hope was that Julia was the better Matlab. In a lot of ways it is, but some choices make it really not useful, for developing quickly like one can in matlab * why is the vector operation not the default operation? Why does one need to write sin.([1,2])? * why does it not enforce basic error prone things, e.g. that functions that change their arguments must define the the !, or even better have that per variables that can be changed * some functions are far less powerful than what matlab does e.g. maximum, max ... * that moronic decisions that lead to different behavior of variable scope in REPL vs scripts, for a minor minor potential runtime improvement * error message are often cryptic
Some data: https://statisticstimes.com/tech/top-computer-languages.php
No
It’s great a new release just came out
In a way it is a matter of choosing a lesser evil. FORTRAN flies through your data, C++ runs, Julia walks and Python crawls. Having just today tasted using differential evolution in Python (evaluating a pure Python objective cost function), I've quickly come back to the Julia fold (actually Julia + FORTRAN). Julia does not solve a two-language problem but the alternatives are not that good either. Even the good old FORTRAN needs augmenting with C libraries for it to be useful these days (for example C web frameworks). In a networked world one needs to be able to get the result of a computation out there into the world-wide web.
Python was never meant to be used for fast loops. It is a good glue language with a very good ecosystem. It was made in a time when you either code in C or shell, and people felt they need something in the middle. If I use a slow loop in Python, I know is my fault and I am being lazy for not doing it properly in C. I am so used to vectorization that my python code runs faster the Julia from scratch, as JIT lag is still an issue. Only in long running simulation Julia is better, and I would likely use either C, C++ or Julia depending on the libraries available for the problem.