Looks good, afaict it worked. Though libjulia also depends on libstdc++, which so far you haven't done anything with here. There's also libz, libtinfo, libdl, librt, libpthread, libm, and libc, but hopefully those are more likely to always be present without having to deal with as many binary incompatibilities? Not sure.

I have a pretty restrictive system that I will eventually be hoping to get a make dist'ed Julia tarball running on. If I can get it running on that without those steps, I'll probably call it good until someone else complains. I think libgfortran is quite a bit more rare than any of the libraries you've listed above, so hopefully we'll be able to target a good percentage of the market without worrying about those.

Rarity isn't the only concern. It's not just whether the destination machine has a copy of the library, it's whether they have a copy of the library that is binary compatible with what was used on the build machine. I've had lots of headaches before from libstdc++ in this regard, the others should be okay but I know very little about libtinfo.

With libstdc++ and libc, I believe you just have to target an old enough ABI that the linker can find the right set of symbols. I assume the problems you ran into were the version 'GLIBCXX_X.Y.ZZ' not found errors that pop up so often in this line of business?

Yeah, that looks familiar in a bad way (don't remember the exact details though, last time I did this kind of thing was a while ago). Is "target[ting] an old enough ABI" something that we're currently doing in Julia and I just missed it? I know there are C++ standards body proposals floating around for introducing standard platform C++ ABI's that you could force via something like extern "ABI", but that's probably at least 5-10 years away from actually being standardized and available in common compilers.

I don't know of any way to target a specific ABI other than just installing on a release old enough that the target ABI is the latest. ;D

It'd be really really cool if there were a way, I just stack overflowed it.

@staticfloat I've built julia with llvm-svn from this PR and can confirm that 1) libraries are there 2) running on Centos 6.5 LiveCD no longer produces openblas error message.
Thanks,

@staticfloat For targeting older glibc ABI - I suppose you can install Centos 5 chroot on Ubuntu and build Julia inside it? I think this is the easiest solution. But you must build with newer tools called devtoolset, RedHat provides them for RHEL 5 too. Scientific Linux rebuilt that devtoolset so I use it on Centos 5.
I googled for other options but didn't find anything useful.

Why not merge?

@ivarne if you agree, I think this is a candidate for backporting.

@JuliaBackports

Why not just link libgcc, libquadmath and libgfortran statically on all platforms?

@staticfloat please backport anything you see fit. I have only the slightest clue about what this patch does, so no opinion from me.

Why not just link libgcc, libquadmath and libgfortran statically on all platforms?

Please don't. It's painfully difficult to actually do this, it can interfere with exception handling across shared library boundaries, you need a specially-compiled GCC for it to be possible on Linux, it increases duplication and binary sizes, etc etc. Matlab on Linux (and Mac too I think) has to bundle its own copies of libgcc, libstdc++, and libgfortran. I don't think Octave, SciPy, or R have Linux binary installers that aren't through package managers, but if they do I suspect they're bundling libgfortran etc rather than statically linking.

Crap, this broke packages with compiled Fortran dependencies: ERROR: could not load module /home/travis/.julia/v0.4/OptimizationServices/deps/usr/lib/libOS.so: /home/travis/julia/bin/../lib/julia/libgfortran.so.3: version GFORTRAN_1.4' not found (required by /home/travis/.julia/v0.4/OptimizationServices/deps/usr/lib/libcoinmumps.so.1)`

@staticfloat, are the buildbots using a modern version of gfortran? Bundling libgfortran like this has put us into Matlab-like territory of causing incompatibility between our binaries (which always have the bundled gcc libs dlopen'ed) and any libraries built by system compilers.

Sigh. It looks like the devtoolset distributed by CERN doesn't ship dynamic libraries of libgfortran, only static ones, so we've been getting the old and busted GCC 4.1 libgfortran all this time. I'm going to try the following to see what the best method is:

I'll spin up a CentOS 6 machine so that I can install devtoolset-3, see if that ships a dynamic copy of libgfortran, and see if I can frankenpaste it onto my CentOS 5 machines. If I can, I'll give the library to you and see if it "just works". Failing that, I'll try just plain building the julia .tar.gz on CentOS 6 and see if that works. If it does, we might have to take a poll of users that use old versions and build our generic binaries on CentOS 6 instead. Failing that, I'll see if I can figure out how to generate the dynamic libgfortran.so.3 file from libgfortran.a, but I'm not optimistic that this is going to be easy to do by hand.

Thanks for checking.

so we've been getting the old and busted GCC 4.1 libgfortran all this time

Oh, so even more like Matlab then, but not intentionally :-P

Failing that, I'll see if I can figure out how to generate the dynamic libgfortran.so.3 file from libgfortran.a, but I'm not optimistic that this is going to be easy to do by hand.

That will depend on whether or not it was compiled with -fPIC, among other complications. If the devtoolset's static libgfortran was compiled with -fPIC, then maybe we should actually be trying to statically link at least base Julia's dependencies against it. OTOH not supporting CentOS 5 any more is probably a simpler choice.

Bump. @staticfloat any additional info here?

I have confirmed devtoolset-3 still does not ship a newer libgfortran, and Scientific Linux 6 only ships with gcc 4.4, which doesn't have GFORTRAN_1.4 ABIs.

Thanks! But crap, so what do we do about this? Is the devtoolset capable of statically linking libgfortran into a shared library? This requires that it was compiled with -fPIC, I'm not sure of an easier way to find out than doing something like this: https://github.com/coin-or/Ipopt/blob/a3c4bd8971a300d2f5213caf9719b339ddb8ebfd/Ipopt/contrib/MatlabInterface/MatlabInterface.site#L63-L94

Or could the devtoolset people be convinced to ship a shared libgfortran? I wonder how often they release new versions... @nalimilan any suggestions?

I have no idea, except that AFAIK the devtoolset is a repackaged version of the one RedHat releases, and they will likely be very conservative about what they ship given that they offer long-term support to their customers.

-static-libgfortran and -static-libgcc would have been ideal, but that does not do the same for libquadmath (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46539).

Yeah, that flag doesn't work on Linux with GCC > 4.5 due to libquadmath. I don't think any of the libraries we use actually do any quad-precision operations though. So if we explicitly link against the full path of libgfortran.a via gfortran -print-file-name=libgfortran.a, it can be made to work. It depends on whether the static libgfortran was compiled with -fPIC. If it wasn't, we need to compile our own libgfortran. At that point shipping it vs statically linking it might not make much difference.

Fired up a docker container of CentOS 5 and tried out devtoolset-2 on it. There is a opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgfortran_nonshared.a that was compiled with -fPIC, and the libgfortran.so that they provide is actually a linker script to link against that in addition to the old lousy system libgfortran shared library. They do something similar for libstdc++, and I'm sure they have their reasons, but unfortunately it's unusable for us. I tried moving libgfortran_nonshared.a in place of libgfortran.a and adding -static-libgfortran to FC, which actually works on my tiny test example, but it does not contain _gfortran_st_write and several other required functions that would be needed to link openBLAS against it.

I have an idea that might allow us to fix this without needing to build our own GCC. @staticfloat can you try the following on the Centos 5 buildbot?

scl enable devtoolset-2 bash
gfortran -shared -Wl,--whole-archive /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgfortran_nonshared.a -Wl,--no-whole-archive -o libgfortran_nonshared.so
mv libgfortran_nonshared.so /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2

Then if you rebuild openspecfun, openblas, and arpack, the devtoolset linker script should result in a shared-library dependency on this libgfortran_nonshared.so, which we can then copy to usr/lib and the make dist tarball - probably only in the buildbot code, since this file won't exist for most people.

We'll need to test the resulting tarballs, but I think this might give us the newer libgfortran ABI's.

Unfortunately, I don't think this works quiiiiiite right:

$ nm libgfortran_nonshared.so | grep GFORTRAN
         U _gfortran_cpu_time_10@@GFORTRAN_1.0
         U _gfortran_flush_i4@@GFORTRAN_1.0
         U _gfortran_ftell@@GFORTRAN_1.0
         U _gfortran_os_error@@GFORTRAN_1.0
         U _gfortran_random_r10@@GFORTRAN_1.0
         U _gfortran_runtime_error@@GFORTRAN_1.0
         U _gfortran_size0@@GFORTRAN_1.0
         U _gfortran_stop_numeric@@GFORTRAN_1.0
         U _gfortran_transfer_array@@GFORTRAN_1.0
         U _gfortran_transfer_character@@GFORTRAN_1.0
         U _gfortran_transfer_character_wide@@GFORTRAN_1.1
         U _gfortran_transfer_complex@@GFORTRAN_1.0
         U _gfortran_transfer_integer@@GFORTRAN_1.0
         U _gfortran_transfer_logical@@GFORTRAN_1.0
         U _gfortran_transfer_real@@GFORTRAN_1.0
         U __iso_c_binding_c_f_pointer@@GFORTRAN_1.0

So we're still not getting @@GFORTRAN_1.4 like we want. Perhaps we can find a non-devtoolset version of gcc/gfortran we can install? I think the HipHopVM guys have one, I'll have to try it out. Ironic how we're trying to install the latest and greatest on the oldest and worst. ;)

I'd be surprised if development tooling for a PHP VM went to extra lengths to include Fortran support...

If we can't find something pre-packaged that will work, GCC isn't all that hard to compile ourselves. Just a bit time consuming.

BUMP! We need to fix this. Like, before our next release. Which should be in about a week.

I'm currently trying to build GCC manually. We'll see where this takes us.

Fantastic, thank you. Be sure to specify --enable-languages=c,c++,fortran since gfortran is non-default

Alright, I compiled GCC 4.8 on the buildbots, then built dist tarballs using that built GCC. I'm a little worried that this is going to stop these binaries from working on older machines, but we'll need to see if that is actually the case. If it is a problem, is shipping libstdc++.so and friends alongside Julia too much? I don't know.

32-bit tarball
64-bit tarball

403 forbidden

Whoopsies. Fixed.

Your intuition was right. In a docker container of centos 5:

[root@d2853f083d24 julia-820aabeec6]# bin/julia
bin/julia: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by /home/julia-820aabeec6/bin/../lib/julia/libjulia.so)
bin/julia: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /home/julia-820aabeec6/bin/../lib/julia/libjulia.so)
bin/julia: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /home/julia-820aabeec6/bin/../lib/julia/libjulia.so)

We're already bundling libgfortran, I don't think adding libstdc++ to that would be so bad. The alternative would be to try hacking things to link statically, which would require a different compilation of GCC using --with-pic. I would've said to do that earlier, but I think there can be potential performance implications of the binaries built that way. I'd rather ship a shared, default-configuration libstdc++ than try to statically link libstdc++ into every other shared library that we ship that needs it.

Yes, I had tried to generate a "static compiler", but ran into some issues. Those issues later turned out to be because I wasn't including <prefix>/lib64 on my 64-bit machine, so I think I could do that as well, but I'll try bundling libstdc++ first. Here's the file, if you plop that into <julia_prefix>/lib/julia, does it "just work"?

32-bit libstdc++
64-bit libstdc++

Have to save it to libstdc++.so.6, but Julia starts. Running the tests now. I get an odd warning but I don't know if it matters:

/sbin/ldconfig: Can't open cache file /etc/ld.so.cache
: No such file or directory

Oh right, the spawn test fails if you don't have perl installed, the only thing I had added from an empty centos:5 container was wget. Everything else from Base.runtests() looks fine on 64 bit. 32 bit is harder for me to test, better to use vm's/vagrant instead of docker for testing anything that isn't 64-bit linux.

I'll try using these binaries for the problematic package with Fortran dependencies on Travis and see how it looks.

Worked perfectly!!! Didn't need to worry about libstdc++ on Travis, Ubuntu 12.04 is apparently good enough. https://api.travis-ci.org/jobs/49236855/log.txt?deansi=true

So do we add libstdc++.so.6 to this script, switch the buildbots to use your new compiler, and call this fixed?

To demystify the libstdc++ thing, it looks like it's just tied directly to the version of GCC that it's shipped with. In other words, if ubuntu 12.04 ships with gcc 4.8 by default, it should "just work". If it ships with 4.7, then it should "just work" as long as the ABI of whatever functions we try to use hasn't changed between 4.7 and 4.8.

12.04 ships with 4.6. They don't break the ABI on every minor version, just some of them.

Furthermore, I believe the ABI is set on a per-function basis.