Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

FWIW I believe that Anaconda may mitigate this problem by shipping their own gcc toolchain. @kalefranz or @asmeurer can you advise? Much appreciated

Although the gcc toolchain in defaults is gcc 4.8.5 (which means that gcc 4.9.x -- i.e C++11 -- stuff should work OK), it would be great to get gcc 4.9.x in conda-forge

Right, if you build against libstdc++ version X, and you run against version Y, then you have to have X <= Y or things might (and eventually will) break. When they break, you don't get a segfault or memory corruption, you get an immediate error complaining about missing symbols. So that's nice. But it's still not as nice as having things work :-).

There are a number of options for how to deal with this. You can make sure to build against an old version of libstdc++ (e.g. scipy wheels are built against the CentOS 5 one, but using the RH devtoolset toolchain to get c++11 support); then it's safe run against the system version. You can statically link against libstdc++. You can ship a newer version of libstdc++ vendored inside your package with a unique name (like how auditwheel vendors libraries), which is mostly equivalent to static linking. Or you can ship libstdc++ as a separate conda package. If you ship a libstdc++ conda package, and you don't rename the library to have a unique name, then you need to coordinate across the whole conda ecosystem to make sure everyone's using the same version, and there can be nasty edge cases if the system ever ends up with a newer version of libstdc++ than the one that you're shipping -- but OTOH it does make it easy to support newer toolchains.

I think currently (ana)conda is taking the last approach, so you need to figure out how to interoperate with that if you want to... it might be as simple as adding a depends: to your conda recipe. But I don't know the details and I could be out of date.

Ok, it seems there is some confusion about how we are building things. Let me please try to clarify. 😄

We are building on CentOS 6 with devtoolset-2.1 using a Docker container built from this Dockerfile. There are a few exceptions to this model, but they tend to involve things like Fortran or OpenMP. Boost is not one of those exceptions. This provides C++11 support, but that effectively lets someone use an old libstdc++/glibc version. It also saves us from having to ship libraries like libstdc++ around.

In the future, this model is likely to change. We will start using a packaged gcc on Mac and Linux. It will probably be version 5.2.1. When we do this, we will start shipping libraries. At some point, we will need to investigate gcc 6, but we will likely wait for Linux distro to clear most of the debris.

@jakirkham you said

This provides C++11 support, but that effectively lets someone use an old libstdc++/glibc version.

Is this actually true? I think this is only true if you have a sufficiently new libstdc++ on the target host (or you statically linkt the symbols). Correct me if I'm wrong

Yeah, devtoolsets are strange beasts. I wish I understood how they worked better.

The best I have been able to say is it is like partial static linking. I was unable to find the source for devtoolset-2.1, but found the source for devtoolset-4 some weeks back. Here is the source for devtoolset-4 on CentOS 7. We looked at this a couple of weeks back and sadly we are still mystified. If you know how this process works better, @njsmith I would be more than happy to learn.

Suffice it to say, if your concern is using the binary we build of Boost on CentOS 6, don't be. I regularly use this binary in my stack both in Docker containers and on a cluster that both use CentOS 6 without issues.

Also, resurfaced these slides from Internet Archive 👼. Basically, they say the same thing that this does partial static linking to get the newer symbols.

Yeah, basically the libstdc++ that comes with gcc 5.2, say, is like the one that comes with gcc 4.9, but with extra symbols added. (That's how libstdc++ keeps backwards compatibility -- they never change the underlying ABIs, just add new ones, and redirect source code to point to the new versions of the ABI at build time.) The devtoolset toolchains split out the new symbols that are in gcc v[whatever the devtoolset ships with], but not in gcc v[whatever is on the last supported version of RHEL/CentOS], and put just those into a static library that your code gets linked against.

Ok, so do those patches simply build more limited forms of the static libraries for libstdc++ and friends?

Actually, best person from Continuum to chime in here is @msarahan. He's working on an updated conda gcc 5.x or 6.x package (current 4.8.x was built by @asmeurer and has been solid for me personally on CentOS 5 & 6).

I see, then with devtoolset-2.1 we should be in good shape (even on ubuntu 12.04 onward, I guess). conda artifacts built with stock gcc or clang on Linux must take care, then to statically link libstdc++, but the conda-forge builds are fine.

I'm closing the pull request. I'm interested to keep learning more from you all, so thank you. We can revisit if it ever becomes a problem

@njsmith was accurate in saying what we do right now. In cooperation with conda-forge, we're about to undertake updating the compiler. We'll have a conda package for the compiler, and for runtimes (ideally split up). I am very much in favor of renaming libraries as auditwheel does to avoid shadowing conflicts, and will look into what that will take. Any pointers, guides, requests, comments, etc. welcome. I have created an issue at conda-forge/staged-recipes#872

Glad this was helpful for you, @wesm. Would appreciate your thoughts on the aforementioned issue. Particularly as static linking is very important for your workflow. While it is not a common workflow within conda-forge, IMHO we should strive to make packages that allow for people to effectively use static linking if they so chose to.

In general, to the extent that you can limit installing transitive dependencies that you only make light use of (for example, if you're only pulling a handful of symbols from boost), it makes using your library less onerous for end users.

I personally avoid the devtoolset as its (partial) static linking approach never worked properly for Julia. If you move quickly enough in adopting the latest gcc before your users do, then vendoring your own (shared library) libstdc++ has been working well.

Could you please clarify what about the devtoolset strategy didn't work for Julia, @tkelman? I'd be very curious to know specific use cases where it failed.

Sure, @wesm. Not questioning the value of static linking. Just wanting to make sure that our new compiler strategy (building a new gcc and distributing libraries from it) is amenable to your use case. To be explicit, will distributing a dynamic libstdc++ be ok for you in the future or will this cause problems?

From what I'm able to find (JuliaLang/julia#10043 (comment), JuliaLang/julia#8442, and JuliaLang/julia#8397) the problems with the devtoolset were more on the fortran / openblas side than with libstdc++. But once we found a workable solution for libgfortran to get around the devtoolset still creating shared libraries that depended on shared libgfortran (only for old symbols, but the library itself isn't always present on all user systems), we had to vendor libstdc++ as well. If you don't touch fortran or openblas at all (and none of your dependencies do either) then devtoolset is more likely to be usable for you.

FWIW, numpy and scipy are currently shipping wheels built with openblas and the last centos5 devtoolset, with vendored libgfortran and system libstdc++. Figuring out how to properly vendor libgfortran was non-trivial -- this is what prompted us to figure out all that arcane library renaming stuff I posted above -- but it seems to be working fine for us now. (I haven't read @tkelman's links in detail but a quick skim left me with the impression that the name collision issues might have been the underlying problem?)

(And when I say "non-trivial" I mean that AFAICT it was simply not possible to reliably vendor libgfortran until we fixed a bug in patchelf. Shared libraries! Fun for the whole family!)

We have had very few issues with vendoring libgfortran or libstdc++, even without renaming them - we just have to ensure we use a newer compiler version than our users do, which I don't consider to be much of a burden. We'd be open to experimenting with renaming our vendored libraries in Julia (we already use patchelf at build time, but requiring it at install time is a no-go given GPL), but haven't seen the need to go and implement it ourselves.

The problem wasn't entirely name collisions, it was also that the centos5 system shared libgfortran is too old and useless to actually give you anything you want if you try to vendor it. Perhaps renaming the devtoolset copy of libgfortran would also be a valid solution, but if that was broken until recently it wouldn't have done us much good in 2014.

Right, the point of renaming is that it avoids the issue where your version can collide with the system version. You're solving that by shipping a very new version of the library, and making sure that your library is always found first, and not supporting old julia builds on new systems. Renaming is another option for solving that problem that has different trade-offs.

The problem wasn't entirely name collisions, it was also that the centos5 system shared libgfortran is too old and useless to actually give you anything you want if you try to vendor it. Perhaps renaming the devtoolset copy of libgfortran would also be a valid solution, but if that was broken until recently it wouldn't have done us much good in 2014.

Not sure what you mean here... the whole point of the devtoolset compilers is that they don't have their own version of libgfortran, but instead use the stock centos5 (or centos6 or whatever) version. NumPy and SciPy seem to be doing fine with the centos5 shared libgfortran.

If you try to build libraries that need modern Fortran features that have had compiler fixes any time in the last 7-10 years, the stock centos version of libgfortran is deeply inadequate.

not supporting old julia builds on new systems

There's a really simple workaround for this one. Make sure the newer system versions of libstdc++ and libgfortran are installed when you want to use an old build, and just move the vendored copies out of the way.

If you try to build libraries that need modern Fortran features

@tkelman: Ah, right, scipy probably doesn't do that, since they've been stuck on fortran 77 until recently due to Windows horribleness. This is basically the same as the c++ problem, I guess -- if your code needs c++14 then centos5 is not going to work.

Right, not with the devtoolset anyway. The devtoolset has a newish gfortran compiler that can build things and tries to statically link the modern parts. But the shared part won't be useful for others who aren't using the exact same devtoolset setup.

Does the devtoolset also handle glibc differences? Basically I'm finding that libraries built with stock gcc 4.9.x on Ubuntu 14.04 or thereabouts are a non-starter unless you want to vendor glibc / manually put in LD_LIBRARY_PATH

Each devtoolset is built on the target OS. So, a devtoolset built for CentOS 6 has the same glibc guarantees as building with the stock system compiler.

I see, so if your goal is to support CentOS / RHEL 6, then the build machine should be that OS (and the binaries would be forward compatible).

Yes, unless you want to vendor a libc (gnu, musl, or otherwise)

Is it actually possible to vendor libc?

glibc makes it difficult to do so (I think linuxbrew might be doing that though?), it's a bit easier with musl but when you have multiple libc's around on the system then you can get into similar issues that you hit on Windows with passing crt objects around between libraries.

It is theoretically possible to vendor libc, but I don't know what you'd get from it, and I think you'd run into lots of headaches due to the mismatch between the system version and your version.

The simplest approach for glibc seems to be: make a list of the distros that you care about supporting, make a list of which glibc versions they use, and then build on whichever distro has the oldest glibc. If you build against glibc version X, then you can safely run against any glibc version Y >= X, no matter who ships it. (No distro will formally commit to cross-distro compatibility like this, but it works nonetheless.)

Distrowatch has a really useful database of which distro releases ship which versions of glibc. For example (scroll down a bit):

https://distrowatch.com/table.php?distribution=redhat
https://distrowatch.com/table.php?distribution=debian

I have been and remain in favor of creating a whole build chain for conda and/or conda-forge based on musl-libc.

http://www.etalabs.net/compare_libcs.html