ARROW-5868: [Python] Correctly remove liblz4 shared libraries from manylinux2010 image so lz4 is statically linked #4828

wesm · 2019-07-09T03:01:41Z

I pushed the image to Docker Hub (it was on quay.io before). I'm not sure what's the best way to test for this -- I checked the produced wheels locally to verify that liblz4.so is no longer required

codecov-io · 2019-07-09T05:39:35Z

Codecov Report

Merging #4828 into master will decrease coverage by 22.25%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master    #4828       +/-   ##
===========================================
- Coverage   87.43%   65.18%   -22.26%     
===========================================
  Files         997      487      -510     
  Lines      139804    64296    -75508     
  Branches     1418        0     -1418     
===========================================
- Hits       122241    41910    -80331     
- Misses      17201    22386     +5185     
+ Partials      362        0      -362

Impacted Files	Coverage Δ
cpp/src/arrow/util/memory.h	`0% <0%> (-100%)`	⬇️
cpp/src/gandiva/date_utils.h	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/util/memory.cc	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/filesystem/util-internal.cc	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/util/sse-util.h	`0% <0%> (-100%)`	⬇️
cpp/src/gandiva/decimal_type_util.h	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/compute/logical_type.h	`0% <0%> (-100%)`	⬇️
cpp/src/parquet/hasher.h	`0% <0%> (-100%)`	⬇️
cpp/src/gandiva/basic_decimal_scalar.h	`0% <0%> (-100%)`	⬇️
cpp/src/arrow/compute/kernels/boolean.cc	`0% <0%> (-100%)`	⬇️
... and 748 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90affbd...1fb1acb. Read the comment docs.

pitrou · 2019-07-09T08:55:01Z

Hmm... auditwheel didn't complain? Perhaps worth reporting as a bug?

kszucs · 2019-07-09T10:59:05Z

We can add more docker images to test the produced wheels. Have you reproduced the issue?

kszucs · 2019-07-09T11:38:04Z

I cannot really find a linux distribution without lz4 preinstalled.

kszucs · 2019-07-09T13:00:24Z

So we statically link liblz4 in the manylinux1 wheels

# ldd pyarrow-manylinux1/libarrow.so.14 | grep z
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc28cef4000)

but dynamically in the manylinux2010 wheels

# ldd pyarrow-manylinux2010/libarrow.so.14 | grep z
        liblz4.so.1 => not found  (already deleted to reproduce the issue)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f56f7440000)

this what this PR resolves.

What I'm finding strange, that auditwheel seems to bundle libz for manylinux1:

# ls -lah pyarrow-manylinux1/*z*so.*
-rwxr-xr-x 1 root root 115K Jun 29 00:14 pyarrow-manylinux1/libz-7f57503f.so.1.2.11

while ldd still uses the system libz:

# ldd pyarrow-manylinux1/libarrow.so.14 | grep z
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f91fcf3f000)

For manylinux2010 we also have liblz4:

#  ls -lah pyarrow-manylinux2010/*z*so.*
-rwxr-xr-x 1 root root 191K Jun 28 23:38 pyarrow-manylinux2010/liblz4-8cb8bdde.so.1.8.3
-rwxr-xr-x 1 root root 115K Jun 28 23:38 pyarrow-manylinux2010/libz-c69b9943.so.1.2.11

and ldd similarly tries to load the system libs:

# ldd pyarrow-manylinux2010/libarrow.so.14 | grep z
        liblz4.so.1 => not found
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd72764e000)

Inspecting manylinux1 with LD_DEBUG=files,libs ldd libarrow.so.14 it seems like to search the right path, but cannot find the hashed version of libz libz-7f57503f.so.1.2.11

       463:     file=libz.so.1 [0];  needed by ./libarrow.so.14 [0]
       463:     find library=libz.so.1 [0]; searching
       463:      search path=/tmp/pyarrow-manylinux1/.          (RPATH from file ./libarrow.so.14)
       463:       trying file=/tmp/pyarrow-manylinux1/./libz.so.1
       463:      search cache=/etc/ld.so.cache
       463:       trying file=/lib/x86_64-linux-gnu/libz.so.1

There is no libz.so.1 just libz-7f57503f.so.1.2.11.

Similarly for manylinux2010 and libz:

       470:     file=libz.so.1 [0];  needed by ./libarrow.so.14 [0]
       470:     find library=libz.so.1 [0]; searching
       470:      search path=/tmp/pyarrow-manylinux2010/.               (RPATH from file ./libarrow.so.14)
       470:       trying file=/tmp/pyarrow-manylinux2010/./libz.so.1
       470:      search cache=/etc/ld.so.cache
       470:       trying file=/lib/x86_64-linux-gnu/libz.so.1

for liblz4 (again, I've deleted the system one):

       470:     file=liblz4.so.1 [0];  needed by ./libarrow.so.14 [0]
       470:     find library=liblz4.so.1 [0]; searching
       470:      search path=/tmp/pyarrow-manylinux2010/.               (RPATH from file ./libarrow.so.14)
       470:       trying file=/tmp/pyarrow-manylinux2010/./liblz4.so.1
       470:      search cache=/etc/ld.so.cache
       470:      search path=/lib/x86_64-linux-gnu/tls/x86_64:/lib/x86_64-linux-gnu/tls:/lib/x86_64-linux-gnu/x86_64:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/tls/x86_64:/usr/lib/x86_64-linux-gnu/tls:/usr/lib/x86_64-linux-gnu/x86_6$
:/usr/lib/x86_64-linux-gnu:/lib/tls/x86_64:/lib/tls:/lib/x86_64:/lib:/usr/lib/tls/x86_64:/usr/lib/tls:/usr/lib/x86_64:/usr/lib          (system search path)

There are no libz.so.1 nor liblz4.so.1, just libz-c69b9943.so.1.2.11 and liblz4-8cb8bdde.so.1.8.3

According to https://www.python.org/dev/peps/pep-0571/ liblz4 nor libz are part of the whitelist, and while these are bundled with the wheel, seemingly cannot be found - perhaps because of the hash in the library name?

I've tried to inspect the wheels with auditwheel show with version 2 and 1.10, both says the following:

# auditwheel show pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl

pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl is consistent with
the following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.3',
'GCC_3.4', 'GCC_3.0'}, libpthread.so.0 with versions {'GLIBC_2.3.3',
'GLIBC_2.12', 'GLIBC_2.2.5', 'GLIBC_2.3.2'}, libc.so.6 with versions
{'GLIBC_2.4', 'GLIBC_2.6', 'GLIBC_2.2.5', 'GLIBC_2.7', 'GLIBC_2.3.4',
'GLIBC_2.3.2', 'GLIBC_2.3'}, libstdc++.so.6 with versions
{'CXXABI_1.3', 'GLIBCXX_3.4.10', 'GLIBCXX_3.4.9', 'GLIBCXX_3.4.11',
'GLIBCXX_3.4.5', 'GLIBCXX_3.4', 'CXXABI_1.3.2', 'CXXABI_1.3.3'},
librt.so.1 with versions {'GLIBC_2.2.5'}, libm.so.6 with versions
{'GLIBC_2.2.5'}, libdl.so.2 with versions {'GLIBC_2.2.5'}, libz.so.1
with versions {'ZLIB_1.2.0'}

This constrains the platform tag to "manylinux2010_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

# auditwheel show pyarrow-0.14.0-cp37-cp37m-manylinux1_x86_64.whl

pyarrow-0.14.0-cp37-cp37m-manylinux1_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.4',
'GCC_3.0', 'GCC_3.3'}, libc.so.6 with versions {'GLIBC_2.3',
'GLIBC_2.2.5', 'GLIBC_2.3.4', 'GLIBC_2.4', 'GLIBC_2.3.2'},
libstdc++.so.6 with versions {'CXXABI_1.3', 'GLIBCXX_3.4.5',
'GLIBCXX_3.4'}, librt.so.1 with versions {'GLIBC_2.2.5'}, libm.so.6
with versions {'GLIBC_2.2.5'}, libpthread.so.0 with versions
{'GLIBC_2.3.3', 'GLIBC_2.3.2', 'GLIBC_2.2.5'}, libdl.so.2 with
versions {'GLIBC_2.2.5'}, libz.so.1 with versions {'ZLIB_1.2.0'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.24.so",
    "libcrypt.so.1": "/lib/x86_64-linux-gnu/libcrypt-2.24.so",
    "libdl.so.2": "/lib/x86_64-linux-gnu/libdl-2.24.so",
    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",
    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.24.so",
    "libnsl.so.1": "/lib/x86_64-linux-gnu/libnsl-2.24.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.24.so",
    "librt.so.1": "/lib/x86_64-linux-gnu/librt-2.24.so",
    "libstdc++.so.6": "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22",
    "libutil.so.1": "/lib/x86_64-linux-gnu/libutil-2.24.so",
    "libz.so.1": "/lib/x86_64-linux-gnu/libz.so.1.2.8"
}

In order to achieve the tag platform tag "manylinux2010_x86_64" the
following shared library dependencies will need to be eliminated:

libz.so.1

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libz.so.1

I think there are more todo left with the wheels. IMO the manylinux1 wheels are not compliant because of libz and the manylinux2010 wheels are not compliant because of both libz and liblz4 (but incorrectly reported by auditwheel?).

kszucs · 2019-07-09T13:13:53Z

Perhaps we should use setup.py::move_shared_libs for libz on linux too? cc @xhochy

wesm · 2019-07-09T14:21:30Z

Can you open a new JIRA about the libz issue with the information from this issue and we can investigate separately? I'm going to merge this for now.

+1

wesm · 2019-07-09T14:22:06Z

Do we know why auditwheel did not raise either the liblz4 or libz issue before?

kszucs · 2019-07-09T14:22:13Z

Sure, opening it.

kszucs · 2019-07-09T14:23:01Z

No idea, we only run auditwheel repair though.

wesm · 2019-07-09T14:23:23Z

I see. We should do something about that then

kszucs · 2019-07-09T14:28:55Z

In case of manylinux2010 the report seems faulty, but adding it to the issue.

xhochy · 2019-07-09T14:52:01Z

auditwheel repair should package libz.so into the wheel. Only after the repair it should be consistent.

…nylinux2010 image so lz4 is statically linked I pushed the image to Docker Hub (it was on quay.io before). I'm not sure what's the best way to test for this -- I checked the produced wheels locally to verify that liblz4.so is no longer required Author: Wes McKinney <wesm+git@apache.org> Closes #4828 from wesm/ARROW-5868 and squashes the following commits: 1fb1acb <Wes McKinney> Remove liblz4 shared libraries from /usr/local so static linking occurs

Remove liblz4 shared libraries from /usr/local so static linking occurs

1fb1acb

wesm closed this in 7838886 Jul 9, 2019

wesm deleted the ARROW-5868 branch July 9, 2019 14:23

asfimport mentioned this pull request Aug 1, 2019

[Python] manylinux2010 wheels have shared library dependency on liblz4 #16802

Closed

ARROW-5868: [Python] Correctly remove liblz4 shared libraries from manylinux2010 image so lz4 is statically linked #4828

ARROW-5868: [Python] Correctly remove liblz4 shared libraries from manylinux2010 image so lz4 is statically linked #4828

Conversation

wesm commented Jul 9, 2019

Uh oh!

codecov-io commented Jul 9, 2019

Codecov Report

Uh oh!

pitrou commented Jul 9, 2019

Uh oh!

kszucs commented Jul 9, 2019

Uh oh!

kszucs commented Jul 9, 2019

Uh oh!

kszucs commented Jul 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Jul 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wesm commented Jul 9, 2019

Uh oh!

wesm commented Jul 9, 2019

Uh oh!

kszucs commented Jul 9, 2019

Uh oh!

kszucs commented Jul 9, 2019

Uh oh!

wesm commented Jul 9, 2019

Uh oh!

kszucs commented Jul 9, 2019

Uh oh!

xhochy commented Jul 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kszucs commented Jul 9, 2019 •

edited

Loading

kszucs commented Jul 9, 2019 •

edited

Loading