Fix static linking with alwayslink#2584
Fix static linking with alwayslink#2584cstrahan wants to merge 6 commits intobazel-contrib:release-0.23from
Conversation
go/private/rules/cgo.bzl
Outdated
| "darwin": ("-Wl,-force_load", ""), | ||
| "js": ("", ""), | ||
| "linux": ("-Wl,--whole-archive", "-Wl,--no-whole-archive"), | ||
| "windows": ("-Wl,--whole-archive", "-Wl,--no-whole-archive"), |
There was a problem hiding this comment.
Why are these flags OS-specific? I think the assumption here is that clang is always used on darwin, and gcc is always used on linux and windows, but that's not a safe assumption.
Is there any way you can get the right flags out of cc_common from the toolchain? If not, I think it would be better to look at the name of the compiler or linker binary to guess which flags will be accepted.
Why is "js" included here by the way? What toolchain is expected there?
There was a problem hiding this comment.
Why are these flags OS-specific? I think the assumption here is that clang is always used on darwin, and gcc is always used on linux and windows, but that's not a safe assumption.
[...]
Why is "js" included here by the way? What toolchain is expected there?
I cribbed that from #1879.
Is there any way you can get the right flags out of
cc_commonfrom the toolchain? If not, I think it would be better to look at the name of the compiler or linker binary to guess which flags will be accepted.
If there is a way to do that, that would be great; I have no idea, though. There doesn't appear to be any sort of sort of sanctioned forum to ask the devs and others in the know, and I have no ties with Google or the Bazel devs, so I don't know how to inquire. Any advice here would be great.
There was a problem hiding this comment.
I don't see anything directly in cc_common that would help.
I think a "good enough" solution would be to check go.env["CC"], which will be the CC environment variable passed to actions. If the string after the last slash contains clang, let's use the clang flags. Otherwise, let's use the gcc flags. I'm not 100% sure that will always be set, so let's assume gcc if it's not.
(For asking questions by the way, the bazel-discuss mailing list is usually pretty good. There's also a public Slack, though I'm not sure if it's still active).
go/private/rules/cgo.bzl
Outdated
| inputs_direct.append(lib) | ||
| else: | ||
| if library_to_link.alwayslink: | ||
| lib_opts.append(_WHOLE_ARCHIVE_FLAGS_BY_GOOS[go.mode.goos][0]) |
There was a problem hiding this comment.
This will crash if go.mode.goos isn't in the dict above.
There was a problem hiding this comment.
That's a good point. I'd love to scrap _WHOLE_ARCHIVE_FLAGS_BY_GOOS and replace this code with something more robust, though I don't (yet) know how I would do that.
There was a problem hiding this comment.
Hmm, let's replace _WHOLE_ARCHIVE_FLAGS_BY_GOOS with a function:
def _library_args(go, library_to_link):
If alwayslink is false, it can just return a list with the file path. Otherwise, it can return a list with the file path surrounded by the appropriate flags. It can test go.env["CC"] to decide which flags to use.
go/private/rules/cgo.bzl
Outdated
| return libs | ||
|
|
||
| def _cc_libs(library_to_link): | ||
| libs = [] |
There was a problem hiding this comment.
Why return a list from this function instead of a single value?
There was a problem hiding this comment.
Why return a list from this function instead of a single value?
When I extracted that from the original source, I misread the code -- I thought I had read a set of four distinct if conditions (implying that library_to_link might have multiple files that need to be handled), not one chain of if ... elif. I'll change that.
go/private/rules/cgo.bzl
Outdated
| cc_transitive_headers = d[CcInfo].compilation_context.headers | ||
| inputs_transitive.append(cc_transitive_headers) | ||
| cc_libs = _cc_libs(d) | ||
| cc_libs = _cc_libs_for_target(d) |
There was a problem hiding this comment.
This list is now distinct from the libraries returned by _cc_libs, but the names are really similar. What's the distinction? Won't there be an error if there are differences?
There was a problem hiding this comment.
The old _cc_libs fished out the archive/dso path from each LibraryToLink in d, giving back a list of those paths. At that point, we no longer have LibraryToLink.alwayslink to inspect.
So, as the code stands now:
_cc_libs(library_to_link): gets the archive/dso path for the given library (currently as singleton list, but as you pointed out, this ought to just return the path itself)._cc_libs_for_target(target): returns the list of all lib paths for a given target (as_cc_libsused to)._cc_libraries_to_link(target): returns the list of allLibraryToLinks for a giventarget.
The names are terrible, I will admit. Ultimately, my changes were intended to address the fact I need the respective LibraryToLink so I can inspect alwayslink.
With that said, I noticed something odd about the code before I got involved (my notes as comments):
# cc_libs has the path of all libs for target `d`
cc_libs = _cc_libs(d)
# the existing implementation immediately extends inputs_direct and deps_direct with cc_libs
inputs_direct.extend(cc_libs)
deps_direct.extend(cc_libs)
# [... elided ...]
for lib in cc_libs:
if (lib.basename.startswith("lib") and
libname = lib.basename[len("lib"):lib.basename.rindex(".")]
clinkopts.extend(["-L", lib.dirname, "-l", libname])
inputs_direct.append(lib) # <===== isn't this superfluous, since we extended inputs_direct with all of cc_libs above???
elif (lib.basename.startswith("lib") and
has_versioned_shared_lib_extension(lib.basename)):
libname = ":%s" % lib.basename
clinkopts.extend(["-L", lib.dirname, "-l", libname])
inputs_direct.append(lib) # <===== ditto
else:
lib_opts.append(lib.path)Assuming I'm tracking how the code is intended to work, my next commit (which I will push shortly) will greatly clean things up.
|
My last commit (50acc1a) attempts to clean things up a bit, as I described earlier. How does that look? |
|
@jayconrod Any suggestions for how to move this along? I'd love to fix the remaining problems and get this fixed. |
|
@jayconrod I tried implementing your suggestions, as you can see in this diff: diff --git a/go/private/rules/cgo.bzl b/go/private/rules/cgo.bzl
index 863d7240..0407d216 100644
--- a/go/private/rules/cgo.bzl
+++ b/go/private/rules/cgo.bzl
@@ -31,13 +31,6 @@ load(
"cc_library",
)
-_WHOLE_ARCHIVE_FLAGS_BY_GOOS = {
- "darwin": ("-Wl,-force_load", ""),
- "js": ("", ""),
- "linux": ("-Wl,--whole-archive", "-Wl,--no-whole-archive"),
- "windows": ("-Wl,--whole-archive", "-Wl,--no-whole-archive"),
-}
-
def cgo_configure(go, srcs, cdeps, cppopts, copts, cxxopts, clinkopts):
"""cgo_configure returns the inputs and compile / link options
that are required to build a cgo archive.
@@ -147,11 +140,7 @@ def cgo_configure(go, srcs, cdeps, cppopts, copts, cxxopts, clinkopts):
libname = ":%s" % lib_file.basename
clinkopts.extend(["-L", lib_file.dirname, "-l", libname])
else:
- if library_to_link.alwayslink:
- lib_opts.append(_WHOLE_ARCHIVE_FLAGS_BY_GOOS[go.mode.goos][0])
- lib_opts.append(lib_file.path)
- if library_to_link.alwayslink:
- lib_opts.append(_WHOLE_ARCHIVE_FLAGS_BY_GOOS[go.mode.goos][1])
+ lib_opts.extend(_library_args(go, lib_file, library_to_link.alwayslink))
cc_link_flags = d[CcInfo].linking_context.user_link_flags
clinkopts.extend(cc_link_flags)
@@ -196,7 +185,7 @@ def _cc_libraries_to_link(target):
libraries_to_link = as_iterable(target[CcInfo].linking_context.libraries_to_link)
return libraries_to_link
-# Returns the library path to link for the given LibraryToLink.
+# Returns the library File to link for the given LibraryToLink.
def _cc_lib_file(library_to_link):
if library_to_link.static_library != None:
return library_to_link.static_library
@@ -208,6 +197,17 @@ def _cc_lib_file(library_to_link):
return library_to_link.dynamic_library
return None
+# Returns the linker flags to link the given static library File.
+def _library_args(go, lib_file, alwayslink):
+ if not alwayslink:
+ return [lib_file.path]
+
+ cc_basename = go.env["CC"].rpartition("/")[-1]
+ if cc_basename == "clang":
+ return ["-Wl,-force_load", lib_file.path]
+ else:
+ return ["-Wl,--whole-archive", lib_file.path, "-Wl,--no-whole-archive"]
+
_DEFAULT_PLATFORM_COPTS = select({
"@io_bazel_rules_go//go/platform:darwin": [],
"@io_bazel_rules_go//go/platform:windows_amd64": ["-mthreads"],
But I run into this error when I try to use it: Is there something I'm doing wrong? |
|
Just to set expectations, I probably won't have time to debug this issue this week, so I can't offer direct advice on what to do at the moment. The C compiler path should be available somewhere within the |
Use basename(c_compiler_path) to choose between clang or gcc style linker flags.
No worries -- thanks for the heads up, and all of the help!
I just gave that a go, and with the latest commits I just pushed, my test project works with both Whenever you get a chance, let me know what next steps you'd like to see, and I'd be more than happy to push this along. |
|
Actually, I take back what I said in that last comment about working with If I set I have some more work to do. I'll see if I can sort things out later today when I have more time. |
|
Hi @cstrahan , thanks for working on this! I'm doing a maintenance pass on all our PRs and I wanted to see if this was still an area of active development. Unfortunately I don't have Jay's knowledge to help with the implementation, just trying to clean up shop a bit. |
…g packages (bazel-contrib#2514) (bazel-contrib#2584) This reverts commit fbf8bc1 (bazel-contrib#2514) Also, update the CHANGELOG about the reverting. Fixes bazel-contrib#908, which is about the `pip-compile` not using the right files for performing the locking. It seems that the `pip` upgrade regressed this error.
…hen using alwayslink = True (#4438) <!-- Thanks for sending a PR! Before submitting: 1. If this is your first PR, please read CONTRIBUTING.md and sign the CLA first. We cannot review code without a signed CLA. 2. Please file an issue *first*. All features and most bug fixes should have an associated issue with a design discussed and decided upon. Small bug fixes and documentation improvements don't need issues. 3. New features and bug fixes must have tests. Documentation may need to be updated. If you're unsure what to update, send the PR, and we'll discuss in review. 4. Note that PRs updating dependencies and new Go versions are not accepted. Please file an issue instead. --> **What type of PR is this?** Bug fix **What does this PR do? Why is it needed?** Fixes static linking against libraries that have `alwayslink = True` specified. Fixes C++'s dynamic initialization of static variables when using cgo **Which issues(s) does this PR fix?** Fixes #1486 --- Related to #2584 and #2294
What type of PR is this?
Bug fix
What does this PR do? Why is it needed?
Fixes static linking against libraries that have
alwayslinkspecified (where the linker must be directed to link the whole archive).Which issues(s) does this PR fix?
Fixes #1486
Other notes for review
N/A
I'd be more than happy to perform any clean up / refactoring / bug fixes necessary to get this merged in. I'm also willing to block out time at your convenience to talk design/implementation in real time (whether text chat or video conferencing). Please advise.