Fixup relative and absolute path handling#1329
Closed
AndydeCleyre wants to merge 34 commits intojazzband:masterfrom
Closed
Fixup relative and absolute path handling#1329AndydeCleyre wants to merge 34 commits intojazzband:masterfrom
AndydeCleyre wants to merge 34 commits intojazzband:masterfrom
Conversation
This comment has been minimized.
This comment has been minimized.
This was referenced Feb 22, 2021
This comment has been minimized.
This comment has been minimized.
fb0190e to
b48444c
Compare
3 tasks
dde20b5 to
ab47233
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
atugushev
reviewed
Feb 25, 2021
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
ab47233 to
4fae968
Compare
4fae968 to
e246ee9
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
e246ee9 to
da86792
Compare
This comment has been minimized.
This comment has been minimized.
This brings with it more involved string manipulation
If passed, from_dir will be used instead of CWD to interpret relative paths. Special care is taken to guarantee _was_relative in the yielded ireqs, if originally relative. When calling format_requirement, _was_relative's presence now determines what's passed as from_dir: either CWD or None (None leads to an absolute path output)
With test, and needed changes: - Use this flag to decide what to pass as from_dir to parse_requirements - Add init param to OutputWriter: write_relative_to_output - Add corresponding from_dir logic and use to OutputWriter._format_requirement - Add test_write_relative_to_output which runs with the new flag both on and off
This ensures absolute intact paths for git+file: reqs
With test_sync_relative_path If the reqs in the output were written relative to that output file (pip-compile --write-relative-to-output), then it's wise for pip-sync to interpret reqs it finds in there from the location of the output file. This is now possible with pip-sync --read-relative-to-input. To be clear: pip-sync's "input" is pip-compile's "output."
This fixes at least two problems, for which tests are added.
And add tests via parameterization
For compatibility with pip<21.1
e.g. ../../rawr#egg=rawr-cli
... to properly set .extras
This fixes false-positive _was_relative assignments for some direct references like: ptrender[yaml] @ file:///home/andy/Code/ptrender
As we always want to set that True
Preserve ireq.constraint and _source_ireqs
At this code point we currently expect the original ireq.extras to be an empty set every time, so this does not change any current behavior. But if in the future ireqs with already parsed extras are passed, it should now preserve those.
4 tasks
Contributor
Author
|
Closing in favor of #1650 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial Summary (Outdated)
Summary, Take 2 (also outdated)
Fixup relative and absolute path handling:
These changes have been made with the general guideline of
storing paths as absolute as soon as we can, and rendering
them as relative or absolute as needed.
--upgrade-packagerelative to output fileI think: relative to output file if passed as relative, absolute if passed as absolute, pathless if passed as pathlessgit+file:ireq from source filepipdoesn't support relative paths in that form)Itemized Changes by File
utils.pyformat_requirement:strkwargfrom_dir:from_dir).path_to_urlfor abs paths.Function
abs_ireq:Context manager
working_dir:Function
fragment_string:pip_compat.pyparse_requirements:strkwargfrom_dirtoparse_requirements.If left to its default,
None, the parent of the source file is used.Either way it's passed to
abs_ireq, so any yielded local ireqshave absolute
.links, and some have._was_relative.install_req_from_parsed_requirementis called from a sensible folder,to better resolve relative paths; and try to detect if each ireq was initially relative,
to "manually" mark the resulting (absolute) ireq with
_was_relative.compile.pyClickargument type for the output file fromFiletoPath.When
Click'sFileobject is initialized with the absolute path,that full path is preserved as the
.nameattribute. So we now instantiatethe output
Fileourselves after resolving its absolute path.src_filesto their absolute paths.ensure it's properly adjacent to the input,
and stay safer and more predictable when the basename has no dot and the path does,
or the input file ends in
.txt(see Choose an approach for coherent path handling #1067, pip-compile annotations should be relative to file they are in #1107, and tests below).abs_ireqwhen collectingupgrade_install_reqs(--upgrade-package),passing the invocation dir as
from_dir.setup.pyinstall_requires,given the discussion @ https://discuss.python.org/t/what-is-the-correct-interpretation-of-path-based-pep-508-uri-reference/2815/18
from_diris passed toparse_requirementswhen parsing from
setup.pyor stdin, which really parses a temp file.This means
setup.py's parent folder, or the invocation dir if the source is stdin.writer.pycomes_from_line_project_repattern for parsing and rewritingcomes_fromstrs that point tosetup.pys andpyproject.tomls.strip_comes_from_line_re:comes_from_line_re,with named groups for
opts(-r/-c),path, andline_num._comes_from_as_string:strkwargfrom_dir.If the
ireq.comes_fromis already astrandfrom_diris passed,in addition to stripping the line number as before, rewrite the path as relative.
comes_from_line_project_rematches._format_requirement:ireqhas._was_relativeand the output is a file,pass the output file's parent as
from_dirtoformat_requirement,ensuring the written path for the
ireqis relative in that case.from_dirto_comes_from_as_string.test_cli_compile.pytest_relative_local_packagefile:or..test_input_file_with_txt_extension.txtto a separate output file (*.txt.txt),without overwriting the input file.
test_input_file_without_extension_and_dotted_pathinto an input-adjacent file with
.txtas the extension.test_annotation_relative_pathsreqs.infiles use paths relative to thereqs.txt.test_local_vcs_packagetest_duplicate_reqs_combinedpath_to_urlto detect the normalized package path in URL form.test_writer.pywriterfixture:OutputWriter,rather than passing the click
ctxentry (now just aPath).test_write_header:writer.click_ctx.params["output_file"](now just aPath), rather thanchecking that for a
.name.test_iter_lines__hash_missing:test_iter_lines__no_warn_if_only_unhashable_packages:test_format_requirement_annotation_source_ireqstest_utils.pytest_format_requirement_editable_local_pathtest_working_dirtest_local_abs_ireq_preserves_source_ireqsChangelog-friendly one-liners:
file:,<vcs>+file:, or..Changelog-friendly one-liners:
file:,<vcs>+file:, or..pip-compilewill interpret relative paths in an input file as relative to that input file, rather than the current folder, if--read-relative-to-inputis passed.pip-compilewill reconstruct relative req paths as relative to the output file, rather than the current folder, if--write-relative-to-outputis passed.pip-syncwill interpret relative paths in an input file as relative to that input file, rather than the current folder, if--read-relative-to-inputis passed.pkg[extra1,extra2].Contributor checklist