Thanks! It's really useful!

Very usefull, tnx!

Big deal, thanks.

Also, if I've installed poetry just from scratch and has never called poetry config and don't have ~/config/poetry/config.toml, poetry add source ... fails with an error because it does not find this file.

Can you please add a check if config.toml exists (and create it automatically if it doesn't)?

@Felix-neko updated to handle non-existent config file

@Felix-neko updated to handle non-existent config file

Great! Thank you ^_^

@bmarroquin , hi again! I'm trying to use it under Windows, but cannot set global sources.

I've installed the poetry from your fork, run poetry config and after them run

poetry source add --global --default alfabank "http://binary/artifactory/api/pypi/pipy-virtual/simple" -vvv

It looks like it does not search for config.toml in proper place. Here's its output:

C:\>poetry source add --global --default alfabank "http://binary/artifactory/api/pypi/pipy-virtual/simple" -vvv 
                                                                                                                
  Stack trace:                                                                                                  
                                                                                                                
  12  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:330 in run                          
       328│                                                                                                     
       329│             try:                                                                                    
     → 330│                 exit_code = self._run(io)                                                           
       331│             except Exception as e:                                                                  
       332│                 if not self._catch_exceptions:                                                      
                                                                                                                
  11  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\application.py:189 in _run               
       187│         self._load_plugins(io)                                                                      
       188│                                                                                                     
     → 189│         return super()._run(io)                                                                     
       190│                                                                                                     
       191│     def _configure_io(self, io: "IO") -> None:                                                      
                                                                                                                
  10  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:425 in _run                         
       423│                 io.set_input(ArgvInput(argv))                                                       
       424│                                                                                                     
     → 425│         exit_code = self._run_command(command, io)                                                  
       426│         self._running_command = None                                                                
       427│                                                                                                     
                                                                                                                
   9  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:467 in _run_command                 
       465│                                                                                                     
       466│         if error is not None:                                                                       
     → 467│             raise error                                                                             
       468│                                                                                                     
       469│         return event.exit_code                                                                      
                                                                                                                
   8  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:451 in _run_command                 
       449│                                                                                                     
       450│             if event.command_should_run():                                                          
     → 451│                 exit_code = command.run(io)                                                         
       452│             else:                                                                                   
       453│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED                                
                                                                                                                
   7  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\base_command.py:118 in run                
       116│         io.input.validate()                                                                         
       117│                                                                                                     
     → 118│         status_code = self.execute(io)                                                              
       119│                                                                                                     
       120│         if status_code is None:                                                                     
                                                                                                                
   6  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\command.py:85 in execute                  
        83│                                                                                                     
        84│         try:                                                                                        
     →  85│             return self.handle()                                                                    
        86│         except KeyboardInterrupt:                                                                   
        87│             return 1                                                                                
                                                                                                                
   5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\source\add.py:78 in handle      
        76│             name=name, url=url, default=is_default, secondary=is_secondary                          
        77│         )                                                                                           
     →  78│         existing_global_sources = deepcopy(self.poetry.config.get("sources", {}))                   
        79│         existing_local_sources = self.poetry.get_sources()                                          
        80│         if is_global:                                                                               
                                                                                                                
   4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\command.py:20 in poetry         
        18│     def poetry(self) -> "Poetry":                                                                   
        19│         if self._poetry is None:                                                                    
     →  20│             return self.get_application().poetry                                                    
        21│                                                                                                     
        22│         return self._poetry                                                                         
                                                                                                                
   3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\application.py:121 in poetry             
       119│                                                                                                     
       120│         self._poetry = Factory().create_poetry(                                                     
     → 121│             Path.cwd(), io=self._io, disable_plugins=self._disable_plugins                          
       122│         )                                                                                           
       123│                                                                                                     
                                                                                                                
   2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\factory.py:41 in create_poetry                   
        39│             io = NullIO()                                                                           
        40│                                                                                                     
     →  41│         base_poetry = super().create_poetry(cwd)                                                    
        42│                                                                                                     
        43│         locker = Locker(                                                                            
                                                                                                                
   1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\core\factory.py:35 in create_poetry              
        33│         from .pyproject.toml import PyProjectTOML                                                   
        34│                                                                                                     
     →  35│         poetry_file = self.locate(cwd)                                                              
        36│         local_config = PyProjectTOML(path=poetry_file).poetry_config                                
        37│                                                                                                     
                                                                                                                
  RuntimeError                                                                                                  
                                                                                                                
  Poetry could not find a pyproject.toml file in C:\ or its parents                                             
                                                                                                                
  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\core\factory.py:437 in locate                     
      433│                                                                                                      
      434│         else:                                                                                        
      435│             raise RuntimeError(                                                                      
      436│                 "Poetry could not find a pyproject.toml file in {} or its parents".format(           
    → 437│                     cwd                                                                              
      438│                 )                                                                                    
      439│             )                                                                                        
      440│                                                                                                      
                                                                                                                

Can you test it under Windows?

And one more thing...

The feature you've added will be used mostly by users that don't have direct access to PyPi from their machines. Id est, to install poetry with install-poetry.py --path ... they have to configure pip first to make pip use given local package repo instead of PyPi. Can you add a few words about what to write to pip config -v before installation to poetry docs?

@Felix-neko This is failing because the source command expects to be executed from within your project directory. It is looking for the pyproject.toml not config.toml. I will see if this can be skipped when global is set.

And one more thing...

The feature you've added will be used mostly by users that don't have direct access to PyPi from their machines. Id est, to install poetry with install-poetry.py --path ... they have to configure pip first to make pip use given local package repo instead of PyPi. Can you add a few words about what to write to pip config -v before installation to poetry docs?

This is outside of the scope of this PR. Regardless, install-poetry.py pip usage runs with --isolated and ignores existing configuration. The best way to implement this would be to add pip specific options to the install script and use them during install.

@Felix-neko you can also install with pipx and that should respect your pip configuration.

Thanks for working on this!

I will see if this can be skipped when global is set.

@bmarroquin , if you can skip this check of pyproject.toml for this command, it will be really cool, because a user like me can try to set package repos before initialization of its project ; )

The best way to implement this would be to add pip specific options to the install script and use them during install.

Well, if you can add an option for install-poetry.py to use a custom package repo for its pip instance -- it will be etremely cool. But even if you just add some information about how to properly configure C:/ProgramData/pip/pip.ini -- it will help new poetry users like me = )

Here's what's written in my pip.ini:

[global]
index-url = https://artapp.moscow.alfaintra.net/artifactory/api/pypi/pypi-remote/simple
trusted-host = artapp.moscow.alfaintra.net

The best way to implement this would be to add pip specific options to the install script and use them during install.

Well, if you can add an option for install-poetry.py to use a custom package repo for its pip instance -- it will be etremely cool. But even if you just add some information about how to properly configure C:/ProgramData/pip/pip.ini -- it will help new poetry users like me = )

Here's what's written in my pip.ini:

[global]
index-url = https://artapp.moscow.alfaintra.net/artifactory/api/pypi/pypi-remote/simple
trusted-host = artapp.moscow.alfaintra.net

There is an existing for this request: #4738

Thanks.

Hey @bmarroquin, thanks for working on this! We came across this as we're trying to adopt Poetry, but our organization requires that we use a different PyPI registry depending on the type of host we're running on (e.g. CI needs a different URL than local development environments), so this support would be very useful for us since we can't hard-code a single URL in our checked-in project config files.

I noticed this PR has stalled out a bit, is there anything we could do to help push it over the line? We're happy to contribute with bugfixes or any remaining work if that would be helpful.

@chriskuehl, at the time there was a lot of typing/format changes happening that it was some work keeping up with it. I plan to get to this sometime next week.

Any movement on this? I also work on a team behind corporate fence and this would be incredibly helpful. I'd be happy to adopt this change to move it along.

Apologies if this is too off-topic but couldn't find a better place to comment this. In case it's useful to anyone else, we ended up working around this lack of support using a Poetry plugin (requires the latest Poetry beta) to automatically adjust the index server. Our code looks something like:

from poetry.plugins.plugin import Plugin
from poetry.repositories.legacy_repository import LegacyRepository

class InternalPyPiPlugin(Plugin):
    def activate(self, poetry, io) -> None:
        for repo in poetry.pool.repositories:
            poetry.pool.remove_repository(repo.name)
        poetry.pool._default = False
        # We actually read the URL from a config file, but you get the idea...
        repository = LegacyRepository(name="InternalPyPi", url="https://my.pypi/simple")
        poetry.pool.add_repository(repository, default=True)

Would love to help get this change merged as well though to avoid this.

Apologies if this is too off-topic but couldn't find a better place to comment this. In case it's useful to anyone else, we ended up working around this lack of support using a Poetry plugin (requires the latest Poetry beta) to automatically adjust the index server. Our code looks something like:

from poetry.plugins.plugin import Plugin
from poetry.repositories.legacy_repository import LegacyRepository

class InternalPyPiPlugin(Plugin):
    def activate(self, poetry, io) -> None:
        for repo in poetry.pool.repositories:
            poetry.pool.remove_repository(repo.name)
        poetry.pool._default = False
        # We actually read the URL from a config file, but you get the idea...
        repository = LegacyRepository(name="InternalPyPi", url="https://my.pypi/simple")
        poetry.pool.add_repository(repository, default=True)

Would love to help get this changed merged as well though to avoid this.

Hi @chriskuehl , thanks for sharing this piece of code.

Could you please elaborate about how you created/use such a plugin ? I am also need to add a token in the URL of my private repository and so it cannot be part of the pyproject.toml

Thanks

This PR is important to a lot of people: anyone working within an organization that disallows install from public pypi and anyone installing from a pypi mirror. Issues #1070, #1632, #2940, #4738, #5005, #5147, and #5219 all refer to the issues this PR starts to address.

What is needed to get this PR merged in? Just need to get it passing the build?

@Felix-neko @JacobHenner , any chance we could get a comment from either of you on what is missing to get this merged?

This patch is important for my project as I have a poor network connection to pypi and I have to use pypi mirror, but I DO NOT want to add my pypi mirror site to the project local config file, I hope this could be merged soon.

This patch is important for anyone that can't access PyPI directly and does not want repo configuration on pyproject.toml. What is needed to merge?

I'm going to close this for now to avoid false hope -- the approach here isn't really tractable as it's leaking abstractions across different layers of Poetry internals (more than we already do), and the design is broken with regard to how we capture sources in lock files, etc (as well as the content-hash of the lock file being broken as well).

Basically, if we are to implement this in Poetry, we have to spend a good amount of time thinking about the design first and how we can gracefully support it (if it makes sense to support), without the compromises here. Also, I would like to complete refactoring of the related code before we introduce anything like this -- adding multi-layer configuration adds a substantial amount of complexity and moving parts if we don't reduce technical debt here first.

Well, this is unfortunate. I mean poetry will remain a toy for opensource if we don't address corporate use case of private repos+mirrors, who solve staging with multiple repos.
I accept that the current design is limited but this issue is nearly a showstopper. Is the project open to contributor aspiring to clean up whatever needs to be cleaned up and refactored?
I am so upset reading the above that I would love to "rage-contribute", bringing in some customer-focus.

The project is of course open for contributions, and you are welcome to explore a design and even implementation if you want.

Keep in mind that for complex changes like this, it can often be a process to gain consensus. The design has to be something that is generic (useful to all/doesn't disadvantage some users), maintainable (as we are all volunteers and time is limited), and consistent with the existing design/scope/goals of the project.

Generally before embarking on an ambitious change I would suggest starting with smaller contributions so that you can gain experience with the code base and process. Making large changes without having experience contributing to the project can often end with disappointment as you may not be able to come up with a mergable design/implementation.

I think @bmarroquin is potentially in a good spot to work on a V2 of this, if there is time, as they have experience contributing to Poetry in the past, as well having obviously thought about this problem space. However, if you are dead set on trying to take this on as your first contribution, I strongly suggest that you at least join Discord and try and workshop concepts there. After coming up with a design that you are happy with and you think would be accepted for merge, I would create an issue describing it for discussion of the specifics.

Keep in mind that this is a very hard problem -- sources are very much coupled to the project level with the current design and architecture of Poetry, and it may be difficult/not desirable to change that. Also, keep in mind that what many people want is disparate despite it sounding similar. Some users are looking to add additional sources to all their projects (and many are in monorepos where monorepo features might make more sense anyway), and others are looking to do some sort of blanket URL replacement.

That URL replacement becomes difficult when you consider that the most common use for this functionality, files.pythonhosted.org, does not follow the typical file layout of a PEP 508 repository. Indeed, most existing proxies operate at the index level and not the individual package file level.

Finally keep in mind that this is complicated by other items that we already intend to implement in Poetry, such as 'lock file aware sdist builds' that we would like to introduce (e.g. the install-ability of your project as a dependency will be affected by any features in this space and needs to be factored in).

Basically, what I'm saying is that this is hard for even a regular contributor to attempt, and all of us are voulenteers without particular interest in exploring this feature/problem space. This is much too complex and far-reaching for a drive-by pull request to be very successful -- implementing sources past the project/monorepo level will have to be a thoughtful process and will require a lot of patience and motivation. I don't want to discourage people from contributing, but I do want people to realize this is a lot harder than "why don't you just do X."

Edit: Brett Cannon's blog post on the social dynamics of open source is quite helpful.

For what it's worth, my team landed on a similar solution to #4944 (comment). This is an area where poetry 1.2's plugin architecture really shines; a plugin can be tailored to your team's specific needs instead of having to solve everyone's problems (including the poetry maintainers themselves!) at the same time.

@neersighted thanks for the heads-up, I agree this should not be the first contribution of mine, but it is the trigger to start on smaller ones. When I work on something, I will eventually gain understanding and broader scope of the decisions behind, including in this case the ones that implicitly dismiss an aspect of opinionated pipelines.

Once I understood, and if @bmarroquin or anyone else has not started yet, then I jump into that and will either find a solution that obviously will need to be challenged first, or conclude that the design of poetry does not worth it, and stop converting more teams to use poetry. In both cases, I work backwards from the userbase, not based on limitation-driven excuses.

I am already on poetry Discord, have discussed editable git path-dependencies there some weeks and months earlier.

@jre21 I fear this will be a plugin for version 1 anyways, maybe a plugin forever, exposed to the whims of later changes, never stable enough as it is working against the design. I like to solve root causes, I am that naive. 😃

@systematicguy Thanks for your open mind and collaboration-focused attitude -- it certainly makes it sound like you could make meaningful contributions to Poetry.

However, I do need to dispute the ideation expressed here:

In both cases, I work backwards from the userbase, not based on limitation-driven excuses.

Keep in mind that there are no "excuses" in open source. Nobody here owes you, or any particular user anything. That's not to say that upstream is meant to be hostile or unhelpful, but that people naturally work on scratching their itch, because their work is voluntary and not driven by the needs of an employer. Asserting that "excuses" exist is implying that because someone wants functionality, there is a moral or ethical (since there is no business here) imperative to deliver or support it. There is not.

You can never be all things to all people, and the challenge of larger more established projects like Poetry is to maintain continuity while at the same time including new functionality and end users without harming the health of the project. That can be caused by everything from creating confusion for users, accepting hard to maintain code that creates technical debt, or losing focus and doing many things poorly instead of doing core competencies well.

Contributions to these more established projects will always have a higher bar, and it's not because people have fun being gatekeepers. The issue is that when you bring a contribution to the table, it's kind of like a puppy -- you're excited about this really cute and wonderful creature that you are bringing us, and we're left wondering about vet bills, grooming, feeding, walking, and taking care of the dog for the next 15 years, after you leave us with the dog and go do other things.

In short, maintainers will always, naturally scratch their own itch, or address use cases that interest them/excite them. If you want to see a feature realized in a project that no one is interested in developing/supporting, you need to become part of that project generally speaking. If no maintainer is interested in adding a feature, the solution is not declaring a moral failure because no one will do free work for you -- you (or someone else who is interested) need to become the solution and be the interested maintainer.

Merging substantial changes to a project, whether in code or design, is always more palatable when we know that someone will stick around to maintain it and that we won't be stuck with an albatross around our necks that no one really wanted for years to come. If you're familiar with, say, how contributions to CPython or Linux occur this is nothing new, but I think that most new contributors to Poetry come from small open source projects without a large user base, or corporate projects where money, not volunteer interest, is what drives features.

Also, as some added context, maybe this LWN article about Python's own development process and how tough decisions (or mistakes, depending on your perspective) get made might be helpful: https://lwn.net/Articles/907572/

This was definitely more complicated than i expected when i started the change. Anything I tried failed like a hack that made the user experience worse. I hadn't considered things in @neersighted's post yet.

This change didn't get into the URL replacement as i don't really need that feature. I was just tired of adding the sources to every project i created. Currently dealing with that by adding the source configuration to a cookiecutter template. We needed that to configure static code analysis tools anyway.

For your particular use case @bmarroquin, have you considered a feature that adds a default = true repository during poetry new? That seems like a much better fit for your use case.

Thanks for the idea. The small annoyance was always having to remember what the URL for our repository is and/or having to look it up. Setting it once and forgetting about it was what i was hoping for. Similar to the authentication and publishing set up. We still need cookiecutter for quick set up of other tools (mypy, black, flake8, CI files) and would not use the new command either way.

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.