How can I use git repos as dependencies for my PyPi package?

Question

I have a package which I'm pushing to PyPi and some of the depedencies are not packages, but installable git repositories. My requirements.txt looks like this

sphinx_bootstrap_theme>=0.6.5
matplotlib>=2.2.0
numpy>=1.15.0
sphinx>=1.7.5
sphinx-argparse>=0.2.2
tensorboardX
tqdm>=4.24.0
Cython>=0.28.5

# git repos
git+git://github.com/themightyoarfish/svcca-gpu.git

Accordingly, my setup.py has this content:

#!/usr/bin/env python

from distutils.core import setup
import setuptools
import os

with open('requirements.txt', mode='r') as f:
    requirements = f.read()
    required_pkgs, required_repos = requirements.split('# git repos')
    required_pkgs = required_pkgs.split()
    required_repos = required_repos.split()

with open('README.md') as f:
    readme = f.read()

setup(name=...
      ...
      packages=setuptools.find_packages('.', include=[...]),
      install_requires=required_pkgs,
      dependency_links=required_repos,
      zip_safe=False,   # don't install egg, but source
)

But running pip install <package> does not actually install the git dependency. I assume that pip doesn't actually use the setup script. It works when I run python setup.py install manually.

Edit:

I also tried removing dependency_links and just using install_requires with the repository, but when installing my repository from GitHub (the project including the above files), I'm met with

    Complete output from command python setup.py egg_info:
error in ikkuna setup command: 'install_requires' must be a string or 
list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+git://g'"

It has been suggested in other answers that one can put something like

git+https://github.com/themightyoarfish/svcca-gpu.git#egg=svcca

into requirements.txt, but that fails with

   error in <pkg> setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+https:/'

Question: (How) Can I list git repositories as dependencies for a pip package?

Possible duplicate of How to state in requirements.txt a direct github source — Bram Vanroy
– Bram Vanroy, Commented Feb 26, 2019 at 15:10
The linked question and top answers do not seem to solve the problem because it deals with requirements files, but not with pip. The idea is having to state the requirement in a way that setup() understands it. — oarfish
– oarfish, Commented Feb 26, 2019 at 21:33
I should have made it clearer in the question that requirements.txt is merely a proxy for setuptools dependencies in this case. — oarfish
– oarfish, Commented Feb 26, 2019 at 21:44

oarfish · Accepted Answer · 2019-02-26 21:26:03Z

15

Out of the 50 or so different ways to specify git dependencies for Pip, the only one that did what I intended was this one (outline in PEP 508):

svcca @ git+ssh://[email protected]/themightyoarfish/svcca-gpu

This can be used in install_requires, which solves the issue of dependency_links being ignored by pip.

An amusing side-effect is that the package cannot be uploaded to PyPi with such a dependency:

HTTPError: 400 Client Error: Invalid value for requires_dist. Error: Can't have direct dependency: 'svcca @ git+ssh://[email protected]/themightyoarfish/svcca-gpu' for url: https://upload.pypi.org/legacy/

answered Feb 26, 2019 at 21:26

oarfish

4,7745 gold badges43 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

TG Gowda Over a year ago

I am stuck on " Error: Can't have direct dependency:" issue and not sure how to solve it :-(

oarfish Over a year ago

"Don't use PyPi" is the only solution until they change this.

TG Gowda Over a year ago

I had to release a package to PyPi. So I ended up releasing two packages: first, the direct dependency, and then releasing my actual package after converting direct dependency to regular dependency!

mara004 Over a year ago

Ran into the same problem with pypdfium2 (issue 177). Not so amusing after all.

David Davó Over a year ago

Where is the docs about this? Can I set the dependency in extras_require?

user12520082 · Accepted Answer · 2023-06-13 18:56:13Z

3

We tried it with pyproject.toml and it didn't work for us either.

requires = [
    "geci_plots @ git+https://github.com/IslasGECI/[email protected]",
    "lmfit",
    "pandasql",
    "scikit-learn==1.1.3",
]

It seems that direct references don't allow it.

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Invalid value for requires_dist. Error: Can't have direct dependency:  
         'geci_plots @ git+https://github.com/IslasGECI/[email protected]'

I assume the direct references are the GitHub repos.

answered Jun 13, 2023 at 18:56

user12520082

312 bronze badges

Comments

cov · Accepted Answer · 2025-04-15 11:38:07Z

I hit

400 Can't have direct dependency: hadolint-py@ git+https://github.com/AleksaC/hadolint-py.git ; extra ==
         "precommit". See https://packaging.python.org/specifications/core-metadata for more information.
           The server could not comply with the request since it is either malformed or otherwise incorrect.

when trying to move my requirements.in into the [project.optional-dependencies] section of pyproject.toml.

To resolve the issue, my first approach was to keep the git+https dependency in requirements.in, while moving everything else to pyproject.toml. A more elaborate fix was to add the following to the [build-system] section of pyproject.toml:

build-backend = 'pypi_compatible_build'
backend-path = ['']

And adding this pypi_compatible_build.py file:

"""
Avoid git URLs breaking PyPI uploads.
Similar problem to:
https://stackoverflow.com/questions/54887301/how-can-i-use-git-repos-as-dependencies-for-my-pypi-package
"""

from io import StringIO
from typing import TextIO

import setuptools
from packaging.metadata import Metadata
from setuptools._core_metadata import _write_requirements  # type: ignore[import-not-found]
from setuptools.build_meta import *  # noqa: F403


def write_pypi_compatible_requirements(self: Metadata, final_file: TextIO) -> None:
    """Mark requirements with URLs as external."""
    initial_file = StringIO()
    _write_requirements(self, initial_file)
    initial_file.seek(0)
    for initial_line in initial_file:
        final_line = initial_line
        metadata = Metadata.from_email(initial_line, validate=False)
        if metadata.requires_dist and metadata.requires_dist[0].url:
            final_line = initial_line.replace('Requires-Dist:', 'Requires-External:')
        final_file.write(final_line)


setuptools._core_metadata._write_requirements = write_pypi_compatible_requirements

References:

user30942190 · Accepted Answer · 2025-07-01 21:51:24Z

0

I removed all git dependencies from my toml/setup.py since pypi is very strict about that and added a try except to my entry point before the import in question. So I have an intentionally failed import when you run the code the first time. This triggers the subprocess to install the package then reruns the code. So the user can still pip install your package and then your package installs the git dependency when they run it.

This is dependent on the user having git on their system. Not a perfect solution but a decent workaround.

Edit: Thank you @sinoroc for pointing that out this version asks the users permission to install the missing git dependency.

import subprocess
import sys
import os

try:
    from svcaa import something
except ImportError:
    user_input = input("svcaa not found. Would you like to install it? (y/n): ").strip().lower()
    if user_input == 'y':
        print("Installing svcaa from GitHub...")
        subprocess.check_call([
            sys.executable,
            "-m", "pip", "install",
            "svcaa @ git+https://github.com/..."
        ])
        print("Installation complete. Relaunching, please wait...")
        os.execv(sys.executable, [sys.executable] + sys.argv)
    else:
        print("Installation skipped. Exiting the program.")
        sys.exit(0)

edited Jul 1, 2025 at 21:51

answered Jul 1, 2025 at 19:57

user30942190

12 bronze badges

1 Comment

sinoroc Jul 1, 2025 at 20:43

That seems quite bad. I would hate to have one of the libraries I use install things behind my back. I strongly recommend against ever distributing code that does this without explicit warning to the users and without asking for explicit consent from the users.

R. García · Accepted Answer · 2019-02-26 14:13:16Z

-1

According to the next post related to How to state in requirements.txt a direct github source. You could add a package from git remote repository with the next syntax:

-e git://github.com/themightyoarfish/svcca-gpu.git

Reference: Install a project in editable mode (i.e. setuptools “develop mode”) from a local project path or a VCS url with-e

answered Feb 26, 2019 at 14:13

R. García

8239 silver badges20 bronze badges

6 Comments

oarfish Over a year ago

I don't want to use editable mode, but it doesn't seem to work without -e.

oarfish Over a year ago

See edited question; this does not work because install_requires' must be a string or list of strings containing valid project/version requirement specifiers;, which apparently that is not.

R. García Over a year ago

Hi oarfish! I really recommend you follow the next structure created by kennethreitz. On this repository you will see how to create an awesome Setup.py

R. García Over a year ago

Maybe, you should split a break space, something like that: required_pkgs = required_pkgs.splitlines() Reference

oarfish Over a year ago

I really don't see how your linked setup.py file deals with non-pypi packages at all.

|

Collectives™ on Stack Overflow

How can I use git repos as dependencies for my PyPi package?

5 Answers 5

5 Comments

Comments

Comments

1 Comment

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

5 Comments

Comments

Comments

1 Comment

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related