also fix a little ugly style.

This pull request passes (merged 6cf0d0e into 2f84e14).

Hm, maybe you should add the --recursive option to update command. I have worked on python projects that had submodules with submodules, and this patch would not work.

This is a dupe of #289 and #421 - thus far, the decision was to not add submodule support. I'm reconsidering that, though - this is clearly a pain point, and a good case can be made that including submodules would still be within the goal of "simply checking out a copy of the repo" - when a git repo contains submodules, those submodules are essentially part of the repo. And it's hard to actually come up with any way that this would break in real usage. So I'd be ready to merge something like this (though I do think it should probably use --recursive instead; the versions of git without it are old enough now that I don't think we have to worry about them). @jezdez, would you be open to it?

why shouldn't support submodule? It's just a few lines of code.

submodule is a good thing.

I'm -0 on this one, as I previously said over and over again. I avoid git submodules simply because they add a level of indirection to something that would be better solved with multiple requirements. Hence I won't take responsibility for any instability such a feature would introduce.

@jezdez how could it be done with multiple requirements?

for example, I have a repo: july , it has a module july.auth which is tornado.third. I have tried:

-e git+git://github.com/lepture/july.git#egg=july
-e git+git://github.com/lepture/tornado.third.git#egg=july/auth

and

-e git+git://github.com/lepture/july.git#egg=july
-e git+git://github.com/lepture/tornado.third.git#egg=july.auth

both failed.

Why should I do such thing? Because tornado.third is a collection of code snippets which I created earlier, but later, I want to add it to july, I don't have to copy and paste, I just submodule it.

Git submodule makes it efficient to do such thing.

How could I solved this with multiple requirements?

@jezdez Besides, submodule support won't bother anything. It just makes life easier.

I think the benefits here outweigh the downsides, so if @jezdez is a -0 I'm willing to take responsibility for adding this and any fallout from it (I don't think there will be any - if there is I wlll graciously accept any and all "I told you so"s :P). I don't think this pull request is ready, though; at clone time I think it should just --recursive, and it also needs to handle the case where the repo is updated and the submodule(s) may have changed; needs git submodule update.

This pull request fails (merged 933d0e9 into 2f84e14).

@carljm this pull request fails on 3.1.

@lepture that's actually a known intermittent failure, the pull request tests out fine. It does, however, still need a test. The test should actually create the repo with a submodule locally in the test scratch directory, and then try to install it and make sure it installs correctly.

@lepture Well, your tornado.third repository isn't a Python package, that's why it doesn't work. IMO, you're trying to shoehorn something into pip that it's not supposed to handle. The most common requirement for pip is having a proper Python package (setup.py et al). In other words, if you don't want your dependency to be a proper Python package, I have to say, this makes me more than nervous about this feature request. I don't say that lightly, learned that lesson the hard way, but if you're using a version control system feature for software dependency management, you're doing it wrong.

I do trust @carljm to take care of any backfiring though, hence the -0. I've had my share of maintenance of the git module.

@jezdez no, it's not, and that's why I need submodule. If it is a python package, I don't need this feature.

@jezdez

for example, I have a repo: july , it has a module july.auth which is tornado.third.

tornado.third is a submodule for july.auth

@jezdez OMG, you don't even understand what a submodule is. I don't say that lightly, learn how to use submodule is hard for you. But if your're using submodule feature for your projects, you're doing it wrong.

This is a nice-to-have feature. This kind of requirement has been around
for more than one year.

If it is just as simple as a few lines in the corresponding backend, why
not just have it?

@jezdez

I understand your concern. However I have different opinion. Not every
dependecy is worthy to make into a Python package. Furthur more, a Python
project might have dependency that's not written in Python. So having
submodule to manage dependecies is not a bad idea. It can even make your
life eaiser.

@liuw I appreciate your opinion, as I have @lepture's. As a maintainer of pip I have to weigh the cost of maintenance of a "nice-to-have feature" against its usefulness. Having written the git vcs backend for pip in the first place I have a fair amount of experience to do that. And it has been nothing but a fragile piece of code in the past, partly due to the way it's implemented, but also part of the level of indirection it introduces into requirement set resolving (e.g. network bound, varying git API, etc), making it flaky. Hence my strong feelings against adding support for git submodules as it would increase the likelihood of something going wrong for our users.

@lepture, @liuw, let it rest, please (and no more personal attacks on someone's understanding). I've already agreed to merge the feature, so further discussion of the value of the feature is useless. The only useful further contribution to this pull request is a test, so that it can actually be merged. Anything else is noise.

@carljm I don't mean to attack anyone. And I totally agree with you that there
needs no more discussion on the value of this feature.

However, one thing I notice in the docs/usage.txt is that pip claims it will
not handle advanced VCS-specific features such as submodules or subrepos. This
looks more like a policy than a technical issue. Do you need to change this
policy before actaully accepting any patches?

I'd suggest @lepture updates the docs when he does the test for the pull request. If you need help with testing strategies feel free to ask me.

@carljm @jezdez My apologies.

@pnasrat I will try it, and if I can't do the job, I would ask for your help. Thank you.

@carljm first, thank you so much for agreeing to merge this.

i'd volunteer to write a test for this, but i'm not sure how to test this.

@pnasrat have you already thought about that?

can we depend on github for such tests?
if not, i'd consider including two sample modules (.tgz'd), but i'm not sure if relative paths would work here.

@fin Thanks for offering to add tests! I'd prefer to avoid adding new github dependencies whenever possible. Rather than packaging up git repos as tgzs, I would actually call git in a subprocess (using env.run) to create the two repos in the test-env scratch path, set the one up as submodule of the other, write out the setup.py for the main repo, and then try installing that main repo with -e git+file:///path/to/the/repo and make sure the submodule comes along with it. Make sense? It'll be a bit of an involved test, but all the pieces should be pretty straightforward.

Merged as part of #577

@fin thanks. I am busy these days.