Skip to content

Add support for .dockerignore#4165

Closed
tmc wants to merge 2 commits intomoby:masterfrom
tmc:dockerignore
Closed

Add support for .dockerignore#4165
tmc wants to merge 2 commits intomoby:masterfrom
tmc:dockerignore

Conversation

@tmc
Copy link
Contributor

@tmc tmc commented Feb 15, 2014

Fixes #2224

This improves the archive package's API and reads a .dockerignore file as a list of paths to exclude while building the context.

@SvenDowideit
Copy link
Contributor

Can you please sign their commits following these rules: https://github.com/dotcloud/docker/blob/master/CONTRIBUTING.md#sign-your-work

Each commit in your PR must be signed in the following format:
Docker-DCO-1.1-Signed-off-by: Joe Smith <joe.smith@email.com> (github: github_handle)

Please rebase and sign each commit.

Also, this will need documentation to go with it,

@tmc
Copy link
Contributor Author

tmc commented Feb 16, 2014

@SvenDowideit yes. Updated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm presuming that the relative paths can be files and directories, and some kind of wildcards? Can you tell the reader about how to specify them? (I note that the tar man page punts it to PATTERN - which tells us a little more than just 'list of paths'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't mention any globbing because there isn't any.

@crosbymichael
Copy link
Contributor

ping @shykes How do you feel about his new feature?

@vieux
Copy link
Contributor

vieux commented Feb 18, 2014

I like the idea, but I think it needs globbing.

At least docs/ should work. Now you have to use docs

@tmc
Copy link
Contributor Author

tmc commented Feb 19, 2014

@vieux I didn't want to add the complexity and overhead of globbing. I'd like to see this go in and someone can add globbing in an efficient and reasonable manner later.

@tianon
Copy link
Member

tianon commented Feb 19, 2014

Something like path.Match? http://golang.org/pkg/path/#Match

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be "Skipping excluded directory:" ?

@tmc
Copy link
Contributor Author

tmc commented Feb 20, 2014

Added path.Match support for exclusion.

@vieux
Copy link
Contributor

vieux commented Mar 6, 2014

@tmc could you please rebase ?

@tmc
Copy link
Contributor Author

tmc commented Mar 7, 2014

@vieux yep.

@vieux
Copy link
Contributor

vieux commented Mar 7, 2014

with:

cat .dockerignore
AUTHORS
FIXME
LICENSE
MAINTAINERS
NOTICE

It fails, is that normal ?

@vieux
Copy link
Contributor

vieux commented Mar 10, 2014

ping @tmc do you still have that do to this ?

could you also add a .dockerignore file to the repo and automatically add .dockerignore to the exclude list

@tianon
Copy link
Member

tianon commented Mar 10, 2014

+1 we should definitely ".dockerignore" on our repo, and I'd say it should include "bundles" and ".gopath" off the top of my head

@vieux
Copy link
Contributor

vieux commented Mar 11, 2014

Also *.md docs contrib pretty much all ALL CAPS FILES in / .git I guess

@tianon
Copy link
Member

tianon commented Mar 11, 2014

Whoa @vieux slow down. "contrib" is 100% necessary in our container, docs is useful (will be more useful in the future), several of those ALL CAPS FILES ("VERSION" especially comes to mind) are useful but don't hurt (they're all tiny), and adding ".git" would completely break @crosbymichael's workflow not to mention our build scripts. So I still stand by my currently two-item list.

@vieux
Copy link
Contributor

vieux commented Mar 11, 2014

@tianon ok I guess I had cache, because I rm contrib and I was still able to do a make.

@tianon
Copy link
Member

tianon commented Mar 11, 2014

@vieux try building the ubuntu package without contrib ;)

@vieux
Copy link
Contributor

vieux commented Mar 11, 2014

It's ok, I'll have my own .dockerignore that would speedup my daily testing process by skipping all the stuff not needed by go build

@unclejack
Copy link
Contributor

@tmc Can you rebase, please?

@tianon
Copy link
Member

tianon commented Apr 2, 2014

I just wanted to jump on and give this another +1 :)
(also, I'll note while I'm here that it has been rebased by @tmc as requested 👍)

@unclejack
Copy link
Contributor

@tianon Thank you for the ping.

@tmc Thank you for rebasing. It looks like there are some failures in the tests:

+ go test github.com/dotcloud/docker/archive
--- FAIL: TestTarWithOptions (0.00 seconds)
        archive_test.go:157: Expected 1 changes, got 2 for &{Includes:[.] Excludes:[1] Compression:0}:
FAIL
exit status 1
FAIL    github.com/dotcloud/docker/archive      0.031s

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please link to the rules if they're complicated, or replicate them here if not?

@SvenDowideit
Copy link
Contributor

Docs LGTM - I just noted some little tweaks - @jamtur01

@shykes
Copy link
Contributor

shykes commented Apr 3, 2014

I have a bad gut feeling about this, but don't really have the time to think about a better alternative, and since this seems like a popular request I don't want to deal with the whining, so let's do it. I am not guaranteeing this won't break before 1.0.

@tmc
Copy link
Contributor Author

tmc commented Apr 3, 2014

Hrm. Let's not land this then and make sure we do the right thing.

@tmc tmc closed this Apr 3, 2014
@tristanz
Copy link

Breaking this at 1.0 if necessary doesn't seem like a big deal, especially if there's a replacement. This is a major pain for rapid builds and .dockerignore is consistent with .gitignore and .npmignore so +1 to reopen and merge this.

@wyaeld
Copy link

wyaeld commented Jun 1, 2014

@tmc @LK4D4 it looks like the problem is related to multiline .dockerignore I've been able to get it to build all the tests by making them all a single entry only.

I'm guess string.Split(s, '\n') isn't handling trailing whitespace or newlines very well, I'll see if I can fix that.

@wyaeld
Copy link

wyaeld commented Jun 2, 2014

Splitting doesn't appear to be the problem, see comments in tmc#1

@unclejack
Copy link
Contributor

I'm getting some failures in the tests with this PR:

--- FAIL: TestTarWithOptions (0.00 seconds)                                                                                                │2014/06/04 03:06:12 GET /v1.12/containers/4169009efb
        archive_test.go:158: Expected 1 changes, got 2 for &{Includes:[.] Excludes:[1] Compression:0 NoLchown:false}:                      │13b067aa54e4788a121cfd44cf8678198758832d950df49240ae
FAIL                                                                                                                                       │97/json
exit status 1                                                                                                                              │[d876da5f] +job container_inspect(4169009efb13b067aa
FAIL    github.com/dotcloud/docker/archive      0.024s                                                                                     │54e4788a121cfd44cf8678198758832d950df49240ae97)
                                                                                                                                           │[d876da5f] -job container_inspect(4169009efb13b067aa
Tests failed: ./archive

@wyaeld
Copy link

wyaeld commented Jun 4, 2014

That test is incorrect. See the changes on tmc#1

Still not working overall, there is an issue with multiline dockerignores that still hasn't been resolved, and I haven't had the time yet to track it down.

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 6, 2014

I'll take a look today

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 8, 2014

@tmc Hi! Are you here?

@tmc
Copy link
Contributor Author

tmc commented Jun 8, 2014

@LK4D4 yes?

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 8, 2014

@tmc There is an error in your PR, you must return SkipDir only for dirs.
It must be something like:

                    if matched {
                        fmt.Println("MATCHED", relFilePath)
                        utils.Debugf("Skipping excluded path: %s\n", relFilePath)
                        fi, err := os.Stat(relFilePath)
                        if err != nil {
                            utils.Errorf("Failed to stat file %s\n", relFilePath)
                            return err
                        }
                        if fi.IsDir() {
                            return filepath.SkipDir
                        }
                        return nil
                    }

Do you have time to fix it? And I think we should check that there is no Dockerfile in .dockerignore. Also @wyaeld made PR in your branch with tests.

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 8, 2014

@tmc if you have no time, then let me know. I can take care of #2224 if you don't mind.

@wyaeld
Copy link

wyaeld commented Jun 9, 2014

@LK4D4 I didn't understand enough of what the subtleties were with the issue to be confident in a fix, but I like what you've done in your branch.

Also your fakecontext is much nicer that actual fixtures

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 10, 2014

@wyaeld This is about SkipDir if you're interested. When you return it for file, then whole directory that contains this file will be skipped.
If there will no news from @tmc, then I'll post my branch as another PR.

@tmc
Copy link
Contributor Author

tmc commented Jun 10, 2014

I've rebased and fixed the SkipDir deal and am incorporating @wyaeld's pr.

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 10, 2014

@tmc Oh, hi! Could you pls add docs again? And take look on last two commits here. I think I have some cool additions for your PR, such as handling ignoring Dockerfile and more sophisticated tests for archive/archive.go. Also there are cool tests without fixtures :)

Docker-DCO-1.1-Signed-off-by: Travis Cline <travis.cline@gmail.com> (github: tmc)
@tmc
Copy link
Contributor Author

tmc commented Jun 12, 2014

Alright rebased with docs and tests now.

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 12, 2014

ping @vieux
Not sure if we should protect Dockerfile from ignoring?
@tmc also you can add .dockerignore to Docker repo, bundles and .gopath for start.

@wyaeld
Copy link

wyaeld commented Jun 13, 2014

I think erroring if they try to dockerignore the Dockerfile is ideal.

It wasn't intuitive to me until learning from the tests and code that the archive.go step tars the context before trying to build, so excluding the Dockerfile gives you an unbuildable context.

@omeid
Copy link

omeid commented Jun 13, 2014

+1 I was talking about this today in IRC. :)

Fixes moby#2224

Docker-DCO-1.1-Signed-off-by: Travis Cline <travis.cline@gmail.com> (github: tmc)
@tmc
Copy link
Contributor Author

tmc commented Jun 13, 2014

Added Dockerfile ignore erroring and a .dockerignore.

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 13, 2014

ping @tianon @vieux
Really useful feature!

@wyaeld
Copy link

wyaeld commented Jun 20, 2014

reping @tianon @vieux

This is something some of us are dying to have available

Looked like it might need rebased again though

@tianon
Copy link
Member

tianon commented Jun 20, 2014

IANTM and have already expressed my +1 ;)

@vieux
Copy link
Contributor

vieux commented Jun 20, 2014

@crosbymichael @unclejack do you agree on this ? If you LGTM I'll carry it to master

@unclejack
Copy link
Contributor

@vieux Yes, I think it's going to be useful.

@LK4D4
Copy link
Contributor

LK4D4 commented Jun 20, 2014

Ah, cool

@vieux vieux mentioned this pull request Jun 20, 2014
@thaJeztah thaJeztah added the area/builder Build label Jun 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow files and paths to be ignored when uploading context