The documentation is not available anymore as the PR was closed or merged.

Thanks @ArthurZucker, done (and added a test!)

Thanks for the ping @LysandreJik! I'm currently reviewing it. Most probably good but I want to make a few tests first with the blob patterns (I found them a bit un-intuitive TBH). Will update soon :)

@muellerzr Unfortunately, ignore_patterns=["_*", "[!runs]**/*"], will not work as expected. I've made some tests with fnmatch and TIL that even tough it claims to provide support for "Unix shell-style wildcards", I feel it's not the case.

Especially:

• * matches everything no matter if there is a / in it. Both README.md and subfolder/README.md are matched.
• "[!runs]*" matches anything that starts with a characters that is not a "r", a "u", a "n" or a "s". Meaning runs/foobar, nqwerty/foobar, sqwerty/foobar/..., etc. are all matching
• => So "[!runs]**/*" matches any folder starting with r/u/n/s which is not the expected result

Note that the filename separator ('/' on Unix) is not special to this module.

I've tried to define a good set of allow_pattern + ignore_pattern that would achieve what you want to do but unfortunately I don't think it's possible. I feel bad about it because I introduce this fnmatch module thinking it was the good one to use but since it doesn't implement the same specs as the Unix pattern it kinda useless. I expected that the pattern would follow the same rules as described when doing man 7 glob.

EDIT: just realized that ignore_patterns = ["_*", "[!r]*/*", "?[!u]*/*", "??[!n]*/*", "???[!s]*/*"] would work but it feels so stupid...

(here is my test script if you want to play with it. At first I thought it would be better to include any *.tfevents.* file, no matter the folder hence the tests. But doesn't seem possible to do anyway

from fnmatch import fnmatch

paths = [
    "readme.md",
    "subfolder/readme.md",
    "folder/readme.md",
    "_private.json",
    "subfolder/_private.json",
    "foo.tfevents.bar",
    "subfolder/foo.tfevents.bar",
    "runs/subfolder/foo.tfevents.bar",
    "not/subfolder/foo.tfevents.bar",
]

# allow_patterns = ["*.tfevents.*", "*[!\]*"]
allow_patterns = ["*"]

ignore_patterns = ["_*", "[!r]*/*", "?[!u]*/*", "??[!n]*/*", "???[!s]*/*"]

# pattern = '**/*[!tfevents]*'
# pattern = '[!runs]**/*'

def is_included(path):
    if any(fnmatch(path, r) for r in ignore_patterns):
        return False
    if any(fnmatch(path, r) for r in allow_patterns):
        return True
    return False

for path in paths:
    print(f"{path:<40}", is_included(path))

)

Actually looking back at the implementation before #25095, do we really need to exclude every folder -so the current ignore_patterns="**/*"- ? I don't see anywhere where Repositoryused to exclude some folders. If it was in a.gitignore`, I would be keen to see it.

I already thought about the fact that we should respect the .gitignore file (if any) when using upload_folder. It is not very straightforward but even a subset of the .gitignore's spec would be good. (only thinking out loud here, let's fix the PR independently of that)

we should respect the .gitignore file (if any) when using upload_folder

Or maybe server-side support (in case the .gitignore is uploaded first, of course). I don't know if it's the case server-side already or not, to be honest.