Skip to content

Bug: Broken URLs block entire import when using depth=1 #944

@devsorice

Description

@devsorice

Describe the bug

While using command "archivebox add",
impossible to import a list of hundred, or thousands of good links, if there is just one broken link
Is there a way to ignore broken links?
I understand that devs might have intended this as a feature, however the cli is not usable, if you happen to
have even one link that doesn't work, you cant' import anything.
See also #444

Steps to reproduce

Just put any broken link, or a link to a website that is currently not online in a txt file

Screenshots or log output

Command used
docker-compose run -e ONLY_NEW=true archivebox add --depth=1 --tag=barra-dei-preferiti < barra-dei-preferiti.txt

Output
[!] Failed to download https://www.wunderlist.com/webapp/#/tasks/859529680

HTTPSConnectionPool(host='www.wunderlist.com', port=443): Max retries exceeded with url: /webapp/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0adabe64c0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
ERROR: 1

ArchiveBox version

docker tag :latest

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions