Add `Zeno get list` command for archiving URLs from files by Copilot · Pull Request #514 · internetarchive/Zeno

Copilot · 2025-10-31T00:08:03Z

Implementing Zeno get list command to archive URLs from a file

Summary

Successfully implemented the Zeno get list command that allows archiving URLs from text file(s).

Features

✅ Read URLs from local files
✅ Support remote files (http/https URLs)
✅ Handle multiple files in a single command
✅ Ignore empty lines and comments (lines starting with #)
✅ Proper error handling for missing files and empty lists
✅ Consistent with existing command patterns (get url, get hq)
✅ Default user agent set for remote downloads

Testing

✅ E2E test added and passing (checks multiple URLs)
✅ Manual testing with local files
✅ Manual testing with multiple files
✅ Manual testing with comments and empty lines
✅ Code review completed and feedback addressed
✅ Security checks passed (CodeQL: 0 alerts)

Recent Changes

Addressed PR review feedback:

Added missing urls.txt test file with multiple URLs
Updated command Usage to show [FILE|URL...]
Set default user agent for remote file downloads
Enhanced test to verify all specific URLs are archived

Usage

# Archive URLs from a single file
./Zeno get list urls.txt --workers 1

# Archive URLs from multiple files
./Zeno get list list1.txt list2.txt --workers 2

# Archive URLs from a remote file
./Zeno get list https://example.com/urls.txt --workers 1

Original prompt

This section details on the original issue you should resolve

<issue_title>Add command to archive based on a list</issue_title>
<issue_description>- [ ] Archive based on a list (similar to the old get list functionality)

Allow a remote list to be used (Allow specifying an URL for get list to use a remote list #135)</issue_description>

<agent_instructions>We want to add a new command that lets us archive based on a text file (with a list of URLs). This should be added as Zeno get list. </agent_instructions>

Comments on the Issue (you are @copilot in this section)

Fixes Add command to archive based on a list #513

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

- Created get_list.go with support for local and remote files - Added helper functions to read URLs from files (supporting comments and empty lines) - Registered list command in get.go - Added e2e test for the list command - Updated e2e package with helper functions for list command testing Co-authored-by: NGTmeaty <2244519+NGTmeaty@users.noreply.github.com>

The MinimumNArgs(1) already ensures at least one argument is provided Co-authored-by: NGTmeaty <2244519+NGTmeaty@users.noreply.github.com>

NGTmeaty

@copilot

- Add missing urls.txt test file with multiple URLs - Update command Use field to specify FILE|URL - Set default user agent for remote file downloads - Enhance test to verify all URLs from list are archived Co-authored-by: NGTmeaty <2244519+NGTmeaty@users.noreply.github.com>

codecov-commenter · 2025-10-31T01:13:07Z

Codecov Report

❌ Patch coverage is 41.11111% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.42%. Comparing base (2abbcd3) to head (6b48e9c).

Files with missing lines	Patch %	Lines
cmd/get_list.go	31.94%	39 Missing and 10 partials ⚠️
e2e/e2e.go	76.47%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #514      +/-   ##
==========================================
- Coverage   56.61%   56.42%   -0.19%     
==========================================
  Files         130      131       +1     
  Lines        6481     6570      +89     
==========================================
+ Hits         3669     3707      +38     
- Misses       2448     2488      +40     
- Partials      364      375      +11

Flag	Coverage Δ
e2etests	`41.98% <41.11%> (+0.70%)`	⬆️
unittests	`28.81% <0.00%> (-0.34%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

NGTmeaty

Code looks good and tests are passing. I have tested locally and it works correctly.

willmhowes · 2025-11-05T01:41:40Z

+	if userAgent == "" {
+		userAgent = "Mozilla/5.0 (compatible; Zeno)"
+	}
+	req.Header.Set("User-Agent", userAgent)


The UA string should be generated to match whatever Zeno would generate by default. Okay to fix in a separate issue.

willmhowes · 2025-11-05T01:43:05Z

+	// Read file line by line
+	scanner := bufio.NewScanner(resp.Body)
+	for scanner.Scan() {
+		line := strings.TrimSpace(scanner.Text())
+		if line == "" || strings.HasPrefix(line, "#") {
+			continue
+		}
+		urls = append(urls, line)
+	}


This code chunk appears twice (also on lines 97-104), should we refactor into it's own function?

willmhowes

Looks good, so long as separate issues are made where noted

Initial plan

e5c997d

Copilot AI assigned Copilot and NGTmeaty Oct 31, 2025

Copilot started work on behalf of NGTmeaty October 31, 2025 00:08 View session

Copilot AI and others added 2 commits October 31, 2025 00:20

Remove redundant validation check in get_list

4768cfe

The MinimumNArgs(1) already ensures at least one argument is provided Co-authored-by: NGTmeaty <2244519+NGTmeaty@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add command to archive based on a list~~ Add Zeno get list command for archiving URLs from files Oct 31, 2025

Copilot AI requested a review from NGTmeaty October 31, 2025 00:28

Copilot finished work on behalf of NGTmeaty October 31, 2025 00:28

NGTmeaty reviewed Oct 31, 2025

View reviewed changes

Comment thread e2e/test/getlist/getlist_test.go

Comment thread cmd/get_list.go Outdated

Comment thread cmd/get_list.go Outdated

Comment thread e2e/test/getlist/getlist_test.go

Copilot started work on behalf of NGTmeaty October 31, 2025 00:44 View session

Copilot AI requested a review from NGTmeaty October 31, 2025 00:56

Copilot finished work on behalf of NGTmeaty October 31, 2025 00:56

NGTmeaty marked this pull request as ready for review October 31, 2025 18:00

NGTmeaty approved these changes Oct 31, 2025

View reviewed changes

willmhowes requested changes Nov 5, 2025

View reviewed changes

Copilot started work on behalf of willmhowes November 5, 2025 01:52 View session

Copilot AI requested a review from willmhowes November 5, 2025 01:54

Copilot finished work on behalf of willmhowes November 5, 2025 01:55

willmhowes approved these changes Nov 5, 2025

View reviewed changes

NGTmeaty merged commit 73c7dde into main Nov 5, 2025
7 of 8 checks passed

NGTmeaty deleted the copilot/add-archive-command-list branch November 5, 2025 01:58

This was referenced Nov 5, 2025

The UA string used in "Get List" should be generated to match the Zeno default generation scheme #517

Open

expectedURLs in the "Get List" testing are defined separate from the validation condition #518

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Zeno get list` command for archiving URLs from files#514

Add `Zeno get list` command for archiving URLs from files#514
NGTmeaty merged 4 commits into
mainfrom
copilot/add-archive-command-list

Copilot AI commented Oct 31, 2025 •

edited

Loading

Uh oh!

NGTmeaty left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Oct 31, 2025 •

edited

Loading

Uh oh!

NGTmeaty left a comment

Uh oh!

willmhowes Nov 5, 2025

Uh oh!

willmhowes Nov 5, 2025

Uh oh!

Uh oh!

willmhowes left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Copilot AI commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Features

Testing

Recent Changes

Usage

Comments on the Issue (you are @copilot in this section)

Uh oh!

NGTmeaty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

NGTmeaty left a comment

Choose a reason for hiding this comment

Uh oh!

willmhowes Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

willmhowes Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

willmhowes left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Oct 31, 2025 •

edited

Loading

codecov-commenter commented Oct 31, 2025 •

edited

Loading

willmhowes left a comment •

edited

Loading