Skip to content

documentation + automated tests for project's deployment scripts#125

Merged
levibostian merged 20 commits intomainfrom
test-toolkit
Jan 17, 2026
Merged

documentation + automated tests for project's deployment scripts#125
levibostian merged 20 commits intomainfrom
test-toolkit

Conversation

@levibostian
Copy link
Copy Markdown
Owner

Related GitHub Issues

Problem

So, you wrote some scripts for decaf deployment. Have you properly tested those scripts? decaf comes with test mode when you open a PR, but for some projects, that may not be enough.

Solution

This PR...

  1. Wrote automated tests for the decaf scripts used in this project.
  2. Made documentation giving you suggestions on how to test your scripts as well.

Testing

  • Added automated tests.
  • Manually tested. If you check this box, provide instructions for others to test, too.

Notes for reviewers

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 9, 2026

decaf

Running deployments in test mode. Results will appear below.
If this pull request and all of it's parent pull requests are merged using the...

...🟩 squash 🟩 merge method... 🌴 It will not trigger a deployment. No new version will be deployed.

Learn more
Latest release: 0.9.2
Commit of latest release: fe10806

Commits since last release:
- documentation + automated tests for project's deployment scripts

Related GitHub Issues

Problem

So, you wrote some scripts for decaf deployment. Have you properly tested those scripts? decaf comes with test mode when you open a PR, but for some projects, that may not be enough.

Solution

This PR...

  1. Wrote automated tests for the decaf scripts used in this project.
  2. Made documentation giving you suggestions on how to test your scripts as well.

Testing

  • Added automated tests.
  • Manually tested. If you check this box, provide instructions for others to test, too.

Notes for reviewers


- chore(deps): update granodigital/report-annotate action to v3.1.3 (#128)

This PR contains the following updates:

Package Type Update Change

|
granodigital/report-annotate
| action | patch | v3.1.1v3.1.3 |


Release Notes

granodigital/report-annotate (granodigital/report-annotate)

v3.1.3

Compare
Source

What's Changed

  • Fixed PR file links to use proper diff anchors instead of file path
    URLs, keeping users in PR context

Full Changelog:
granodigital/report-annotate@v3.1.1...v3.1.3

v3.1.2

Compare
Source

Fixed
  • Changed error Markdown panel from [!ERROR] to [!CAUTION] as ERROR
    is not a
    supported GitHub alert type
  • Fixed duplicate @mention escaping in PR comments if already escaped
  • Fixed file being relative to runner root instead of repository root in
    PR
    comments
  • Fixed PR comments to be minimized when the latest run has no skipped
    annotations, ensuring old comments are cleared when they're no longer
    relevant
Improved
  • Truncate long file paths in PR comments for better readability

Full Changelog:
granodigital/report-annotate@v3.1.1...v3.1.2


Configuration

📅 Schedule: Branch creation - "every weekend" in timezone
America/Chicago, Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

Rebasing: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update
again.


  • If you want to rebase/retry this PR, check
    this box

This PR was generated by Mend Renovate.
View the repository job
log
.

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
- chore(deps): update dependency deno to v2.6.5 (#127)

This PR contains the following updates:

Package Update Change
deno patch 2.6.4
2.6.5

Release Notes

denoland/deno (deno)

v2.6.5

Compare
Source

  • feat(ext/canvas): support gif, webp for createImageBitmap
    (#​31402)
  • feat(ext/web): add performance.clearResourceTimings() and
    setResourceTimingBufferSize()
    (#​31603)
  • feat(node/fs): implement FileHandle.readableWebStream()
    (#​31745)
  • fix(audit): respect --level flag for exit code
    (#​31721)
  • fix(ci): rename trusted-signing-action to artifact-signing-action
    (#​31849)
  • fix(cli/test): close Node HTTP connections before sanitizer check
    (#​31782)
  • fix(ext/fetch): read file urls via vfs
    (#​31814)
  • fix(ext/fetch): support ipv6 destination host with http and socks
    proxy

(#​31683)

  • fix(ext/node): DatabaseSync garbage collection invalidating
    associated
    resources
    (#​31737)
  • fix(ext/node): add tls.setDefaultCACertificates support
    (#​31522)
  • fix(ext/node): emit deprecation warning for SlowBuffer
    (#​31772)
  • fix(ext/node): fix use-after-free in StatementSync JS iterator
    (#​31770)
  • fix(ext/node): use correct block sizes for HMAC algorithms
    (#​31775)
  • fix(ext/web): handle empty TextDecoder input when stream is true
    (#​31691)
  • fix(io): ignore BrokenPipe error in op_print
    (#​31844)
  • fix(node/child_process): ChildProcess constructor without args
    (#​31838)
  • fix(node/child_process): improve arg type validation for execFile and
    fork

(#​31829)

  • fix(node/child_process): reject null bytes in arguments
    (#​31828)
  • fix(node/child_process): use correct syscall name in spawn error
    (#​31836)
  • fix(npm): deduplication pass for initially partially resolved graphs
    (#​31787)
  • fix(runtime): handle options parameter in Deno.errors.*
    constructors

(#​31510)

  • fix(task): preserve signal-based exit codes for child processes
    (#​31789)
  • fix(tsgo): nil deref for exported symbols
    (#​31786)
  • fix(webgpu): correctly expose GPUCompilationInfo &
    GPUCompilationMessage

(#​31780)

  • fix: add more details when panicking getting dir path from deno.json
    (#​31805)
  • fix: allow multiple --allow-all args
    (#​31833)
  • fix: queue dynamic imports for modules with pending TLA
    (#​31670)
  • fix: update config schema for read permission to include ignore
    property

(#​31785)


Configuration

📅 Schedule: Branch creation - "every weekend" in timezone
America/Chicago, Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you
are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update
again.


  • If you want to rebase/retry this PR, check
    this box

This PR was generated by Mend Renovate.
View the repository job
log
.

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
- chore(deps): pin asdf-vm/actions action to b7bcd02 (#126)

This PR contains the following updates:

Package Type Update Change
asdf-vm/actions
action pinDigest b7bcd02

Configuration

📅 Schedule: Branch creation - "every weekend" in timezone
America/Chicago, Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you
are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update
again.


  • If you want to rebase/retry this PR, check
    this box

This PR was generated by Mend Renovate.
View the repository job
log
.

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

...🟩 rebase 🟩 merge method... 🌴 It will not trigger a deployment. No new version will be deployed.

Learn more
Latest release: 0.9.2
Commit of latest release: fe10806

Commits since last release:
- rename deno.json to jsonc to add comments
- build: fix generating lcov coverage file

turns out, it wasn't making it.
- ci: remove unneeded command from test workflow

the 'deno task test' command already generates the coverage report
- fix tests on the CI

Tests were passing on my development machine, but not on the CI. After looking into it, I found that when the test suite was running the deploy script, it was actually running the Dino JSR scripts, which had different behavior in the different environments.

We actually shouldn't be running those anyway; they are dependencies, and so they should be blocked. I mocked them to fix the problem.

In order to successfully mock these, I needed to update the mockabin version to a new version that contains a new feature where you can more conditionally mock binary areas. This is required because originally you could only mock all Deno commands, but that broke the ability for the test runner to even run the deploy script, because it is a Deno script. So now we can conditionally mock, which means we just mock when it's a JSR command.
- update deno lock file
- debug script tests by displaying the stdout/stderr to console

I have some failing tests with the new decaf script tests I wrote. but it's hard to know why because I can't see the script logs! I added this feature ability to the SDK and updated to that version in here.
- fix cleaning up files in steps/ automated tests
- update script testing doc to mention the decaf SDK
- install new runScript testing util functions from decaf-sdk 0.3.0
- docs: write document for how to write automated tests for your scripts
- write stdout logging test for steps/get-latest-release.ts to make sure it's human readable
- fix code coverage reports to only include code from this project

after adding mock-a-bin, code coverage report now includes the deno code for the bin override code written in the steps/*.test.ts files. Exclude them.
- write stdout logging test for steps/deploy.ts to make sure it's human readable
- implement easier method for removing anscii colors from stdout in tests
- fix printing stderr/stdout to console and returning it to the test function

the method I was using would output stdout and then stderr, but the order matters! Fixed it.
- wrote tests for steps/deploy
- use git dry-run in steps/deploy to get closer to environment parity

it's best when test mode and non-test mode runs as much similar code as possible.
- wrote tests for steps/get-latest-release
- chore(deps): update granodigital/report-annotate action to v3.1.3 (#128)

This PR contains the following updates:

Package Type Update Change

|
granodigital/report-annotate
| action | patch | v3.1.1v3.1.3 |


Release Notes

granodigital/report-annotate (granodigital/report-annotate)

v3.1.3

Compare
Source

What's Changed

  • Fixed PR file links to use proper diff anchors instead of file path
    URLs, keeping users in PR context

Full Changelog:
granodigital/report-annotate@v3.1.1...v3.1.3

v3.1.2

Compare
Source

Fixed
  • Changed error Markdown panel from [!ERROR] to [!CAUTION] as ERROR
    is not a
    supported GitHub alert type
  • Fixed duplicate @mention escaping in PR comments if already escaped
  • Fixed file being relative to runner root instead of repository root in
    PR
    comments
  • Fixed PR comments to be minimized when the latest run has no skipped
    annotations, ensuring old comments are cleared when they're no longer
    relevant
Improved
  • Truncate long file paths in PR comments for better readability

Full Changelog:
granodigital/report-annotate@v3.1.1...v3.1.2


Configuration

📅 Schedule: Branch creation - "every weekend" in timezone
America/Chicago, Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

Rebasing: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update
again.


  • If you want to rebase/retry this PR, check
    this box

This PR was generated by Mend Renovate.
View the repository job
log
.

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
- chore(deps): update dependency deno to v2.6.5 (#127)

This PR contains the following updates:

Package Update Change
deno patch 2.6.4
2.6.5

Release Notes

denoland/deno (deno)

v2.6.5

Compare
Source

  • feat(ext/canvas): support gif, webp for createImageBitmap
    (#​31402)
  • feat(ext/web): add performance.clearResourceTimings() and
    setResourceTimingBufferSize()
    (#​31603)
  • feat(node/fs): implement FileHandle.readableWebStream()
    (#​31745)
  • fix(audit): respect --level flag for exit code
    (#​31721)
  • fix(ci): rename trusted-signing-action to artifact-signing-action
    (#​31849)
  • fix(cli/test): close Node HTTP connections before sanitizer check
    (#​31782)
  • fix(ext/fetch): read file urls via vfs
    (#​31814)
  • fix(ext/fetch): support ipv6 destination host with http and socks
    proxy

(#​31683)

  • fix(ext/node): DatabaseSync garbage collection invalidating
    associated
    resources
    (#​31737)
  • fix(ext/node): add tls.setDefaultCACertificates support
    (#​31522)
  • fix(ext/node): emit deprecation warning for SlowBuffer
    (#​31772)
  • fix(ext/node): fix use-after-free in StatementSync JS iterator
    (#​31770)
  • fix(ext/node): use correct block sizes for HMAC algorithms
    (#​31775)
  • fix(ext/web): handle empty TextDecoder input when stream is true
    (#​31691)
  • fix(io): ignore BrokenPipe error in op_print
    (#​31844)
  • fix(node/child_process): ChildProcess constructor without args
    (#​31838)
  • fix(node/child_process): improve arg type validation for execFile and
    fork

(#​31829)

  • fix(node/child_process): reject null bytes in arguments
    (#​31828)
  • fix(node/child_process): use correct syscall name in spawn error
    (#​31836)
  • fix(npm): deduplication pass for initially partially resolved graphs
    (#​31787)
  • fix(runtime): handle options parameter in Deno.errors.*
    constructors

(#​31510)

  • fix(task): preserve signal-based exit codes for child processes
    (#​31789)
  • fix(tsgo): nil deref for exported symbols
    (#​31786)
  • fix(webgpu): correctly expose GPUCompilationInfo &
    GPUCompilationMessage

(#​31780)

  • fix: add more details when panicking getting dir path from deno.json
    (#​31805)
  • fix: allow multiple --allow-all args
    (#​31833)
  • fix: queue dynamic imports for modules with pending TLA
    (#​31670)
  • fix: update config schema for read permission to include ignore
    property

(#​31785)


Configuration

📅 Schedule: Branch creation - "every weekend" in timezone
America/Chicago, Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you
are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update
again.


  • If you want to rebase/retry this PR, check
    this box

This PR was generated by Mend Renovate.
View the repository job
log
.

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
- chore(deps): pin asdf-vm/actions action to b7bcd02 (#126)

This PR contains the following updates:

Package Type Update Change
asdf-vm/actions
action pinDigest b7bcd02

Configuration

📅 Schedule: Branch creation - "every weekend" in timezone
America/Chicago, Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you
are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update
again.


  • If you want to rebase/retry this PR, check
    this box

This PR was generated by Mend Renovate.
View the repository job
log
.

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

I have some failing tests with the new decaf script tests I wrote. but it's hard to know why because I can't see the script logs! I added this feature ability to the SDK and updated to that version in here.
Tests were passing on my development machine, but not on the CI. After looking into it, I found that when the test suite was running the deploy script, it was actually running the Dino JSR scripts, which had different behavior in the different environments.

We actually shouldn't be running those anyway; they are dependencies, and so they should be blocked. I mocked them to fix the problem.

In order to successfully mock these, I needed to update the mockabin version to a new version that contains a new feature where you can more conditionally mock binary areas. This is required because originally you could only mock all Deno commands, but that broke the ability for the test runner to even run the deploy script, because it is a Deno script. So now we can conditionally mock, which means we just mock when it's a JSR command.
the 'deno task test' command already generates the coverage report
turns out, it wasn't making it.
@coveralls
Copy link
Copy Markdown

coveralls commented Jan 17, 2026

Pull Request Test Coverage Report for Build 21095627508

Details

  • 44 of 48 (91.67%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.8%) to 81.597%

Changes Missing Coverage Covered Lines Changed/Added Lines %
steps/deploy.ts 37 41 90.24%
Totals Coverage Status
Change from base Build 21094458760: 0.8%
Covered Lines: 1334
Relevant Lines: 1667

💛 - Coveralls

@levibostian levibostian force-pushed the test-toolkit branch 5 times, most recently from 821e102 to 8eee6d4 Compare January 17, 2026 14:12
@levibostian levibostian merged commit 40e88ca into main Jan 17, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants