Skip to content

fix(tests): make test_modal_terminal actually assert, not return booleans#13318

Open
Sanjays2402 wants to merge 1 commit into
NousResearch:mainfrom
Sanjays2402:fix/modal-terminal-test-assertions
Open

fix(tests): make test_modal_terminal actually assert, not return booleans#13318
Sanjays2402 wants to merge 1 commit into
NousResearch:mainfrom
Sanjays2402:fix/modal-terminal-test-assertions

Conversation

@Sanjays2402

@Sanjays2402 Sanjays2402 commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Six of the tests in tests/integration/test_modal_terminal.py used:

success = <condition>
...
return success

Pytest treats a returned value as a warning, not a failure. Every one of these tests has been reported as PASSED regardless of whether Modal actually worked — the file was effectively a no-op under pytest, masking real Modal terminal regressions.

Fix:

  • Replace return success / return isolated / return requirements_met with assert <value>, <diagnostic message>. The diagnostic includes exit code and error text so a CI failure is actionable.
  • Replace the TERMINAL_ENV != modal → return False in test_modal_requirements() with pytest.skip(...), which is the correct signal when a precondition isn't met. The test is no longer misreported as PASSED when Modal isn't configured — it's cleanly skipped.
  • Keep the imperative main() runner usable (for the python tests/integration/test_modal_terminal.py usage the module docstring advertises). Rewrote it to tolerate AssertionError / pytest.skip.Exception and classify each test as PASSED / FAILED / SKIPPED in the summary.

Related Issue

Fixes #13286

Type of Change

  • ✅ Tests (adding or improving test coverage)

Changes Made

  • tests/integration/test_modal_terminal.py (+52 / −29): every test now ends with assert ..., <diagnostic>; test_modal_requirements uses pytest.skip(); main() rewritten to classify exceptions.
-    if config['env_type'] != 'modal':
-        print(...)
-        return False
+    if config['env_type'] != 'modal':
+        print(...)
+        pytest.skip(f"TERMINAL_ENV='{config['env_type']}', skipping Modal-specific test")
...
-    return success
+    assert success, f"Simple command failed: exit={result_json.get('exit_code')}, error={result_json.get('error')}"

How to Test

  1. pytest tests/integration/test_modal_terminal.py -q with TERMINAL_ENV unset → all tests reported SKIPPED (was: all PASSED silently).
  2. TERMINAL_ENV=modal pytest tests/integration/test_modal_terminal.py -q → real pass/fail based on Modal behavior.
  3. python tests/integration/test_modal_terminal.py → still works as a script runner with PASSED/FAILED/SKIPPED classification.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(tests):)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes — N/A (fixes existing tests to actually assert)
  • I've tested on my platform: macOS 26.5 (arm64)

Documentation & Housekeeping

  • Documentation updates — N/A
  • cli-config.yaml.example — N/A
  • CONTRIBUTING.md / AGENTS.md — N/A
  • Cross-platform impact considered — pytest/stdlib only
  • Tool descriptions/schemas — N/A

Not in scope

Screenshots / Logs

  • python3 -m py_compile tests/integration/test_modal_terminal.py → clean.
  • Walked the file top to bottom, confirmed all six test bodies now end with an assert carrying a descriptive message, and that test_modal_requirements() uses pytest.skip() instead of return False when TERMINAL_ENV != "modal".

…eans

Closes NousResearch#13286.

Six of the tests in tests/integration/test_modal_terminal.py used

    success = <condition>
    return success

Pytest treats the return value as a warning, not a failure, so every
one of these tests was reported as PASSED regardless of whether Modal
worked. The file was effectively a no-op under pytest, masking real
Modal terminal regressions.

Fix:

- Replace each 'return success / return isolated / return requirements_met'
  with an 'assert <value>, <diagnostic message>'. The diagnostic
  includes the exit code and error so a CI failure is actionable.

- Replace the 'TERMINAL_ENV != modal -> return False' in
  test_modal_requirements() with pytest.skip(), which is the correct
  signal when a precondition isn't met. The test is no longer
  misreported as PASSED when Modal isn't configured; it's cleanly
  skipped.

- Keep the existing imperative main() runner usable (for the 'python
  tests/integration/test_modal_terminal.py' usage documented in the
  module docstring). Rewrote it to tolerate AssertionError / pytest
  skip signals and classify each test as PASSED / FAILED / SKIPPED
  in the summary.
@alt-glitch alt-glitch added type/test Test coverage or test infrastructure P3 Low — cosmetic, nice to have backend/modal Modal.com cloud execution labels Apr 22, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #13325 — both convert test_modal_terminal return-based tests to assert-based. Fixes #13286.

@Sanjays2402

Copy link
Copy Markdown
Contributor Author

Hey @alt-glitch — small context note: this PR was opened ~6 minutes before #13325 (04:48:53Z vs 04:54:55Z), so the timeline runs the other direction here. Happy to defer to @sgaofen's implementation if maintainers prefer it — just wanted to flag the ordering. Whichever lands first is fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/modal Modal.com cloud execution P3 Low — cosmetic, nice to have type/test Test coverage or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: tests/integration/test_modal_terminal.py returns booleans so logical failures still pass under pytest

2 participants