Skip to content

feat(registry): Make graceful shutdown test robust#4720

Merged
joaodrp merged 1 commit into
distribution:mainfrom
Sumedhvats:fix-graceful-shutdown-4696
Oct 19, 2025
Merged

feat(registry): Make graceful shutdown test robust#4720
joaodrp merged 1 commit into
distribution:mainfrom
Sumedhvats:fix-graceful-shutdown-4696

Conversation

@Sumedhvats

Copy link
Copy Markdown
Contributor

Description

Fixes the TestGracefulShutdown test that has been failing with Go 1.25.

Closes #4696

Problem

The test was failing with unexpected EOF when run with Go 1.25. The root cause was that the test attempted to send an incomplete HTTP request by splitting the request line into two separate writes:

// Old (broken) approach
fmt.Fprintf(conn, "GET /v2/ ")              // Incomplete request line
registry.quit <- os.Interrupt                // Trigger shutdown
fmt.Fprintf(conn, "HTTP/1.1\r\nHost: ...\r\n\r\n")  // Try to complete it

This approach has several issues:

  1. Invalid HTTP protocol: The request line "GET /v2/ " is incomplete - it's missing the required HTTP version and CRLF terminator. A valid HTTP/1.1 request line must follow the format: METHOD SP REQUEST-URI SP HTTP-VERSION CRLF

  2. Undefined behavior: The server cannot distinguish between:

    • A slow client still typing the request
    • A malformed/incomplete request
    • A network issue
  3. Go 1.25 stricter behavior: Go 1.25 improved graceful shutdown to immediately close connections with incomplete or invalid requests, rather than waiting indefinitely for them to complete.

Solution

Updated the test to send a complete, valid HTTP/1.1 request before triggering graceful shutdown:

// New (correct) approach
conn, err := net.Dial("tcp", "localhost:5000")
fmt.Fprintf(conn, "GET /v2/ HTTP/1.1\r\nHost: localhost:5000\r\n\r\n")  // Complete request
time.Sleep(100 * time.Millisecond)  // Let server receive it
registry.quit <- os.Interrupt        // Trigger shutdown

This properly tests graceful shutdown behavior:

  • Existing valid requests complete successfully during shutdown
  • New connection attempts are rejected during shutdown
  • Server respects the drain timeout period

Changes Made

  • Send complete HTTP/1.1 request before shutdown signal
  • Improved error messages for better debugging
  • Added defer conn.Close() for proper resource cleanup
  • Added defer conn2.Close() when second connection test fails
  • Better variable initialization (errchan := make(chan error, 1))

The `TestGracefulShutdown` test was failing intermittently, especially
with stricter HTTP handling in newer Go versions (e.g., 1.25). This was
caused by sending an incomplete HTTP request in two separate writes,
creating a race condition where the server could shut down before
receiving the full request.

This commit fixes the test's flakiness by sending a single, complete,
and valid HTTP/1.1 request before triggering the shutdown. This ensures
the test accurately verifies the intended behavior: that a valid,
in-flight request is fully processed while new connections are rejected.

Fixes:distribution#4696
Signed-off-by: Sumedh Vats <sumedhvats2004@gmail.com>
@joaodrp joaodrp merged commit 4d613f7 into distribution:main Oct 19, 2025
40 checks passed
@Sumedhvats Sumedhvats deleted the fix-graceful-shutdown-4696 branch October 19, 2025 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test failure with Go 1.25

3 participants