Skip to content

fix(x): return ErrClosed for abnormal closures#377

Merged
sbruens merged 4 commits into
mainfrom
sbruens/abnormal-closure
Mar 3, 2025
Merged

fix(x): return ErrClosed for abnormal closures#377
sbruens merged 4 commits into
mainfrom
sbruens/abnormal-closure

Conversation

@sbruens

@sbruens sbruens commented Feb 25, 2025

Copy link
Copy Markdown
Contributor

Errors returned from a Gorilla connection's NextReader() call require applications to break out of the application's read loop, as errors are permanent. Subsequent calls return the same error and after 1000 calls panic:

Feb 24 23:33:38.357 INF http: panic serving 127.0.0.1:57514: repeated read on failed websock
et connection                                                                               
goroutine 180 [running]:                                                                    
net/http.(*conn).serve.func1()                                                              
        /usr/lib/google-golang/src/net/http/server.go:1924 +0xd3                            
panic({0xa14340?, 0xb87160?})                                                               
        /usr/lib/google-golang/src/runtime/panic.go:799 +0x132                              
github.com/gorilla/websocket.(*Conn).NextReader(0xc0004b3600)                               
        /usr/local/google/home/sbruens/go/pkg/mod/github.com/gorilla/websocket@v1.5.3/conn.g
o:1030 +0x1fe                                                                               
github.com/Jigsaw-Code/outline-sdk/x/websocket.(*gorillaConn).Read(0xc000362d00, {0xc0003920
00, 0x10000, 0x10000})                                                                      
        /usr/local/google/home/sbruens/outline/outline-sdk/x/websocket/endpoint.go:157 +0xd1
github.com/Jigsaw-Code/outline-ss-server/service.(*associationHandler).HandleAssociation(0xc
00021ad80, {0xb8c620, 0xc00014f130}, {0x7f800f23c818, 0xc000368320}, {0xb8c8f8, 0xc000201320
})                                                                                          
        /usr/local/google/home/sbruens/outline/outline-ss-server/service/udp.go:162 +0x2da  
main.(*OutlineServer).runConfig.func1.2.8({0xb8bee8?, 0xc00029ca80?}, 0xc0000fb540)         
        /usr/local/google/home/sbruens/outline/outline-ss-server/cmd/outline-ss-server/main.
go:376 +0x2ba                                                                               
net/http.HandlerFunc.ServeHTTP(0xc0000fb540?, {0xb8bee8?, 0xc00029ca80?}, 0xab3720?)        
        /usr/lib/google-golang/src/net/http/server.go:2271 +0x29                            
main.(*OutlineServer).runConfig.func1.2.ProxyHeaders.23({0xb8bee8, 0xc00029ca80}, 0xc0000fb5
40)                                                                                         
        /usr/local/google/home/sbruens/go/pkg/mod/github.com/gorilla/handlers@v1.4.1/proxy_h
eaders.go:59 +0x143                                                                         
net/http.HandlerFunc.ServeHTTP(0x0?, {0xb8bee8?, 0xc00029ca80?}, 0xb?)                      
        /usr/lib/google-golang/src/net/http/server.go:2271 +0x29
main.(*OutlineServer).runConfig.func1.2.StripPrefix.24({0xb8bee8, 0xc00029ca80}, 0xc0000fb40
0)
        /usr/lib/google-golang/src/net/http/server.go:2333 +0x262
net/http.HandlerFunc.ServeHTTP(0xc00029c000?, {0xb8bee8?, 0xc00029ca80?}, 0x807856?)
        /usr/lib/google-golang/src/net/http/server.go:2271 +0x29
net/http.(*ServeMux).ServeHTTP(0x5ca179?, {0xb8bee8, 0xc00029ca80}, 0xc0000fb400)
        /usr/lib/google-golang/src/net/http/server.go:2801 +0x1c7
net/http.serverHandler.ServeHTTP({0xc000211f50?}, {0xb8bee8?, 0xc00029ca80?}, 0x6?)
        /usr/lib/google-golang/src/net/http/server.go:3281 +0x8e
net/http.(*conn).serve(0xc000373440, {0xb8c5e8, 0xc00011e3f0})
        /usr/lib/google-golang/src/net/http/server.go:2079 +0x625
created by net/http.(*Server).Serve in goroutine 44
        /usr/lib/google-golang/src/net/http/server.go:3436 +0x4bc

So we return net.ErrClosed error for all other close errors. This will ensure the UDP loop knows to exit as the connection has closed.

@sbruens sbruens requested a review from fortuna February 25, 2025 04:57
Comment thread x/websocket/endpoint.go Outdated
Comment thread x/websocket/endpoint.go Outdated
@sbruens sbruens force-pushed the sbruens/abnormal-closure branch from d5131c5 to f19b02e Compare February 26, 2025 15:03
@sbruens sbruens requested a review from fortuna February 26, 2025 15:04
Comment thread x/websocket/endpoint.go Outdated
Comment thread x/websocket/endpoint.go
if websocket.IsCloseError(err, websocket.CloseNormalClosure) {
return 0, io.EOF
}
return 0, fmt.Errorf("%w %w", net.ErrClosed, closeError)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good solution. We are encoding that Websockets is done in case of error.

Perhaps for another PR, we should still consider fixing the calling loop, so the same issue doesn't happen with future protocols. My thought is that we should break the loop on any error, except io. ErrShortBuffer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the breaking out of the calling loop in: OutlineFoundation/tunnel-server@9621558 Is that not what you mean?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I forgot about that. But I was also thinking about errors after the association is established.

The timeout should take care of that, but we may run into issues because Write will update the timeout regardless of the error:
https://github.com/Jigsaw-Code/outline-ss-server/blob/962155844aed338b15b5d322212e87102b00e096/service/udp.go#L456

So we may have a situation where the writes keep happening and failing, with the association broken, but kept alive.

I believe we should fix the code to only update the deadline on successful writes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. That makes sense. I don't think that would be a regression, I'm pretty sure we had that behavior before the refactor, but I agree it would be good to be more strict there. I'll make a note to address it.

Co-authored-by: Vinicius Fortuna <fortuna@users.noreply.github.com>
@sbruens sbruens merged commit 59218bd into main Mar 3, 2025
@sbruens sbruens deleted the sbruens/abnormal-closure branch March 3, 2025 19:44
appgood pushed a commit to appgood/outline-sdk that referenced this pull request Jul 25, 2025
* fix(x): return ErrClosed for abnormal closures

* Wrap error in `net.ErrClosed` instead of replacing it.

* Update x/websocket/endpoint.go

Co-authored-by: Vinicius Fortuna <fortuna@users.noreply.github.com>

---------

Co-authored-by: Vinicius Fortuna <fortuna@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants