Skip to content

WebSocket Client and V5 RemoteCommand Subprotocol#119157

Merged
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
seans3:websocket-executor
Sep 5, 2023
Merged

WebSocket Client and V5 RemoteCommand Subprotocol#119157
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
seans3:websocket-executor

Conversation

@seans3

@seans3 seans3 commented Jul 7, 2023

Copy link
Copy Markdown
Contributor
  • Implements WebSocket client as WebSocketExecutor. The executor is not yet called by any client.
  • Creates new version 5 of remote command subprotocol
    • RemoteCommand Version 5 adds a CLOSE signal, which is handled on the websocket server by closing the STDIN stream allowing other streams to complete transmission and close gracefully.
    • doc.go in wsstream package describes every version of binary WebSockets RemoteCommand protocols.
  • Adds unit tests for new WebSocket client functionality
    • Increases tools/remotecommand test coverage from 53.8% to 75.5%.
    • File k8s.io/client-go/tools/remotecommmand/websocket.go has test coverage: 88.3%
  ok  	k8s.io/client-go/tools/remotecommand	coverage: 75.5% of statements
  ok  	k8s.io/client-go/transport/websocket	coverage: 71.4% of statements

Example stress test for WebSocket client testing STDIN and STDOUT stream

This loopback test sends 1MB of random data onto the STDIN stream copying the data back to the client onto the STDOUT stream.

$ stress ./remotecommand.test -test.run TestWebSocketClient_LoopbackStdinToStdout
5s: 4964 runs so far, 0 failures
10s: 9952 runs so far, 0 failures
15s: 14929 runs so far, 0 failures
20s: 19879 runs so far, 0 failures
25s: 24866 runs so far, 0 failures
30s: 29862 runs so far, 0 failures
35s: 34858 runs so far, 0 failures
40s: 39869 runs so far, 0 failures
45s: 44853 runs so far, 0 failures
50s: 49852 runs so far, 0 failures
55s: 54858 runs so far, 0 failures
1m0s: 59861 runs so far, 0 failures

Example stress test for WebSocket client sending multiple write streams concurrently (STDIN, TTY Resize)

$ stress ./remotecommand.test -test.run TestWebSocketClient_MultipleWriteChannels
5s: 675 runs so far, 0 failures                                                                                                                                                                        
10s: 1351 runs so far, 0 failures                                                                                                                                                                      
15s: 2057 runs so far, 0 failures                                                                                                                                                                      
20s: 2742 runs so far, 0 failures                                                                                                                                                                      
25s: 3426 runs so far, 0 failures                                                                                                                                                                      
30s: 4126 runs so far, 0 failures                                                                                                                                                                      
35s: 4836 runs so far, 0 failures                                                                                                                                                                      
40s: 5534 runs so far, 0 failures                                                                                                                                                                      
45s: 6225 runs so far, 0 failures                                                                                                                                                                      
50s: 6917 runs so far, 0 failures                                                                                                                                                                      
55s: 7612 runs so far, 0 failures                                                                                                                                                                      
1m0s: 8300 runs so far, 0 failures                                                                                                                                                                     
1m5s: 9006 runs so far, 0 failures                                                                                                                                                                     
1m10s: 9681 runs so far, 0 failures                                                                                                                                                                    
1m15s: 10368 runs so far, 0 failures 

/kind feature

Helps fix: #89163

NONE

NOTE: WebSocket client is implemented with the Gorilla WebSockets library

  • The other two possibilities are:
    1. golang stdlib websocket - which is used in the current WebSocket server code
    2. https://pkg.go.dev/nhooyr.io/websocket
  • All three libraries are very mature.
  • The Gorilla WebSockets library was used for the client because:
    1. It has far better support for client creation. The gorilla library client creation functions fit well with Kubernetes use of RoundTripper, because they return an http response which is part of the RoundTripper interface when other libraries do not.
    2. It has far better support for data messages other than binary (e.g. heartbeat ping/pong).
    3. Gorilla has direct support for proxies in the client, while the golang stdlib websockets does not.
    4. Until recently, the golang stdlib websocket library suggested using the Gorilla library as more feature rich.

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/feature Categorizes issue or PR as related to a new feature. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 7, 2023
@seans3

seans3 commented Jul 7, 2023

Copy link
Copy Markdown
Contributor Author

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 7, 2023
@seans3

seans3 commented Jul 7, 2023

Copy link
Copy Markdown
Contributor Author

/priority important-soon

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. area/cloudprovider area/kubectl sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 7, 2023
@k8s-ci-robot k8s-ci-robot requested review from bart0sh and cheftako July 7, 2023 17:31
@seans3

seans3 commented Jul 7, 2023

Copy link
Copy Markdown
Contributor Author

/cc @aojea
/cc @ardaguclu

@k8s-ci-robot k8s-ci-robot requested review from aojea and ardaguclu July 7, 2023 17:39
@seans3 seans3 force-pushed the websocket-executor branch from ae80a42 to ec34bf5 Compare July 7, 2023 18:28
@seans3

seans3 commented Jul 7, 2023

Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@seans3

seans3 commented Jul 7, 2023

Copy link
Copy Markdown
Contributor Author

/retest

@seans3

seans3 commented Jul 7, 2023

Copy link
Copy Markdown
Contributor Author

/cc @jpbetz

@seans3

seans3 commented Aug 25, 2023

Copy link
Copy Markdown
Contributor Author

/retest

4 similar comments
@seans3

seans3 commented Aug 25, 2023

Copy link
Copy Markdown
Contributor Author

/retest

@seans3

seans3 commented Aug 28, 2023

Copy link
Copy Markdown
Contributor Author

/retest

@seans3

seans3 commented Aug 28, 2023

Copy link
Copy Markdown
Contributor Author

/retest

@seans3

seans3 commented Sep 1, 2023

Copy link
Copy Markdown
Contributor Author

/retest

@dex4er

dex4er commented Sep 1, 2023

Copy link
Copy Markdown

@jpbetz @cheftako @bart0sh Hello! Is there anything to do with this PR or it might be approved?

Comment thread staging/src/k8s.io/client-go/tools/remotecommand/websocket.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The duration d must be greater than zero; if not, NewTicker will panic. Stop the ticker to release associated resources.

https://pkg.go.dev/time#NewTicker

can the user disable the heartbeat?
can the user set it to zero?

@seans3 seans3 Sep 1, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The heartbeat ping period is a constant (5 seconds) and is not user-configurable. The period will never be zero.

       // pingPeriod defines how often a heartbeat "ping" message is sent.                                                                                                                            
        pingPeriod = 5 * time.Second 

The reason it is a field in the heartbeat struct is for testing purposes. It allows us to set significantly shorter ping periods for unit tests.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI #115493

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case this becomes configurable we should ensure zero disables the heartbeat

Comment thread staging/src/k8s.io/client-go/tools/remotecommand/websocket.go Outdated
@aojea

aojea commented Sep 1, 2023

Copy link
Copy Markdown
Member

LGTM

some questions that. may be follow ups if they make sense

@aojea

aojea commented Sep 2, 2023

Copy link
Copy Markdown
Member

/lgtm

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: e63541a9b1e44c41bd81732a42f56e4a220c19c1

@liggitt liggitt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve
/hold for one question about the websocket impl matching the spdy impl lack of thread safety

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the returned executor's Stream / StreamWithContext methods are not threadsafe, right? I think the spdy executor has the same issue (due to the coupled upgrader storing state during the connection setup) but can you verify that? would be good to explicitly document that here and on the spdy constructor, but it's fine if that goes into a follow up

@seans3 seans3 Sep 5, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. You are correct that neither the SPDY nor WebSocket Stream/StreamWithContext are thread-safe for a single executor. As you mention, the UpgradeRoundTripper for both store state (the dialed/created connection). SPDY already has some documentation related to this when defining the SpdyRoundTripper struct. Above the connection field is the following comment

type SpdyRoundTripper struct {
...
       /* TODO according to http://golang.org/pkg/net/http/#RoundTripper, a RoundTripper                                                                                                              
           must be safe for use by multiple concurrent goroutines. If this is absolutely                                                                                                               
           necessary, we could keep a map from http.Request to net.Conn. In practice,                                                                                                                  
           a client will create an http.Client, set the transport to a new insteace of                                                                                                                 
           SpdyRoundTripper, and use it a single time, so this hopefully won't be an issue.                                                                                                            
        */                                                                                                                                                                                             
        // conn is the underlying network connection to the remote server.                                                                                                                             
        conn net.Conn                                                                        

I have added documentation about the lack of thread safety to WebSocket.Stream/StreamWithContext

@seans3

seans3 commented Sep 5, 2023

Copy link
Copy Markdown
Contributor Author

/retest

unrelated required test failure

@aojea

aojea commented Sep 5, 2023

Copy link
Copy Markdown
Member

/lgtm

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 4cf19be17ea27119104412468df71489446462a9

@aojea

aojea commented Sep 5, 2023

Copy link
Copy Markdown
Member

/lgtm /approve /hold for one question about the websocket impl matching the spdy impl lack of thread safety

@liggitt it is on hold and the approve label was not added

@liggitt

liggitt commented Sep 5, 2023

Copy link
Copy Markdown
Member

/hold cancel
/lgtm
/approve

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, seans3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/cloudprovider area/kubectl area/stable-metrics Issues or PRs involving stable metrics cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Projects

Archived in project
Archived in project

Development

Successfully merging this pull request may close these issues.

Support kubectl exec/attach using Websocket