Skip to content

Client return "context deadline exceeded" err with disable RoleAuth and the username and password are used #10430

@cfc4n

Description

@cfc4n

etcd version

3.23
ETCD Cluster doesn't enable Role Auth, and the username and password are used in client.

For example:

var Etcd_dsn []string = ["192.168.1.10:2379","192.168.1.11:2379","192.168.1.12:2379"]
var Etcd_username string = "root"
var Etcd_passwd string = "pass"
cfg := etcdclient.Config{
		Endpoints: Etcd_dsn,
		TLS:       _tlsConfig,
		DialTimeout:ETCD_CONNECT_TIMEOUT,
		AutoSyncInterval: time.Second * 300,
		Username: Etcd_username,
		Password: Etcd_passwd,
	}
client, err := clientv3.New(cfg)
if err != nil {
	return err
}

Two results:

  1. It will work well with less endpoints
  2. It will return context deadline exceed with many endpoints

The rootcause of BUG:

  1. When client code use username and password ,it will go to the code err := c.getToken(ctx) at client.go line 356.
  2. In getToken function, it will not stop trying another endpoint until sucessfull if get error after call the auth.authenticate.
  3. It will return rpctypes.ErrAuthNotEnabled after trying all endpoints if ETCD cluster don't use Auth.
  4. When with less endpoints, the ctx will not get timeout after trying all endpoints.
  5. When with many endpoints, the ctx will get timeout when after trying most endpoints. The error rpctypes.ErrAuthNotEnabled in the 311th line of client.go will turn to be context deadline exceeded
  6. In the line client.go 365, it will be ignored when the err is not rpctypes.ErrAuthNotEnabled when handling the result of c.getToken(ctx). It will not communicate normaly when get other error types.

We can get a conclusion that it will not function as well as we wish when the number of endpoints is not sure.

In my production , the ETCD cluster in the TEST environment has three endpoints which are deployed in different IDC , while it in PRODUCT environment has seven or nine endpoints which are deployed in different IDC. It will work well in the TEST but doesn't work in the PRODUCT when the code isn't changed.

The err rpctypes.ErrAuthNotEnabled can be ignored if the communication to ETCD works well just like in client.go 358 line .
I think it can return directly when the function getToken get err rpctypes.ErrAuthNotEnabled. It don't need to try other endpoints.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions