Skip to content

ccl/sqlproxyccl: include DRAINING pods in the directory cache#79368

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
jaylim-crl:jay/220404-cache-draining-pods
Apr 5, 2022
Merged

ccl/sqlproxyccl: include DRAINING pods in the directory cache#79368
craig[bot] merged 1 commit intocockroachdb:masterfrom
jaylim-crl:jay/220404-cache-draining-pods

Conversation

@jaylim-crl
Copy link
Copy Markdown
Contributor

@jaylim-crl jaylim-crl commented Apr 4, 2022

Previously, #67452 removed DRAINING pods from the directory cache. This commit
adds that back. The connector will now need to filter for RUNNING pods manually
before invoking the balancer. This is needed so that we could track DRAINING
pods, and wait until 60 seconds has elapsed before transferring connections
away from them. To support that, we also update the Pod's proto definition to
include a StateTimestamp field to reprevent that timestamp that the state field
was last updated.

The plan is to have a polling mechanism every X seconds to check DRAINING pods,
and use that information to start migrating connections.

Release note: None

Jira issue: CRDB-14759

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@jaylim-crl jaylim-crl marked this pull request as ready for review April 4, 2022 19:38
@jaylim-crl jaylim-crl requested review from a team as code owners April 4, 2022 19:38
@jaylim-crl jaylim-crl requested review from andy-kimball and jeffswenson and removed request for a team April 4, 2022 19:39
Copy link
Copy Markdown
Contributor

@andy-kimball andy-kimball left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @jaylim-crl and @jeffswenson)


pkg/ccl/sqlproxyccl/tenant/directory.proto, line 48 at r1 (raw file):

  // TenantID is the tenant that owns the pod.
  uint64 tenant_id = 2 [(gogoproto.customname) = "TenantID"];
  // addr is the ip and port combination identifying the tenant pod, (e.g.

NIT: The first letter of these comments should be capitalized, b/c this comment gets written to the generated Go file, where the field is capitalized. Our convention is to make the comment match the generated Go definition, not the protobuf definition.


pkg/ccl/sqlproxyccl/tenant/directory_cache.go, line 197 at r1 (raw file):

	// Trigger resumption if there are no RUNNING pods.
	runningPods := make([]*Pod, 0, len(tenantPods))

You're not actually using runningPods here. Instead, you can iterate through the tenantPods list and keep a foundRunning boolean that you set if you find a RUNNING pod.

Previously, cockroachdb#67452 removed DRAINING pods from the directory cache. This commit
adds that back. The connector will now need to filter for RUNNING pods manually
before invoking the balancer. This is needed so that we could track DRAINING
pods, and wait until 60 seconds has elapsed before transferring connections
away from them. To support that, we also update the Pod's proto definition to
include a StateTimestamp field to reprevent that timestamp that the state field
was last updated.

The plan is to have a polling mechanism every X seconds to check DRAINING pods,
and use that information to start migrating connections.

Release note: None
@jaylim-crl jaylim-crl force-pushed the jay/220404-cache-draining-pods branch from 5c78e85 to 7586717 Compare April 5, 2022 05:07
@jaylim-crl jaylim-crl requested a review from andy-kimball April 5, 2022 05:08
Copy link
Copy Markdown
Contributor Author

@jaylim-crl jaylim-crl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR!

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andy-kimball and @jeffswenson)


pkg/ccl/sqlproxyccl/tenant/directory.proto, line 48 at r1 (raw file):

Previously, andy-kimball (Andy Kimball) wrote…

NIT: The first letter of these comments should be capitalized, b/c this comment gets written to the generated Go file, where the field is capitalized. Our convention is to make the comment match the generated Go definition, not the protobuf definition.

Done.


pkg/ccl/sqlproxyccl/tenant/directory_cache.go, line 197 at r1 (raw file):

Previously, andy-kimball (Andy Kimball) wrote…

You're not actually using runningPods here. Instead, you can iterate through the tenantPods list and keep a foundRunning boolean that you set if you find a RUNNING pod.

Whoops - done.

Copy link
Copy Markdown
Collaborator

@jeffswenson jeffswenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jaylim-crl
Copy link
Copy Markdown
Contributor Author

TFTR!

bors r=JeffSwenson

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Apr 5, 2022

Already running a review

@craig craig bot merged commit 985344a into cockroachdb:master Apr 5, 2022
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Apr 5, 2022

Build succeeded:

@jaylim-crl jaylim-crl deleted the jay/220404-cache-draining-pods branch November 30, 2022 16:55
@jaylim-crl jaylim-crl restored the jay/220404-cache-draining-pods branch November 30, 2022 16:55
@jaylim-crl jaylim-crl deleted the jay/220404-cache-draining-pods branch November 30, 2022 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants