Skip to content

[dnm] roachprod: start crdb in a container#61901

Closed
tbg wants to merge 1 commit intocockroachdb:masterfrom
tbg:poc-docker
Closed

[dnm] roachprod: start crdb in a container#61901
tbg wants to merge 1 commit intocockroachdb:masterfrom
tbg:poc-docker

Conversation

@tbg
Copy link
Copy Markdown
Member

@tbg tbg commented Mar 12, 2021

This isn't something I am planning to finish, but I wanted to see if I
would immediately run into any snags when trying to get roachprod start to spin up a Docker container instead of running bare-metal.

The motivation is twofold:

  • I'm generally interested in how we involve roachprod over time.
    Specifically, the question is whether in the long run systems testing
    should be done in k8s and which, if any, stepping stones there are to
    evolve what we currently do into that direction.
  • The current bare-metal setup goes unresponsive when CRDB goes into
    overdrive, and tests fail in the most opaque ways. We then incur a
    large tax for debugging these situations since we can't access the
    cluster in that state.

Work that would need to be done to really finish this:

  • maintain ubuntu images that come with docker installed (right now
    needs to be manually set up) and the ubuntu user set up and the images
    we need cached
  • we hit the old problem of having to pass the uid/gid to the container
    to avoid creating files as root, I hacked around it by hard-coding
    them
  • Setting the cluster settings, etc, is all done via the ./cockroach
    binary but that is no longer a thing. Ideally that should use SQL
  • The whole premise of uploading binaries is out of the window, we
    need to deal in CRDB containers exclusively, though we could
    conceivably use a wrapper container running the uploaded binary
    if we wanted to retain how roachprod/test work.

cc @irfansharif

Release note: None

This isn't something I am planning to finish, but I wanted to see if I
would immediately run into any snags when trying to get `roachprod
start` to spin up a Docker container instead of running bare-metal.

The motivation is twofold:

- I'm generally interested in how we involve roachprod over time.
  Specifically, the question is whether in the long run systems testing
  should be done in k8s and which, if any, stepping stones there are to
  evolve what we currently do into that direction.
- The current bare-metal setup goes [unresponsive] when CRDB goes into
  overdrive, and tests fail in the most opaque ways. We then incur a
  large tax for debugging these situations since we can't access the
  cluster in that state.

Work that would need to be done to really finish this:

- maintain ubuntu images that come with docker installed (right now
  needs to be manually set up) and the `ubuntu` user set up and the images
  we need cached
- we hit the old problem of having to pass the uid/gid to the container
  to avoid creating files as root, I hacked around it by hard-coding
  them
- Setting the cluster settings, etc, is all done via the `./cockroach`
  binary but that is no longer a thing. Ideally that should use SQL
- The whole premise of uploading binaries is out of the window, we
  need to deal in CRDB containers exclusively, though we could
  conceivably use a wrapper container running the uploaded binary
  if we wanted to retain how roachprod/test work.

[unresponsive]: cockroachdb#59424 (comment)

Release note: None
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants