Skip to content

test failure: "Failed to find a type oid for ..." #1219

@davepacheco

Description

@davepacheco

While trying to repro #1130, I ran into a failure I hadn't seen before and can't find a ticket for:

running 1 test
test integration_tests::disks::test_disk_create_disk_that_already_exists_fails ... FAILED

failures:

---- integration_tests::disks::test_disk_create_disk_that_already_exists_fails stdout ----
log file: "/dangerzone/omicron_tmp/try_repro.13782/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.17780.0.log"
note: configured to log to "/dangerzone/omicron_tmp/try_repro.13782/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.17780.0.log"
thread 'integration_tests::disks::test_disk_create_disk_that_already_exists_fails' panicked at 'Failed to notify Nexus about new Dataset: Error Response', /home/dap/omicron/sled-agent/src/sim/storage.rs:240:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: /dangerzone/omicron_tmp/try_repro.13782/.tmpSGA0LA
WARN: dropped ClickHouseInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)


failures:
    integration_tests::disks::test_disk_create_disk_that_already_exists_fails

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 74 filtered out; finished in 5.93s

Log file:

$ bunyan "/dangerzone/omicron_tmp/try_repro.13782/test_all-d586ea57740e3382-test_disk_create_disk_that_already_exists_fails.17780.0.log"
[2022-06-15T22:22:04.900085563Z]  INFO: test_disk_create_disk_that_already_exists_fails/17780 on ivanova: cockroach temporary directory: /dangerzone/omicron_tmp/try_repro.13782/.tmpSGA0LA
[2022-06-15T22:22:04.900451226Z]  INFO: test_disk_create_disk_that_already_exists_fails/17780 on ivanova: cockroach: copying from seed directory (/home/dap/omicron/target/debug/build/nexus-test-utils-308521ed0d0eed98/out/crdb-base) to storage directory (/dangerzone/omicron_tmp/try_repro.13782/.tmpSGA0LA/data)
[2022-06-15T22:22:04.907769128Z]  INFO: test_disk_create_disk_that_already_exists_fails/17780 on ivanova: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/dangerzone/omicron_tmp/try_repro.13782/.tmpSGA0LA/data,ballast-size=0 --listen-addr 127.0.0.1:0 --listening-url-file /dangerzone/omicron_tmp/try_repro.13782/.tmpSGA0LA/listen-url
[2022-06-15T22:22:06.627376916Z]  INFO: test_disk_create_disk_that_already_exists_fails/17780 on ivanova: cockroach pid: 17785
[2022-06-15T22:22:06.628304134Z]  INFO: test_disk_create_disk_that_already_exists_fails/17780 on ivanova: cockroach listen URL: postgresql://root@127.0.0.1:64766/omicron?sslmode=disable
...
[2022-06-15T22:22:10.628406846Z]  INFO: test_disk_create_disk_that_already_exists_fails/omicron_sled_agent::sim::Server/17780 on ivanova: Created Simulated Crucible Server (kind=storage, server=b6d65341-167c-41df-9b5c-41cded99c229, sled_id=b6d65341-167c-41df-9b5c-41cded99c229, address=[::1]:61800)
[2022-06-15T22:22:10.628731621Z] DEBUG: test_disk_create_disk_that_already_exists_fails/omicron_sled_agent::sim::Server/17780 on ivanova: client request (sled_id=b6d65341-167c-41df-9b5c-41cded99c229, body=Some(Body), method=PUT)
    uri: http://127.0.0.1:36197/zpools/2cfecc89-e67a-4b10-b50c-ec9c130187c2/dataset/7b3fdf75-02ea-43d8-a29e-5f6e8d590a8c
[2022-06-15T22:22:10.629066446Z] TRACE: 5eb2887c-0c0e-4a74-87fc-60a737cec391/dropshot_internal/17780 on ivanova: incoming request (req_id=65465297-194a-47b9-97a7-a49519f3fb06, method=PUT, remote_addr=127.0.0.1:52635, local_addr=127.0.0.1:36197)
    uri: /zpools/2cfecc89-e67a-4b10-b50c-ec9c130187c2/dataset/7b3fdf75-02ea-43d8-a29e-5f6e8d590a8c
[2022-06-15T22:22:10.629428382Z]  INFO: 5eb2887c-0c0e-4a74-87fc-60a737cec391/ServerContext/17780 on ivanova: upserting dataset (address=[::1]:61800, dataset_id=7b3fdf75-02ea-43d8-a29e-5f6e8d590a8c, zpool_id=2cfecc89-e67a-4b10-b50c-ec9c130187c2)
[2022-06-15T22:22:10.792935278Z]  INFO: 5eb2887c-0c0e-4a74-87fc-60a737cec391/dropshot_internal/17780 on ivanova: request completed (req_id=65465297-194a-47b9-97a7-a49519f3fb06, method=PUT, remote_addr=127.0.0.1:52635, local_addr=127.0.0.1:36197, error_message_external="Internal Server Error", response_code=500)
    uri: /zpools/2cfecc89-e67a-4b10-b50c-ec9c130187c2/dataset/7b3fdf75-02ea-43d8-a29e-5f6e8d590a8c
    --
    error_message_internal: Unknown diesel error creating Dataset called "7b3fdf75-02ea-43d8-a29e-5f6e8d590a8c": Failed to find a type oid for `dataset_kind`
[2022-06-15T22:22:10.793844962Z] DEBUG: test_disk_create_disk_that_already_exists_fails/omicron_sled_agent::sim::Server/17780 on ivanova: client response (sled_id=b6d65341-167c-41df-9b5c-41cded99c229)
    result: Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36197), path: "/zpools/2cfecc89-e67a-4b10-b50c-ec9c130187c2/dataset/7b3fdf75-02ea-43d8-a29e-5f6e8d590a8c", query: None, fragment: None }, status: 500, headers: {"content-type": "application/json", "x-request-id": "65465297-194a-47b9-97a7-a49519f3fb06", "content-length": "124", "date": "Wed, 15 Jun 2022 22:22:10 GMT"} })

Metadata

Metadata

Assignees

No one assigned

    Labels

    Test FlakeTests that work. Wait, no. Actually yes. Hang on. Something is broken.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions