Add process dump functionality for WCOW/LCOW by dcantah · Pull Request #1062 · microsoft/hcsshim

dcantah · 2021-07-07T15:39:35Z

This commit adds support for generating process dumps for hypervisor isolated containers. This includes
a new annotation to specify where process dumps should get placed on creation, which is global
to all containers.

This was verified the following way through crictl:

Linux Pod spec
--------------
{
    "metadata": {
        "name": "sandbox-coredump",
        "namespace": "default",
        "attempt": 1
    },
    "annotations": {
        "io.microsoft.container.processdumplocation": "/coredumps/core.%s.%e.%P.%t" 
    }
}

Linux container spec 
-----------------
{
        "metadata": {
                "name": "coredump-test"
        },
        "image": {
                "image": "docker.io/library/ubuntu:latest"
        },
        "command": [
                "sh",
                "-c",
                "while true; do echo 'Hello, World!'; sleep 1; done"
        ],
        "mounts": [
                {
                        "container_path": "/coredumps",
                        "host_path": "sandbox:///core"
                },
                {
                         "container_path": "/host",
                         "host_path": "C:\\users\\dcanter\\desktop\\coredump"
                }
        ],
        "annotations": {
                "io.microsoft.lcow.rlimitcore": "18446744073709551615;18446744073709551615"
        },
        "linux": {}
}

Get a shell into the container and then grab the current pid (echo $$) and do a kill -6 <pid>. Re-exec in and there should be a file in /coredumps named something like "core.6.sh.176.1626339581". Then cp the file to /host and try out gdb on the file through wsl. gdb -c /mnt/c/Users/dcanter/desktop/coredump/core.6.sh.176.1626339581

Below is the Windows configs for this as well:

Windows Pod Spec:
--------------
{
    "metadata": {
        "name": "sandbox-wcow",
        "namespace": "default",
        "attempt": 1
    },
    "annotations": {
        "io.microsoft.container.processdumplocation": "C:\\crashdump"
    }
}

Windows container spec 
-----------------
{
        "metadata": {
                "name": "wcow-test"
        },
        "image": {
                "image": "cplatpublic.azurecr.io/crashdump:latest"
        },
        "command": [
                "C:\\app\\crashtest.exe",
                "so"
        ],
        "mounts": [{
                "container_path": "C:\\crashdump",
                "host_path": "C:\\users\\dcanter\\desktop\\coredump"
        }]
}

Afterwards there should a dmp at whatever the specified directory was. The container image is a slightly modified version of https://github.com/spreadex/win-docker-crash-dump. As we're setting wersvc to auto-start Stackoverflow exceptions work as well.

Signed-off-by: Daniel Canter dcanter@microsoft.com

internal/guest/runtime/hcsv2/spec.go

internal/hcsoci/hcsdoc_wcow.go

internal/uvm/create_wcow.go

internal/guest/runtime/hcsv2/container.go

anmaxvl

lgtm

internal/oci/annotations.go

test/cri-containerd/runpodsandbox_test.go

internal/uvm/create_wcow.go

test/cri-containerd/runpodsandbox_test.go

internal/oci/annotations.go

dcantah · 2021-09-07T14:28:11Z

@katiewasnothere Responded to feedback, ptal when you get another chance!

dcantah · 2021-09-14T17:09:40Z

@katiewasnothere Ping

cmd/gcs/main.go

test/cri-containerd/runpodsandbox_test.go

internal/hcsoci/hcsdoc_wcow.go

katiewasnothere · 2021-09-21T01:14:05Z

Once the windows sandbox mount PR is merged, we should add a test for it here and then I think we're good to merge.

This commit adds support for generating process dumps for hypervisor isolated containers. This includes a new annotation to specify where process dumps should get placed on creation, which is global to all containers. Signed-off-by: Daniel Canter <dcanter@microsoft.com>

This change adds cri-containerd testcases that excercises the process dump functionality. It sets up two containers, using images that stackoverflow/ throw an exception shortly after starting. This should generate a dump file in the sandbox mount location in the test. Then the second container mounts the same sandbox mount and just verify that the core dump file generated by the first is present. Signed-off-by: Daniel Canter <dcanter@microsoft.com>

dcantah · 2021-09-24T23:20:07Z

@katiewasnothere Checking in shortly now that the sandbox pr is in and a new test

Related work items: microsoft#1062, microsoft#1087, microsoft#1089, microsoft#1095, microsoft#1104, microsoft#1112, microsoft#1117, microsoft#1118, microsoft#1125, microsoft#1137, microsoft#1139, microsoft#1140, microsoft#1141, microsoft#1142, microsoft#1143, microsoft#1145, microsoft#1146, microsoft#1150, microsoft#1151, microsoft#1153, microsoft#1154, microsoft#1155, microsoft#1156, microsoft#1157, microsoft#1158, microsoft#1159, microsoft#1161, microsoft#1162, microsoft#1163, microsoft#1164, microsoft#1165, microsoft#1166, microsoft#1167, microsoft#1168, microsoft#1169, microsoft#1171, microsoft#1172, microsoft#1173, microsoft#1174, microsoft#1178

dcantah marked this pull request as ready for review July 13, 2021 16:58

dcantah requested a review from a team as a code owner July 13, 2021 16:58

dcantah force-pushed the coredump branch 3 times, most recently from 0b12030 to e242b1f Compare July 15, 2021 13:34

anmaxvl reviewed Jul 23, 2021

View reviewed changes

internal/guest/runtime/hcsv2/spec.go Outdated Show resolved Hide resolved

internal/hcsoci/hcsdoc_wcow.go Outdated Show resolved Hide resolved

internal/hcsoci/hcsdoc_wcow.go Show resolved Hide resolved

internal/uvm/create_wcow.go Show resolved Hide resolved

dcantah force-pushed the coredump branch from e242b1f to b75e941 Compare July 26, 2021 09:23

anmaxvl reviewed Jul 26, 2021

View reviewed changes

internal/guest/runtime/hcsv2/container.go Show resolved Hide resolved

anmaxvl approved these changes Jul 27, 2021

View reviewed changes

dcantah force-pushed the coredump branch from b75e941 to f68376b Compare July 27, 2021 07:44

dcantah force-pushed the coredump branch 2 times, most recently from 313641f to dd93f38 Compare August 17, 2021 16:13