From the containerd logs from this CI run, which generated these logs artifact

[INFO] starting instance: 665173b26c87393303cfdd55a68427afc5025f083f2e6a56916d11005c02c5a1
[INFO] about to build container instance: 665173b26c87393303cfdd55a68427afc5025f083f2e6a56916d11005c02c5a1
[INFO] about to REALLY build the container instance: 665173b26c87393303cfdd55a68427afc5025f083f2e6a56916d11005c02c5a1
[INFO] cgroup manager V2 will be used
[WARN] Controller rdma is not yet implemented.
[WARN] Controller misc is not yet implemented.
[time="2023-10-03T21:33:53.554067718Z" level=info msg="StartContainer for \"33e8e760bae0ce1ed34b80d16e8e0cf187b51a641a82421fc1379a8a59526365\" returns successfully"
WARN] Controller rdma is not yet implemented.
[WARN] Controller misc is not yet implemented.
[INFO] starting instance: 25633cc2ec2915d7ab31407f5312d86a271498e784f25cd9268b8d300bf76f56
[INFO] about to build container instance: 25633cc2ec2915d7ab31407f5312d86a271498e784f25cd9268b8d300bf76f56
[INFO] about to REALLY build the container instance: 25633cc2ec2915d7ab31407f5312d86a271498e784f25cd9268b8d300bf76f56
[INFO] cgroup manager V2 will be used
[WARN] Controller rdma is not yet implemented.
[WARN] Controller misc is not yet implemented.
[WARN] Controller rdma is not yet implemented.
[WARN] Controller misc is not yet implemented.
[INFO] starting instance: c2f10adebbddfc541ac0eefbeb7382b428f9bd8c87af4fd675cbf49e7cd3bc16
[INFO] about to build container instance: c2f10adebbddfc541ac0eefbeb7382b428f9bd8c87af4fd675cbf49e7cd3bc16
[INFO] about to REALLY build the container instance: c2f10adebbddfc541ac0eefbeb7382b428f9bd8c87af4fd675cbf49e7cd3bc16
[INFO] cgroup manager V2 will be used
[WARN] Controller rdma is not yet implemented.
[WARN] Controller misc is not yet implemented.
[WARN] Controller rdma is not yet implemented.
[WARN] Controller misc is not yet implemented.
[INFO] XXXXXXXX validating container
[INFO] XXXXXXXX container executor get_or_init
[INFO] XXXXXXXX a linux container!
[WARN] intermediate process already reaped
[INFO] successfully built container instance: c2f10adebbddfc541ac0eefbeb7382b428f9bd8c87af4fd675cbf49e7cd3bc16
[INFO] container instance: c2f10adebbddfc541ac0eefbeb7382b428f9bd8c87af4fd675cbf49e7cd3bc16 has pid 5773
[INFO] XXXXXXXX executing container
[INFO] executing linux container
time="2023-10-03T21:33:53.864393966Z" level=info msg="StartContainer for \"c2f10adebbddfc541ac0eefbeb7382b428f9bd8c87af4fd675cbf49e7cd3bc16\" returns successfully"
[INFO] XXXXXXXX validating container
[INFO] XXXXXXXX container executor get_or_init
[INFO] XXXXXXXX a linux container!
[WARN] intermediate process already reaped
[INFO] successfully built container instance: 25633cc2ec2915d7ab31407f5312d86a271498e784f25cd9268b8d300bf76f56
[INFO] container instance: 25633cc2ec2915d7ab31407f5312d86a271498e784f25cd9268b8d300bf76f56 has pid 5758
[INFO] XXXXXXXX executing container
[INFO] executing linux container

There are 3 starting instance: ... corresponding to the 3 nginx linux containers.
There are 3 [INFO] about to REALLY build the container instance: ..., which is printed here just before calling build() on the libcontianer's ContainerBuilder.
The call to build() should call the validate(...) method in the executor here, however, thre are only 2 lines with XXXXXXXX validating container.

This means that we are getting stuck somewhere inside of the build() call.
@utam0k @yihuaf , do you have any clue on where in libcontainer we could be getting stuck?
Thanks!

Sporadic failures in E2E CI #347

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions