Skip to content

fix:userattr-unmount unexpected timeout#7008

Merged
fuweid merged 1 commit intocontainerd:mainfrom
mmmmmmmxl:fix/userattr-unmount
Jun 10, 2022
Merged

fix:userattr-unmount unexpected timeout#7008
fuweid merged 1 commit intocontainerd:mainfrom
mmmmmmmxl:fix/userattr-unmount

Conversation

@mmmmmmmxl
Copy link
Copy Markdown
Contributor

@mmmmmmmxl mmmmmmmxl commented Jun 1, 2022

I am trying to run containerd in docker container environment with user-namespace.

I found that the Unmount logic of this code usually takes tens of seconds when the machine is under high disk load and running multiple containers.And this will cause the startup of docker to fail

I'm not particularly sure what's causing this, but the unmount error just outputs the log and doesn't affect the execution logic of the function. So I think it would be better to change this to asynchronous execution.

This is the log of docker

time="2022-05-31T18:13:34.490264526+08:00" level=info msg="Starting up"
time="2022-05-31T18:13:34.491918527+08:00" level=debug msg="Listener created for HTTP on unix (/var/run/docker.sock)"
time="2022-05-31T18:13:34.491948011+08:00" level=debug msg="Containerd not running, starting daemon managed containerd"
time="2022-05-31T18:13:34.492139165+08:00" level=info msg="libcontainerd: containerd is still running" pid=144
time="2022-05-31T18:13:34.492210488+08:00" level=info msg="parsed scheme: "unix"" module=grpc
time="2022-05-31T18:13:34.492302486+08:00" level=info msg="scheme "unix" not registered, fallback to default scheme" module=grpc
time="2022-05-31T18:13:34.492376002+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc
time="2022-05-31T18:13:34.492407392+08:00" level=info msg="ClientConn switching balancer to "pick_first"" module=grpc
failed to start containerd: timeout waiting for containerd to start

This is the log of containerd

time="2022-05-31T18:13:18+08:00" level=warning msg="deprecated version : 1, please switch to version 2"
time="2022-05-31T18:13:18.171464520+08:00" level=info msg="starting containerd" revision=3df54a852345ae127d1fa3092b95168e4a88e2f8 version=1.5.11
time="2022-05-31T18:13:18.203640835+08:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
time="2022-05-31T18:13:18.203784689+08:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:18.203972923+08:00" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.aufs"..." error="aufs is not supported (modprobe aufs failed: exec: "modprobe": executable file not found in $PATH ""): skip plugin" type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:18.204007046+08:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:18.204351797+08:00" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.btrfs"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:18.204377535+08:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.devmapper"..." type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:18.204407926+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-05-31T18:13:18.204420177+08:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:18.204516354+08:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:41.004910809+08:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:41.005450480+08:00" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-05-31T18:13:41.005487845+08:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
time="2022-05-31T18:13:41.005592205+08:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-05-31T18:13:41.005614328+08:00" level=info msg="metadata content store policy set" policy=shared
time="2022-05-31T18:13:41.025727901+08:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
time="2022-05-31T18:13:41.025789547+08:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
time="2022-05-31T18:13:41.025861278+08:00" level=info msg="loading plugin "io.containerd.service.v1.introspection-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.025936748+08:00" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.025965102+08:00" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.025990351+08:00" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.026016690+08:00" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.026041617+08:00" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.026068688+08:00" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.026107338+08:00" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.026133823+08:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
time="2022-05-31T18:13:41.026410620+08:00" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
time="2022-05-31T18:13:41.026647883+08:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
time="2022-05-31T18:13:41.027386842+08:00" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
time="2022-05-31T18:13:41.027435146+08:00" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
time="2022-05-31T18:13:41.027523152+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027551955+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027583857+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027608009+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027631601+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027656475+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027690787+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027715415+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.027738980+08:00" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
time="2022-05-31T18:13:41.028128713+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.028186241+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.028227378+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.028251473+08:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
time="2022-05-31T18:13:41.029709381+08:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
time="2022-05-31T18:13:41.029854566+08:00" level=info msg=serving... address=/run/containerd/containerd.sock
time="2022-05-31T18:13:41.029899526+08:00" level=debug msg="sd notification" error="" notified=false state="READY=1"
time="2022-05-31T18:13:41.029925993+08:00" level=info msg="containerd successfully booted in 22.860007s"
time="2022-05-31T18:13:41.127070200+08:00" level=debug msg="garbage collected" d="892.627s"

@k8s-ci-robot
Copy link
Copy Markdown

Hi @mmmmmmmxl. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dmcgowan
Copy link
Copy Markdown
Member

dmcgowan commented Jun 1, 2022

I suggest filing this as an issue first. There are likely alternative solutions to solve the issue. We tend not to rely on using asynchronous calls in this way to cleaning up system resources such as mounts.

@fuweid
Copy link
Copy Markdown
Member

fuweid commented Jun 2, 2022

Agreed. @mmmmmmmxl you can try to mount with readonly. It won't sync when umount

@mmmmmmmxl
Copy link
Copy Markdown
Contributor Author

@fuweid Thank you for your suggestion. After I added readonly attribute to mount, in my case the problem was solved.

@fuweid
Copy link
Copy Markdown
Member

fuweid commented Jun 2, 2022

@mmmmmmmxl please squash your commits into one and sign-off with your real name like (git commit -s). Thanks

Signed-off-by: Wei Fu <fuweid89@gmail.com>

@mmmmmmmxl mmmmmmmxl force-pushed the fix/userattr-unmount branch 3 times, most recently from cb52f00 to 720e31d Compare June 7, 2022 14:54
@mmmmmmmxl
Copy link
Copy Markdown
Contributor Author

@fuweid ok, I had squashed my commits into one and sign-off with my real name.

@fuweid
Copy link
Copy Markdown
Member

fuweid commented Jun 8, 2022

@fuweid ok, I had squashed my commits into one and sign-off with my real name.

hi, I didn't see your sign-off 😂 could you please use git commit -s -amend and git show check sign off and then repush again? Thanks

@mmmmmmmxl mmmmmmmxl force-pushed the fix/userattr-unmount branch from 720e31d to 15c0b1b Compare June 8, 2022 03:32
Signed-off-by: Xinlin Ma <forblue.lin@gmail.com>
@mmmmmmmxl mmmmmmmxl force-pushed the fix/userattr-unmount branch from 15c0b1b to fa4720f Compare June 8, 2022 03:34
@mmmmmmmxl
Copy link
Copy Markdown
Contributor Author

@fuweid ok, I had squashed my commits into one and sign-off with my real name.

hi, I didn't see your sign-off 😂 could you please use git commit -s -amend and git show check sign off and then repush again? Thanks

I had pushed again.

Copy link
Copy Markdown
Member

@fuweid fuweid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

NOTE: The approach is the same to #6478

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants