Skip to content

VCH deletion fails with CDE when the VCH has many container VMs #7020

Open
Open
VCH deletion fails with CDE when the VCH has many container VMs#7020
Labels
component/installkind/defectBehavior that is inconsistent with what's intendedpriority/p2status/needs-estimationThe issue needs to be estimated by the teamteam/lifecycle
@lgayatri

Description

@lgayatri
lgayatri
opened on Jan 3, 2018

Build: https://storage.cloud.google.com/vic-product-ova-releases/vic-v1.3.0-rc5-2996-f8cc7317.ova

Steps:

  1. Created 50 VCHs with 53 containers each.
  2. While the containers are running, I deleted all 50 VCHs with 20 minute timeout and --force option
  3. Delete failed with
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/vsphere/tasks.WaitForResult: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:38:58-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
2018/01/03 00:39:16 markDiskAsDeletable([CPBU_2TB_01] 32be64595b22a953e4576cd590dd6f5e14a533bfc0f6da70b94f16917eb4ee05/32be64595b22a953e4576cd590dd6f5e14a533bfc0f6da70b94f16917eb4ee05.vmdk): %!s(<nil>)
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="Failed to destroy VM \"VirtualMachine:vm-51037\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52377\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52792\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-51205\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-51937\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52324\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nFailed to destroy VM \"VirtualMachine:vm-52061\": context deadline exceeded then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=warning msg="Proceeding with delete of VCH due to --force"
time="2018-01-03T00:39:16-05:00" level=info msg="Removing image stores"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=info msg="Removing volume stores"
time="2018-01-03T00:39:16-05:00" level=info msg="Removing appliance VM network devices"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="Failed to get vm power status \"VirtualMachine:vm-50712\": Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=info msg="Collecting e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="Failed to collect e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=warning msg="No log data for e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log"
time="2018-01-03T00:39:16-05:00" level=error msg="Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded\nPost https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/cmd/vic-machine/delete.(*Uninstall).Run.func2: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/cmd/vic-machine/delete.(*Uninstall).Run.func2: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"
time="2018-01-03T00:39:16-05:00" level=error msg=--------------------
time="2018-01-03T00:39:16-05:00" level=error msg="vic-machine-linux delete failed: Delete timed out: use --timeout to add more time\n"
time="2018-01-03T00:39:16-05:00" level=error msg="vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded\nvic/cmd/vic-machine/delete.(*Uninstall).Run:92 vic-machine-linux\nvic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux"

Operations were taking longer than 20 minutes to complete, hence the above error. Attached vi-machine.log.

vic-machine.zip

Few snippets from log:

Line 121128: Jan  3 2018 00:39:23.206-05:00 ERROR op=1374.1: Failed to collect e53fcd49-9375-46f0-9370-1da83c764012 vpxd.log: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded
...
Line 121171: Jan  3 2018 00:39:25.559-05:00 ERROR op=1236.1: Failed to destroy VM "VirtualMachine:vm-51654": ServerFaultCode: The method is disabled by 'VIC' then failed to unregister VM: Post https://wdc-stvc3944-146-dhcp15.eng.vmware.com/sdk: context deadline exceeded
...
Line 121258: Jan  3 2018 00:39:26.670-05:00 ERROR op=1365.1: vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded

@mdubya66 @cgtexmex - Not a blocker for 1.3 as retry helps delete. Can we document this?

Metadata

Metadata

Assignees

No one assigned

    Labels

    component/installkind/defectBehavior that is inconsistent with what's intendedpriority/p2status/needs-estimationThe issue needs to be estimated by the teamteam/lifecycle

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions