/test

Hi~ @grahamwhaley
It is only my guessing. i'm sorry.😰
this error is pretty rare. It always comes with a hanging apt-get process.
And not every time the error shares the same info. I have found a wiki about classifying and fixing those issues. you may help yourself to have a look.
I'm not sure if it is the same thing you talked about. totally confused.

@Pennyzct - probably before we start killing processes and nuking lock files then, we should diagnose what the actual problem is, and then work out if the solution is to, say, disable the auto updaters etc.

I would start by maybe adding some diagnostics in the error case, so try to capture the apt fail (in a TRAP function maybe), and then add something like

$ ps -ef | fgrep apt
$ sudo lsof /var/lib/apt/lists/
$ sudo lsof /var/lib/dpkg/
$ sudo /var/cache/apt/archives/

I do suspect maybe your server has the auto update services running in the background, and occasionally they will be active when you try to process a PR, and they clash on the lock file. We have seen that before (both on the bare metal metrics machines and on the cloud instances iirc) - so, if you have not already deliberately disabled the auto updaters - then do that first...

Hi~ @grahamwhaley thanks three thousand times!!!!! 😍 It is the auto updates and auto upgrades running in the background and it clashed with the manually apt.

root@arm-testing-1:~# systemctl status apt-daily.timer
● apt-daily.timer - Daily apt download activities
   Loaded: loaded (/lib/systemd/system/apt-daily.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Thu 2019-06-06 04:08:18 UTC; 2 weeks 0 days ago
  Trigger: Fri 2019-06-21 17:14:35 UTC; 15h left

root@arm-testing-1:~# systemctl status apt-daily-upgrade.timer
● apt-daily-upgrade.timer - Daily apt upgrade and clean activities
   Loaded: loaded (/lib/systemd/system/apt-daily-upgrade.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Thu 2019-06-06 04:08:18 UTC; 2 weeks 0 days ago
  Trigger: Fri 2019-06-21 06:36:33 UTC; 4h 48min left

So maybe what I need to do is do the auto updates and auto upgrades check before each run in bare-metal, something like systemctl is-active apt-daily-upgrade.service, systemctl is-enabled apt-daily-upgrade.timer?? Wdyt? @grahamwhaley
And with lock file, If this error occurs again with all auto updates and auto upgrades disabled, then i will deal with it.

@Pennyzct :-) np - first time we hit this it took us some days to work out what the problem was :-)
On all other node/agents we have disabled the auto updates, like you can see in the ansible files used to set up the Packet metrics bare metal machines here. I guess you may have to discuss that with your machine devops to see if that is an acceptable solution?
If not, then yes, you will have to try and find some dynamic way to ensure the background timer routines don't trigger and clash with the CI scripts! If triggering the auto-updates just before each CI run works, then great, that sounds like a solution.

Hi~ @grahamwhaley For all our CI nodes in packet.net, I'm thinking just manually disable them, when setting up this bare-metal CI nodes for Kata community. maybe another item in How to set-up-a-bare-metal-jenkins-slave-node. ;)
For this PR, I'm gonna close it. If i disable all auto updates and auto upgrades services, similar error still occured. Then I will re-open and deal with it.

np @Pennyzct . I have probably said before, but we have some ansible scripts for deploying the (metrics) bare metal slaves in Packet: https://github.com/kata-containers/ci/tree/master/deployment/packet
Maybe they will work or be adaptable for other Packet bare metal machines as well.

Hi~@vielmetti could we use ansible scripts for deploying the ARM bare metal slaves in Packet.net?

Yes @Pennyzct , the Ansible scripts mentioned above by @grahamwhaley should work on Packet for Arm servers with only the change in the "plan" type to pick a server of an appropriate type.