Skip to content

Add perturb-nodes make target to test perturbations specified in the manifest#17

Merged
sergio-mena merged 1 commit intomainfrom
sergio/adapt-for-disconnection
May 19, 2023
Merged

Add perturb-nodes make target to test perturbations specified in the manifest#17
sergio-mena merged 1 commit intomainfrom
sergio/adapt-for-disconnection

Conversation

@sergio-mena
Copy link
Collaborator

@sergio-mena sergio-mena commented May 17, 2023

This takes advantage of cometbft/cometbft#852

These changes are not strictly necessary, but are done for convenience in testing.

Sample output (both typical "e2e" logs, and ansible output):

qa-infra % make perturb-nodes
go run github.com/cometbft/cometbft/test/e2e/runner@b6d4751de perturb -f ./ansible/testnet.toml --infrastructure-type digital-ocean --infrastructure-data ansible/testnet/infrastructure-data.json
go: downloading github.com/cometbft/cometbft v0.0.0-20230517145740-b6d4751ded4c
I[2023-05-17|16:58:58.636] perturb node                                 msg="Disconnecting node validator00..."

PLAY [e2e custom playbook] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [164.92.238.97]

TASK [disconnect node] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.238.97]

TASK [disconnect node] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.238.97]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
164.92.238.97              : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

COMMAND: [ansible-playbook ansible/testnet/playbook-action2.yml -f 50 -u root --inventory 164.92.238.97,]

PLAY [e2e custom playbook] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [164.92.238.97]

TASK [reconnect node] ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.238.97]

TASK [reconnect node] ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.238.97]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
164.92.238.97              : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I[2023-05-17|16:59:21.716] perturb node                                 msg="Node validator00 recovered at height 2810"
I[2023-05-17|16:59:24.717] perturb node                                 msg="Disconnecting node validator02..."

PLAY [e2e custom playbook] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [164.92.235.245]

TASK [disconnect node] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

TASK [disconnect node] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
164.92.235.245             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


PLAY [e2e custom playbook] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [164.92.235.245]

TASK [reconnect node] ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

TASK [reconnect node] ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
164.92.235.245             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I[2023-05-17|16:59:48.041] perturb node                                 msg="Node validator02 recovered at height 2822"
I[2023-05-17|16:59:51.043] perturb node                                 msg="Disconnecting node validator02..."

PLAY [e2e custom playbook] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [164.92.235.245]

TASK [disconnect node] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

TASK [disconnect node] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
164.92.235.245             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


PLAY [e2e custom playbook] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [164.92.235.245]

TASK [reconnect node] ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

TASK [reconnect node] ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [164.92.235.245]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
164.92.235.245             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I[2023-05-17|17:00:12.181] perturb node                                 msg="Node validator02 recovered at height 2822"
qa-infra %

While running the command above, this is the resulting prometheus plot of the heights:

Screenshot 2023-05-17 at 17 01 28

@lasarojc
Copy link
Contributor

Being able to use a single test harness for both local and remote testing is really cool.

@sergio-mena sergio-mena merged commit 3cbc0cb into main May 19, 2023
@sergio-mena sergio-mena deleted the sergio/adapt-for-disconnection branch May 19, 2023 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants