Skip to content

Add ARG tests#1470

Merged
sim642 merged 17 commits intomasterfrom
arg-test
May 27, 2024
Merged

Add ARG tests#1470
sim642 merged 17 commits intomasterfrom
arg-test

Conversation

@sim642
Copy link
Copy Markdown
Member

@sim642 sim642 commented May 16, 2024

Like in #1462, I'm always afraid of doing any modifications in the ARG generation because it's very fragile and there have been no tests so far.

At some point I added https://github.com/goblint/analyzer/blob/bffc5e3cbb1f43bd5c9483a6545802cfba5d3d76/scripts/sv-comp/witness-isomorphism.py to semi-automatically check if two sets of GraphML witnesses are isomorphic. But that requires a set of reference witnesses (which we don't have in the repository) and the networkx Python library that implements graph isomorphism algorithms.

This PR adds an alternative means of fully automatic tests for ARGs. This is similar to the CFG tests: a GraphViz graph is rendered as ASCII using graph-easy and expected outputs are in the repository.
The tests are very similar to Cram tests: if something changes, the tests will fail with a diff that can be promoted if intended. However, they aren't Cram tests but just dune expectation tests using rule generation.

The mechanism is somewhat complex (at least without dune 3.14), but nobody other than me will probably ever have to deal with these anyway (because they should just keep passing).

TODO

@sim642 sim642 added testing sv-comp SV-COMP (analyses, results), witnesses labels May 16, 2024
@sim642 sim642 self-assigned this May 16, 2024
@sim642 sim642 marked this pull request as ready for review May 17, 2024 10:38
@sim642 sim642 removed their assignment May 23, 2024
@michael-schwarz michael-schwarz self-requested a review May 26, 2024 16:12
Copy link
Copy Markdown
Member

@michael-schwarz michael-schwarz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked it over and it seems to make sense at first glance. Maybe it would be a good idea for someone else who is familiar with the ARGs (maybe @karoliineh?) to also take a look at this since I'm out of my depth here?

@sim642 sim642 mentioned this pull request May 27, 2024
3 tasks
sim642 added a commit to goblint/GobPie that referenced this pull request May 27, 2024
@sim642 sim642 added this to the SV-COMP 2025 milestone May 27, 2024
@sim642 sim642 merged commit 2eb0440 into master May 27, 2024
@sim642 sim642 deleted the arg-test branch May 27, 2024 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sv-comp SV-COMP (analyses, results), witnesses testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants