Skip to content

Improve: dunify testsuite#881

Merged
gpetiot merged 15 commits intoocaml-ppx:masterfrom
trefis:dunify-test
Jun 28, 2019
Merged

Improve: dunify testsuite#881
gpetiot merged 15 commits intoocaml-ppx:masterfrom
trefis:dunify-test

Conversation

@trefis
Copy link
Copy Markdown
Contributor

@trefis trefis commented Jun 8, 2019

I was a bit confused by the in-repo testsuite when I started hacking on ocamlformat. More specifically, I still don't know how to automatically accept a diff, I currently run ./test.sh --meld from the test directory (which I assume is not the most efficient way to do things).

Since the dune workflow for diffing and promoting seems particularly adapted, this PR proposes to add some dune files under test/.
I needed some minor modifications to the formatting of .opts files to make dune happy, but apart from that I don't think there is any risk of breaking the preexisting test framework.

This is still incomplete (I only add a dune file in tests/passing), but I figured it was a good point to gather feedback and ask for help.

An issue I'm currently having is that there are some tests which are expected to fail (stderr is recorded), and for which ocamlformat will have a non-zero exit code. And I don't know how to tell dune "it's expected to fail, don't worry". @rgrinberg: is that something that dune knows how to handle? Or can I at least ignore the exit code?

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Jun 8, 2019

Note: I made those tests pass using (system "... || true") instead of (run ...) but that makes me a bit sad.

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Jun 8, 2019

Also, if people were willing to give up on the html output, I think we could just get rid of test.sh and switch everything to dune.

@gpetiot
Copy link
Copy Markdown
Collaborator

gpetiot commented Jun 8, 2019

I was a bit confused by the in-repo testsuite when I started hacking on ocamlformat. More specifically, I still don't know how to automatically accept a diff, I currently run ./test.sh --meld from the test directory (which I assume is not the most efficient way to do things).

We have a script tools/update_tests.sh that accepts the filenames of the tests (or --all), but it's indeed probably better to let dune take care of these things.

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Jun 10, 2019

We have a script tools/update_tests.sh that accepts the filenames of the tests (or --all), but it's indeed probably better to let dune take care of these things

I missed that one, I tried the various options of test/test.sh but somehow that always updated more files than I actually wanted, and I'm not sure why.

Anyway, I pushed some more commits, I removed test/test.sh and decided that only dune would take care of the regtests.
The other tests (running on infer, jsoo, etc) are still done through the makefile. I'm not sure it's worth switching that to dune.

@trefis
Copy link
Copy Markdown
Contributor Author

trefis commented Jun 10, 2019

The test failure seems to be because test/Makefile calls dune runtest which will try to build in every context, including coverage which is apparently not currently done on travis.

I guess I want to pass a dune-workspace file manually in that case. (Because I didn't see how to select a specific context from the command line).

Copy link
Copy Markdown
Collaborator

@Julow Julow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome !

regtests:
@echo running regtests with $(OCAMLFORMAT_EXE)
@./test.sh --ocamlformat $(OCAMLFORMAT_EXE)
@dune runtest
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't use --profile=$(MODE) ? Otherwise the echo above will be wrong and the MODE variable useless.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree the echo will be wrong, but I don't think that flag does what you want exactly. I think profile can only be set to release or dev. Which is different from the build context, which afaict one cannot set from the command line.

@rgrinberg: can you confirm?

@rgrinberg
Copy link
Copy Markdown
Contributor

And I don't know how to tell dune "it's expected to fail, don't worry". @rgrinberg: is that something that dune knows how to handle? Or can I at least ignore the exit code?

There's no good way to handle this at the moment. There was a suggestion to add an action that checks for a particular exit code, but no concrete ticket/rfc. If you'd like this feature, then please make a ticket.

@gpetiot gpetiot changed the title [WIP] Dunify testsuite Improve: dunify testsuite Jun 28, 2019
@gpetiot gpetiot requested a review from Julow June 28, 2019 06:31
Copy link
Copy Markdown
Collaborator

@gpetiot gpetiot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Copy link
Copy Markdown
Collaborator

@Julow Julow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be merged !

@gpetiot gpetiot merged commit 6d9f553 into ocaml-ppx:master Jun 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants