Skip to content

Testing all options #49

@georgefst

Description

@georgefst

Until recently, all our output options have been binary choices*, with a neat split in to "the Ormolu version" and "the Fourmolu version". Our tests have thus tested our default options, and Ormolu's default options. This was never going to last. #48 is the first to introduce an option which is not used by default.

We have a few choices:

  • Run all the tests with all the combinations of options. This may take a long time to run (I haven't tried it yet), and it would start to become difficult to review all the outputs to check that everything looks sensible. But it does give us pretty comprehensive regression tests.
    • Obviously this grows exponentially as we add more options...
  • Only test each option on the examples where it should be relevant, e.g. there's no point running with and without record-brace-space on a file that doesn't contain any records. Setting this up would require reviewing each test case to determine which options are relevant. We could also miss bugs where an option affects something it shouldn't.
  • Consider which options are likely to interact with each other, and filter the set of combinations accordingly. This has some of the same drawbacks as the previous choice.

* Apart from indentation, but for the purposes of this discussion that's a choice between two and four. There doesn't seem much point testing with any more than that.

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions