Skip to content

OpenAPI Spec Conformance Tests: Part 1#2358

Merged
ketan merged 7 commits intomainfrom
sahilm/conformance-tests-part-1
Mar 24, 2026
Merged

OpenAPI Spec Conformance Tests: Part 1#2358
ketan merged 7 commits intomainfrom
sahilm/conformance-tests-part-1

Conversation

@sahilm
Copy link
Copy Markdown
Contributor

@sahilm sahilm commented Mar 23, 2026 •
edited
Loading

What:

Introduce OpenAPI Spec conformance tests. In part 1 we validate and document that Specmatic supports 001-http-methods and 002-path-parameters features from the OpenAPI specification (version 3.0.x) by running a loop test. For example a loop test of the specification 002-path-parameters/007-two-params-with-separator.yaml starts a Specmatic mock for this specification and then points a Specmatic test at it. A successful test validates that Specmatic can load, parse, mock and test such a specification.

Why:

This ensures and documents that Specmatic is and remains compliant with the OpenAPI specification.

Design:

  1. All of Specmatic is the System Under Test and is treated as a block box. Only the public interface (exit code and request/responses) are used to validate Specmatic's behaviour.
  2. Since these are long running black box tests it's necessary to run them in parallel. We intend to add several hundreds of tests to this suite in follow up PRs. The tests depend on the docker image of Specmatic only.
  3. The tests live in an independent gradle project. To run the tests please run ./gradlew :conformance-tests:check -PenableConformanceTests=true. These tests will be wired up to the main build in subsequent PRs.

Follow up PRs:

  1. Wire up in the main build and documentation.
  2. Detailed request/response validation.
  3. Header and Content-Type validations.
  4. More features of OpenAPI will be exercised.
  • Unit Tests
  • Build passing locally
  • Sonar Quality Gate
  • Security scans don't report any vulnerabilities
  • Documentation added/updated (share link)
  • Sample Project added/updated (share link)
  • Demo video (share link)
  • Article on Website (share link)
  • Roadmpap updated (share link)
  • Conference Talk (share link)

sahilm added 5 commits March 23, 2026 13:10
@sahilm
First cut of conformance tests.
e67c950 
- Building up a suite of conformance tests to validate and report
  on Specmatic's conformance to the OpenAPI specification.
- Starting out with a set of loop tests that validate that Specmatic
  can mock and test against a set of OpenAPI specs.
@sahilm
Add more tests cases and generate the tests
8e80155
@sahilm
Run the tests in parallel
a78c305
@sahilm
Need to wait for the mock to be healthy
19f947b
@sahilm
Wait for upto a minute for the tests to finish
0e2a29b
@sahilm sahilm requested a review from joelrosario March 23, 2026 06:41
sahilm
sahilm commented Mar 23, 2026
View reviewed changes
conformance-tests/src/test/kotlin/conformance_tests/AbstractConformanceTest.kt
pathToOpenAPISpecFile = openAPISpecFile
)

try {
Copy link
Copy Markdown
Contributor Author

@sahilm sahilm Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not using BeforeAll/AfterAll here because this will change soon in a follow up PR.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 23, 2026

Summary

Line coverage Branch coverage

Assembly Line coverage Branch coverage
application 64.9% 58.6%
application/backwardCompatibility 79.5% 65.1%
application/mcp 84.7% 76.9%
application/validate 1.2% 2.5%
io/specmatic/conformance/tests 0% ****
io/specmatic/conversions 93.4% 78.4%
io/specmatic/conversions/lenient 97.6% 77.4%
io/specmatic/core 81.3% 62.8%
io/specmatic/core/azure 0% 0%
io/specmatic/core/config 61.6% 39.4%
io/specmatic/core/config/v1 97.9% 25%
io/specmatic/core/config/v2 91.9% 78.8%
io/specmatic/core/config/v3 56.7% 30.2%
io/specmatic/core/config/v3/components 78.9% 40%
io/specmatic/core/config/v3/components/runOptions 75.4% 47.5%
io/specmatic/core/config/v3/components/services 89.8% 59%
io/specmatic/core/config/v3/components/settings 100% 63.8%
io/specmatic/core/config/v3/components/sources 83.8% 61.5%
io/specmatic/core/config/v3/specmatic 57.6% 25%
io/specmatic/core/discriminator 52.6% 50%
io/specmatic/core/examples/module 45.7% 28.9%
io/specmatic/core/examples/server 73.8% 36.8%
io/specmatic/core/examples/source 91.6% 60%
io/specmatic/core/filters 83.9% 72.1%
io/specmatic/core/fuzzy 97.3% 93.8%
io/specmatic/core/git 9% 10.8%
io/specmatic/core/jsonoperator 68.3% 70.5%
io/specmatic/core/jsonoperator/value 86.2% 74.4%
io/specmatic/core/lifecycle 0% ****
io/specmatic/core/log 76.2% 62.3%
io/specmatic/core/matchers 86.9% 76.8%
io/specmatic/core/overlay 87.3% 58.8%
io/specmatic/core/pattern 87.5% 77.1%
io/specmatic/core/pattern/config 100% ****
io/specmatic/core/pattern/regex 94.5% 87.5%
io/specmatic/core/report 97.5% 91.6%
io/specmatic/core/route/modules 95% 87.5%
io/specmatic/core/utilities 70.3% 62.1%
io/specmatic/core/value 82.3% 69%
io/specmatic/core/wsdl/parser 81.4% 41%
io/specmatic/core/wsdl/parser/message 90% 72%
io/specmatic/core/wsdl/parser/operation 100% 100%
io/specmatic/core/wsdl/payload 81.1% 50%
io/specmatic/junit5/support 0% ****
io/specmatic/loader 84.6% 53.5%
io/specmatic/mcp/constants 0% ****
io/specmatic/mcp/report 0% 0%
io/specmatic/mcp/test 0% 0%
io/specmatic/mcp/test/client 0% 0%
io/specmatic/mcp/test/client/model 0% 0%
io/specmatic/mock 61.7% 56.5%
io/specmatic/proxy 67% 52%
io/specmatic/specmatic/core 100% ****
io/specmatic/specmatic/executable 100% ****
io/specmatic/specmatic/mcp 0% ****
io/specmatic/stub 79.3% 69.1%
io/specmatic/stub/listener 100% 100%
io/specmatic/stub/report 100% 77%
io/specmatic/test 74.2% 63.6%
io/specmatic/test/asserts 95.2% 86.6%
io/specmatic/test/handlers 89.7% 63.3%
io/specmatic/test/listeners 86.4% 53.2%
io/specmatic/test/reports 60% 33.3%
io/specmatic/test/reports/coverage 97.6% 55.1%
io/specmatic/test/reports/coverage/console 90.9% 82.7%
io/specmatic/test/reports/coverage/html 0% 0%
io/specmatic/test/reports/renderers 20% 24.1%
io/specmatic/test/utils 92.3% 80%

sahilm
sahilm commented Mar 23, 2026
View reviewed changes
conformance-tests/build.gradle.kts
}
}

val generateConformanceTests by tasks.registering {
Copy link
Copy Markdown
Contributor Author

@sahilm sahilm Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to use Codegen here because jUnit can only parallelize if each test lives in its own class.

@sahilm sahilm mentioned this pull request Mar 23, 2026
Merged
@sahilm sahilm requested a review from ketan March 23, 2026 12:25
@sahilm sahilm mentioned this pull request Mar 23, 2026
Merged
@ketan ketan merged commit e0ce01b into main Mar 24, 2026
6 of 9 checks passed
@ketan ketan deleted the sahilm/conformance-tests-part-1 branch March 24, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joelrosario joelrosario Awaiting requested review from joelrosario

@ketan ketan Awaiting requested review from ketan

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sahilm @ketan