fix(oas3.1): correctly validate paths using $ref#2485
Merged
Stranger6667 merged 1 commit intoschemathesis:masterfrom Oct 5, 2024
BronzeDeer:update-oas-3.1-spec
Merged
fix(oas3.1): correctly validate paths using $ref#2485Stranger6667 merged 1 commit intoschemathesis:masterfrom BronzeDeer:update-oas-3.1-spec
Stranger6667 merged 1 commit intoschemathesis:masterfrom
BronzeDeer:update-oas-3.1-spec
Conversation
Schemathesis uses static version of the current published version of the OpenApi 3.1 spec metaschema (https://spec.openapis.org/oas/3.1/schema/2022-10-07) to validate openapi 3.1 spec documents. Unfortunately, the published version has at least one known bug in which the schema for `paths` references the definition of a concrete `path-item` instead of `path-item-or-reference`, which might still be technically incorrect when it comes handling the case of ref and sibling fields, but is correct according to the documented definition of a pathItemObject. This oversight has been noticed multiple times OAI/OpenAPI-Specification#3298 OAI/OpenAPI-Specification#2635 (comment) OAI/OpenAPI-Specification#2635 (comment) OAI/OpenAPI-Specification#3513 OAI/OpenAPI-Specification#2657 (comment) And finally fixed in Feb 2024 OAI/OpenAPI-Specification#3355 with a slightly bigger rework of the pathItem schema. Sadly, due to confusion about how to release fixes in schemas OAI/OpenAPI-Specification#151 (comment) this change has not been published anywhere except schema.yaml in the git repo, not even in schema.json, which appearantly only gets refreshed once per release of the metaschema OAI/OpenAPI-Specification#3355 (comment) This commit updates the stored schema from the most up-to-date 3.1.0 schema.yaml from 0035208 to close the bug and make spec-valid openapi spec files that use $ref under path finally validate correctly in schemathesis. It also adds a corresponding regression test
Contributor
Author
|
I get a very small handful of test failures in local runs: FAILED test/loaders/test_common.py::test_port_override_with_ipv6[from_uri-openapi3_schema_url-requests.get-http://[2002:7f00:1::]:8081/schema.yaml] - ValueError: Host '[2002:7f00:0001::]' cannot contain '[' (at position 0)
FAILED test/loaders/test_common.py::test_port_override_with_ipv6[from_url-graphql_url-requests.post-http://[2002:7f00:1::]:8081/graphql] - ValueError: Host '[2002:7f00:0001::]' cannot contain '[' (at position 0)
FAILED test/hooks/test_hooks.py::test_global_body_hook[Open API 2.0] - AssertionError: assert 0 == 5
FAILED test/hooks/test_hooks.py::test_global_body_hook[Open API 3.0] - AssertionError: assert 0 == 5
FAILED test/cli/test_cassettes.py::test_replay[Open API 2.0-plain-True] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/test_app.py::test_app[flask] - AssertionError: assert 'Schemathesis test server is running!' in ''
FAILED test/test_app.py::test_app[aiohttp] - AssertionError: assert 'Schemathesis test server is running!' in ''
FAILED test/cli/test_cassettes.py::test_replay[Open API 2.0-plain-False] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/cli/test_cassettes.py::test_replay[Open API 2.0-base64-True] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/cli/test_cassettes.py::test_replay[Open API 2.0-base64-False] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/cli/test_cassettes.py::test_replay[Open API 3.0-plain-True] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/cli/test_cassettes.py::test_replay[Open API 3.0-plain-False] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/cli/test_cassettes.py::test_replay[Open API 3.0-base64-True] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/cli/test_cassettes.py::test_replay[Open API 3.0-base64-False] - ValueError: Host '127.0.0.1:56515' cannot contain ':' (at position 9)
FAILED test/hooks/test_hooks.py::test_after_load_schema - AssertionError: assert {'errors': 1,...pped': 0, ...} == {'errors': 0,...pp...
Since they look unrelated to OAS3.1 and they occur both on master and on my fix, I'll assume that these are either pre-existing or my (admittedly hasty and hacky) build and test setup is slightly wonky |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2485 +/- ##
==========================================
+ Coverage 98.09% 98.10% +0.01%
==========================================
Files 150 150
Lines 13582 13596 +14
Branches 2913 2914 +1
==========================================
+ Hits 13323 13339 +16
+ Misses 144 143 -1
+ Partials 115 114 -1 ☔ View full report in Codecov by Sentry. |
CodSpeed Performance ReportMerging #2485 will not alter performanceComparing Summary
|
Member
|
Awesome! Thank you for the report and the fix :) P.S. Test failures are indeed unrelated and happened before due to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2484
Description
Schemathesis uses static version of the current published version of the OpenApi 3.1 spec metaschema
(https://spec.openapis.org/oas/3.1/schema/2022-10-07) to validate openapi 3.1 spec documents. Unfortunately, the published version has at least one known bug in which the schema for
pathsreferences the definition of a concretepath-iteminstead ofpath-item-or-reference, which might still be technically incorrect when it comes handling the case of ref and sibling fields, but is correct according to the documented definition of a pathItemObject.This oversight has been noticed multiple times
OAI/OpenAPI-Specification#3298 OAI/OpenAPI-Specification#2635 (comment) OAI/OpenAPI-Specification#2635 (comment) OAI/OpenAPI-Specification#3513 OAI/OpenAPI-Specification#2657 (comment)
And finally fixed in Feb 2024
OAI/OpenAPI-Specification#3355 with a slightly bigger rework of the pathItem schema.
Sadly, due to confusion about how to release fixes in schemas OAI/OpenAPI-Specification#151 (comment) this change has not been published anywhere except schema.yaml in the git repo, not even in schema.json, which appearantly only gets refreshed once per release of the metaschema
OAI/OpenAPI-Specification#3355 (comment)
This commit updates the stored schema from the most up-to-date 3.1.0 schema.yaml from 0035208 to close the bug and make spec-valid openapi spec files that use $ref under path finally validate correctly in schemathesis. It also adds a corresponding regression test
Checklist