Skip to content

Remove allOf JSON schema workarounds#10029

Merged
sydney-runkle merged 2 commits intopydantic:mainfrom
dpeachey:remove-all-of-json-schema-workarounds
Aug 8, 2024
Merged

Remove allOf JSON schema workarounds#10029
sydney-runkle merged 2 commits intopydantic:mainfrom
dpeachey:remove-all-of-json-schema-workarounds

Conversation

@dpeachey
Copy link
Copy Markdown
Contributor

@dpeachey dpeachey commented Aug 2, 2024

Change Summary

The JSON schema spec was updated to allow sibling keys alongside $ref keys so it is no longer required to wrap the $ref key in a allOf array with a single $ref element. This change removes the allOf workarounds.

Related issue number

fix #6343

Checklist

  • The pull request title is a good summary of the changes - it will be used in the changelog
  • Unit tests for the changes exist
  • Tests pass on CI
  • Documentation reflects the changes where applicable
  • My PR is ready to review

@github-actions github-actions bot added the relnotes-fix Used for bugfixes. label Aug 2, 2024
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Aug 2, 2024

CodSpeed Performance Report

Merging #10029 will not alter performance

Comparing dpeachey:remove-all-of-json-schema-workarounds (708f1ca) with main (441f6fe)

Summary

✅ 17 untouched benchmarks

@dpeachey
Copy link
Copy Markdown
Contributor Author

dpeachey commented Aug 2, 2024

Hi,

fastapi test for OpenAPI schema format failed which is to be expected since the output is changing, however I don't know what to do about this :) Could someone advise?

Thanks,

Dan

@sydney-runkle
Copy link
Copy Markdown
Contributor

Exciting stuff! Let's wait for @adriangb to review this on Monday - he's pretty familiar with this part of the codebase :).

@sydney-runkle sydney-runkle added relnotes-change Used for changes to existing functionality which don't have a better categorization. and removed relnotes-fix Used for bugfixes. labels Aug 2, 2024
@sydney-runkle sydney-runkle requested a review from adriangb August 2, 2024 13:41
@dmontagu
Copy link
Copy Markdown
Contributor

dmontagu commented Aug 2, 2024

I'll note that I am in favor of this change, and my experience has been that many/most existing tools that interop with JSON schema behave as expected when sibling keys to a $ref exist, even though they were designed at a time when that wasn't allowed.

However, just as part of appropriate due diligence — can you share a reference that this is now allowed in JSON schema? In particular, I'm thinking we may need to reference a new draft version or similar.

The FastAPI test failures can be white-listed if we want to move forward with this, there is an existing pattern for that I can dig up if there are no white-listed tests currently.

@sydney-runkle
Copy link
Copy Markdown
Contributor

I believe this might be what we're looking for: https://json-schema.org/draft/2019-09/release-notes#core-vocabulary

Copy link
Copy Markdown
Contributor

@sydney-runkle sydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dpeachey,

This looks great! Let's add an exception for the FastAPI failing test like was done for this one:

# To skip a specific test, add '--deselect path/to/test.py::test_name' to the end of this command
#
# To update the list of deselected tests, remove all deselections, run the tests, and re-add any remaining failures
# TODO remove this once that test is fixed, see https://github.com/pydantic/pydantic/pull/9064
./scripts/test.sh -vv --deselect tests/test_tutorial/test_path_params/test_tutorial005.py::test_get_enums_invalid

Once we do that for the failing test, let's merge!

@pydantic-hooky pydantic-hooky bot added the awaiting author revision awaiting changes from the PR author label Aug 7, 2024
@dpeachey dpeachey force-pushed the remove-all-of-json-schema-workarounds branch from 3cdfb81 to 708f1ca Compare August 7, 2024 19:35
@dpeachey dpeachey requested a review from sydney-runkle August 7, 2024 19:48
@sydney-runkle
Copy link
Copy Markdown
Contributor

@tiangolo, just wanted to notify you re this change, given that it broke one of the fastapi integration tests 👍

whiskeyriver added a commit to whiskeyriver/djantic2 that referenced this pull request Dec 8, 2024
Pydantic's json schema output changed after 2.8, eliminating the `allOf`
wrapping of a single `$ref` element.

See: pydantic/pydantic#10029

Here's a hacky way of ensuring tests run between these versions.
whiskeyriver added a commit to whiskeyriver/djantic2 that referenced this pull request Dec 8, 2024
Pydantic's json schema output changed after 2.8, eliminating the `allOf`
wrapping of a single `$ref` element.

See: pydantic/pydantic#10029

Here's a hacky way of ensuring tests run between these versions.
Mark90 added a commit to workfloworchestrator/orchestrator-core that referenced this pull request Mar 4, 2025
pboers1988 pushed a commit to workfloworchestrator/orchestrator-core that referenced this pull request Mar 5, 2025
* Update pydantic[email] requirement from ~=2.8.2 to ~=2.9.2

Updates the requirements on [pydantic[email]](https://github.com/pydantic/pydantic) to permit the latest version.
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md)
- [Commits](pydantic/pydantic@v2.8.2...v2.9.2)

---
updated-dependencies:
- dependency-name: pydantic[email]
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Remove 'allOf' from expected pydantic schema's

Related change: pydantic/pydantic#10029

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mark90 <mark_moes@live.nl>
psychedelicious added a commit to ebr/InvokeAI that referenced this pull request Mar 31, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
psychedelicious added a commit to ebr/InvokeAI that referenced this pull request Mar 31, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
psychedelicious added a commit to ebr/InvokeAI that referenced this pull request Mar 31, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
ebr pushed a commit to ebr/InvokeAI that referenced this pull request Apr 2, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
psychedelicious added a commit to ebr/InvokeAI that referenced this pull request Apr 2, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
psychedelicious added a commit to invoke-ai/InvokeAI that referenced this pull request Apr 3, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
psychedelicious added a commit to invoke-ai/InvokeAI that referenced this pull request Apr 4, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
psychedelicious added a commit to invoke-ai/InvokeAI that referenced this pull request Apr 4, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
psychedelicious added a commit to invoke-ai/InvokeAI that referenced this pull request Apr 4, 2025
In pydantic/pydantic#10029, pydantic made an improvement to its generated JSON schemas (OpenAPI schemas). The previous and new generated schemas both meet the schema spec.

When we parse the OpenAPI schema to generate node templates, we use some typeguard to narrow schema components from generic OpenAPI schema objects to a node field schema objects. The narrower node field schema objects contain extra data.

For example, they contain a `field_kind` attribute that indicates it the field is an input field or output field. These extra attributes are not part of the OpenAPI spec (but the spec allows does allow for this extra data).

This typeguard relied on a pydantic implementation detail. This was changed in the linked pydantic PR, which released with v2.9.0. With the change, our typeguard rejects input field schema objects, causing parsing to fail with errors/warnings like `Unhandled input property` in the JS console.

In the UI, this causes many fields - mostly model fields - to not show up in the workflow editor.

The fix for this is very simple - instead of relying on an implementation detail for the typeguard, we can check if the incoming schema object has any of our invoke-specific extra attributes. Specifically, we now look for the presence of the `field_kind` attribute on the incoming schema object. If it is present, we know we are dealing with an invocation input field and can parse it appropriately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting author revision awaiting changes from the PR author relnotes-change Used for changes to existing functionality which don't have a better categorization.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove `'allOf' JSON schema workarounds

3 participants