Batch parameter leads to azure pipelines check failure by jamesmyatt · Pull Request #237 · python-jsonschema/check-jsonschema

jamesmyatt · 2023-02-02T15:20:52Z

This file fails locally for me, but is definitely supported. See https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/trigger?view=azure-pipelines#trigger-batch-branches-paths-tags.

> check-jsonschema --builtin-schema vendor.azure-pipelines --data-transform azure-pipelines marshmallow.yaml

Schema validation errors were encountered.
  marshmallow.yaml::$: {'trigger': {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}}, 'resources': {'repositories': [{'repository': 'sloria', 'type': 'github', 'endpoint': 'github', 'name': 'sloria/azure-pipeline-templates', 'ref': 'refs/heads/sloria'}]}, 'jobs': [{'template': 'job--python-tox.yml@sloria', 'parameters': {'toxenvs': ['lint', 'py37', 'py310'], 'os': 'linux'}}, {'template': 'job--pypi-release.yml@sloria', 'parameters': {'dependsOn': ['tox_linux']}}]} is not valid under any of the given schemas
  Underlying errors caused this.
  Best Match:
    $: {'trigger': {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}}, 'resources': {'repositories': [{'repository': 'sloria', 'type': 'github', 'endpoint': 'github', 'name': 'sloria/azure-pipeline-templates', 'ref': 'refs/heads/sloria'}]}, 'jobs': [{'template': 'job--python-tox.yml@sloria', 'parameters': {'toxenvs': ['lint', 'py37', 'py310'], 'os': 'linux'}}, {'template': 'job--pypi-release.yml@sloria', 'parameters': {'dependsOn': ['tox_linux']}}]} is not of type 'string'

Using 0.21.0 installed via pipx on Windows.

Fails locally for me

jamesmyatt · 2023-02-02T15:51:32Z

I'm pretty sure this is an upstream issue (e.g. microsoft/azure-pipelines-vscode#224), but I think it's still useful to confirm whether it's an issue here, and whether there's any regression.

sirosen · 2023-02-02T16:30:17Z

This is not so much a PR as it is a bug report, right? Would you be willing to reopen it as an issue? GitHub provides no facility for converting PRs to issues.

I'm quite tempted to uncritically slap this with the azure-schema-lies label without even trying to understand.
At the time when I initially built out the integration here, I naively thought that I would be able to get Microsoft to invest in fixes to that schema. It has numerous issues, and the validation done here is really best effort.

As for the weirdness of the message you get in this case... I can tell you how we got that message, but I can't really fathom why the schema is structured in a way that leaves validators in this situation.
One of the branches of the Azure Pipelines schema defines the whole document as a string, and since the "best error" logic is just trying to figure out which of the many possible errors is the explanation, this is sort of a degenerate case.

You can see the full slew of errors on all branches of the schema by running with the verbose flag:

$ check-jsonschema --builtin-schema vendor.azure-pipelines --data-transform azure-pipelines batch_param.yaml -v
Schema validation errors were encountered.
  batch_param.yaml::$: {'trigger': {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}}, 'resources': {'repositories': [{'repository': 'sloria', 'type': 'github', 'endpoint': 'github', 'name': 'sloria/azure-pipeline-templates', 'ref': 'refs/heads/sloria'}]}, 'jobs': [{'template': 'job--python-tox.yml@sloria', 'parameters': {'toxenvs': ['lint', 'py37', 'py310'], 'os': 'linux'}}, {'template': 'job--pypi-release.yml@sloria', 'parameters': {'dependsOn': ['tox_linux']}}]} is not valid under any of the given schemas
  Underlying errors caused this.
  Best Match:
    $: {'trigger': {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}}, 'resources': {'repositories': [{'repository': 'sloria', 'type': 'github', 'endpoint': 'github', 'name': 'sloria/azure-pipeline-templates', 'ref': 'refs/heads/sloria'}]}, 'jobs': [{'template': 'job--python-tox.yml@sloria', 'parameters': {'toxenvs': ['lint', 'py37', 'py310'], 'os': 'linux'}}, {'template': 'job--pypi-release.yml@sloria', 'parameters': {'dependsOn': ['tox_linux']}}]} is not of type 'string'
  All Errors:
    $: {'trigger': {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}}, 'resources': {'repositories': [{'repository': 'sloria', 'type': 'github', 'endpoint': 'github', 'name': 'sloria/azure-pipeline-templates', 'ref': 'refs/heads/sloria'}]}, 'jobs': [{'template': 'job--python-tox.yml@sloria', 'parameters': {'toxenvs': ['lint', 'py37', 'py310'], 'os': 'linux'}}, {'template': 'job--pypi-release.yml@sloria', 'parameters': {'dependsOn': ['tox_linux']}}]} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'array'
    $.trigger.batch: True is not valid under any of the given schemas
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $: Additional properties are not allowed ('jobs' was unexpected)
    $: 'stages' is a required property
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'array'
    $.trigger.batch: True is not valid under any of the given schemas
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $: Additional properties are not allowed ('jobs' was unexpected)
    $: 'extends' is a required property
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'array'
    $.trigger.batch: True is not valid under any of the given schemas
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'array'
    $.trigger.batch: True is not valid under any of the given schemas
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $: Additional properties are not allowed ('jobs' was unexpected)
    $: 'phases' is a required property
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'array'
    $.trigger.batch: True is not valid under any of the given schemas
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $: Additional properties are not allowed ('jobs' was unexpected)
    $: 'steps' is a required property
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'array'
    $.trigger.batch: True is not valid under any of the given schemas
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $: Additional properties are not allowed ('jobs' was unexpected)
    $: 'steps' is a required property
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not valid under any of the given schemas
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'string'
    $.trigger: {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}} is not of type 'array'
    $.trigger.batch: True is not valid under any of the given schemas
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $.trigger.batch: True is not of type 'string'
    $: Additional properties are not allowed ('jobs' was unexpected)
    $: 'steps' is a required property
    $: {'trigger': {'batch': True, 'branches': {'include': ['dev', '2.x-line', 'test-me-*']}, 'tags': {'include': ['*']}}, 'resources': {'repositories': [{'repository': 'sloria', 'type': 'github', 'endpoint': 'github', 'name': 'sloria/azure-pipeline-templates', 'ref': 'refs/heads/sloria'}]}, 'jobs': [{'template': 'job--python-tox.yml@sloria', 'parameters': {'toxenvs': ['lint', 'py37', 'py310'], 'os': 'linux'}}, {'template': 'job--pypi-release.yml@sloria', 'parameters': {'dependsOn': ['tox_linux']}}]} is not of type 'string'

It's almost unreadable in this case, since it's a branch-by-branch evaluation finding no joy and reporting all of the errors.

I'm aware of the level of stink about this whole thing, especially for an end user who "just" wants working validation. But without a more serious attitude from Microsoft about making the schema usable and interoperable, I'm not sure I can do much. e.g. I could offer a flag to stringify bools, but who knows what else that might break?

jamesmyatt · 2023-02-02T16:38:44Z

So, yes. It's an upstream issue, in the sense that the azure schema is wrong. I think the relevant issue here is
#42.

I had to raise the PR to get it to run the tests in CI to isolate any issue that may have been local to my machine.

For check-jsonschema, I wonder whether any of the following options are a good idea:

Patch (automatically) the downloaded azure-pipelines schema to fix this.
Add a unit test that will detect when the azure schema is fixed.
Add in the documentation that the schema is known to lie and how to work around it.

sirosen · 2023-02-02T19:37:45Z

I'm going to be closing this because PRs are not the correct way to report and track issues.
This is really an upstream issue anyway.

I think the best thing to do is to report these issues, as you encounter them, on the pipelines-vscode repo, https://github.com/microsoft/azure-pipelines-vscode/
Let Microsoft bear the brunt of this. They're the only ones in any kind of position to fix it.

Feel free to refer them back to me, FWIW. We can do the same silly dance we did last year, in which I try to tell them what needs to be fixed, and they tell me that I should blackhole my feedback into their forums. I am -- I'm sure it's quite clear -- pretty pessimistic about any good outcome. But at least you'll be able to nag people in a better position to fix this.

Patch (automatically) the downloaded azure-pipelines schema to fix this.

This is something I've thought of before, but I've had to reject the idea.

The schema is enormous, broken in innumerable ways, and the full Azure Pipelines definition language is not even expressible in JSON Schema (which is why there's already a transformer for it to handle some of the feasible cases).
Patching it is even more boundary breaking than the existing transform.

Add a unit test that will detect when the azure schema is fixed.

Based on what I've seen, it won't be fixed, or it will be fixed and broken again.
I also don't think it's this project's responsibility to be trying to keep track of this.

(I also have more philosophical objections to tests being used in this way, but it's not super relevant.)

Add in the documentation that the schema is known to lie and how to work around it.

This seems to me like the only path forward in which I can do anything here.

The sloppy handling of booleans is probably going to persist. I can at least doc that bool fields are often broken in Azure, on the FAQ page.
I'll file a distinct issue to document this.

jamesmyatt · 2023-02-03T09:21:01Z

Agreed. Thanks

Add batch parameter to marshmallow.yaml

7b71899

Fails locally for me

jamesmyatt changed the title ~~Add batch parameter to marshmallow.yaml~~ Add batch parameter leads to azure pipelines check failure Feb 2, 2023

jamesmyatt changed the title ~~Add batch parameter leads to azure pipelines check failure~~ Batch parameter leads to azure pipelines check failure Feb 2, 2023

sirosen added the azure-schema-lies The Azure Pipelines Schema is at it again label Feb 2, 2023

jamesmyatt marked this pull request as draft February 2, 2023 16:39

sirosen closed this Feb 2, 2023

sirosen mentioned this pull request Feb 2, 2023

Add FAQ doc showing quoted bool fields for Azure Pipelines issues #238

Closed

jamesmyatt mentioned this pull request Feb 3, 2023

Check for #224 regression microsoft/azure-pipelines-vscode#508

Closed

jamesmyatt deleted the check-azure-pipelines branch February 3, 2023 10:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch parameter leads to azure pipelines check failure#237

Batch parameter leads to azure pipelines check failure#237
jamesmyatt wants to merge 1 commit intopython-jsonschema:mainfrom
jamesmyatt:check-azure-pipelines

jamesmyatt commented Feb 2, 2023 •

edited

Loading

Uh oh!

jamesmyatt commented Feb 2, 2023 •

edited

Loading

Uh oh!

sirosen commented Feb 2, 2023

Uh oh!

jamesmyatt commented Feb 2, 2023

Uh oh!

sirosen commented Feb 2, 2023

Uh oh!

jamesmyatt commented Feb 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jamesmyatt commented Feb 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jamesmyatt commented Feb 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sirosen commented Feb 2, 2023

Uh oh!

jamesmyatt commented Feb 2, 2023

Uh oh!

sirosen commented Feb 2, 2023

Uh oh!

jamesmyatt commented Feb 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jamesmyatt commented Feb 2, 2023 •

edited

Loading

jamesmyatt commented Feb 2, 2023 •

edited

Loading