feat: auto-detect Kubernetes crd schema#1050
feat: auto-detect Kubernetes crd schema#1050datho7561 merged 13 commits intoredhat-developer:mainfrom
Conversation
|
@qvalentin Thanks for the nice work. I actually started working on this as well: https://github.com/tricktron/yaml-language-server/tree/f-kubernetes-improvements However, I just hardcoded the core kubernetes groups but yours looks better in my opinion. I then actually got sidetracked by adding some more features to my https://github.com/tricktron/crd2jsonschema tool so that I could easily test end-to-end with the newest openshift and tekton crds. Maybe you can also take over some tests from my my branch, e.g. main...tricktron:yaml-language-server:f-kubernetes-improvements#diff-5d1a1b48130b3cb749d3ae36d72e52f8882d6fa5dc9eaedc5d6e606e411b176cR1194-R1219 And lastly, we should then probably add some documentation as well. This might be a breaking change for the helm language server where you hardcode the kubernetes schemas to |
|
@tricktron I took a look at the schemaValidation.test.ts but since in my logic the customSchemaProvider has higher priority then the CRD logic the tests don't work with my branch. But I might copy the config options format. |
|
Hi @msivasubramaniaan what are your thoughts on this? |
Add an option to overwrite the default Kubernetes schema URL with a custom one, by setting a `YAMLLS_KUBERNETES_SCHEMA_URL` environment variable. See - redhat-developer#998 - redhat-developer#824 - redhat-developer#841 - redhat-developer#962 - redhat-developer#1050 Main use case is to use https://github.com/ricoberger/kubernetes-json-schema, which contains the schema for Kubernetes and the CRDs I'm using.
|
I'll try to take a look at this sometime this afternoon, it seems really helpful. |
|
The tests are failing it seems, with a type error |
|
I'm trying out: apiVersion: route.openshift.io/v1
kind: Route
spec:It seems it was trying to use a custom schema provider instead of the crd schema autodetection, so I made this change: diff --git a/src/languageservice/services/yamlSchemaService.ts b/src/languageservice/services/yamlSchemaService.ts
index 432851f..9102233 100644
--- a/src/languageservice/services/yamlSchemaService.ts
+++ b/src/languageservice/services/yamlSchemaService.ts
@@ -430,6 +430,22 @@ export class YAMLSchemaService extends JSONSchemaService {
return resolveSchemaForResource([modelineSchema]);
}
+ if (this.yamlSettings?.kubernetesCRDStoreEnabled) {
+ for (const entry of this.filePatternAssociations) {
+ if (entry.uris && entry.uris[0] == KUBERNETES_SCHEMA_URL && entry.matchesPattern(resource)) {
+ return resolveSchemaForResource([KUBERNETES_SCHEMA_URL]).then((schema) => {
+ const kubeSchema = autoDetectKubernetesSchemaFromDocument(
+ doc,
+ this.yamlSettings.kubernetesCRDStoreUrl ?? CRD_CATALOG_URL,
+ schema
+ );
+ if (kubeSchema) {
+ return resolveSchemaForResource([kubeSchema]);
+ }
+ });
+ }
+ }
+ }
if (this.customSchemaProvider) {
return this.customSchemaProvider(resource)
.then((schemaUri) => {
@@ -473,22 +489,6 @@ export class YAMLSchemaService extends JSONSchemaService {
}
);
}
- if (this.yamlSettings?.kubernetesCRDStoreEnabled) {
- for (const entry of this.filePatternAssociations) {
- if (entry.uris && entry.uris[0] == KUBERNETES_SCHEMA_URL && entry.matchesPattern(resource)) {
- return resolveSchemaForResource([KUBERNETES_SCHEMA_URL]).then((schema) => {
- const kubeSchema = autoDetectKubernetesSchemaFromDocument(
- doc,
- this.yamlSettings.kubernetesCRDStoreUrl ?? CRD_CATALOG_URL,
- schema
- );
- if (kubeSchema) {
- return resolveSchemaForResource([kubeSchema]);
- }
- });
- }
- }
- }
return resolveSchema();
}After I do that and try to validate the YAML, I get "Unable to load schema..." as an error in the YAML, since the calculated URL doesn't line up with the location of the schema in the repo (the OpenShift CRDs are split across multiple group ids, whereas in the repo with the CRDs all the schemas are in the folder "openshift/v4.11-strict"). After this I decided to try out: apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
spec:And that appears to work after applying the above patch, but the patch appears to break many other tests. |
|
Hi @datho7561 I'm not sure what should have higher priority, a custom schema provider or the crd check. I would guess the custom schema provider? I'm just using the lsp with neovim where no custom schema provider is active. I wonder which provider is active in your setup. So the mysql example is working for me, but indeed the openshift one is not. Edit: I took a quick look at the catalog and it seems openshift is the only directory that is not using an API group name. I would say it's just a bad example and should be treated separately. |
I'm using vscode-yaml; when vscode-yaml is used, there's a custom schema provider active in yaml-language-server that sends a custom LSP request If you move the CRD resolving code into the
I have written a hack to get the correct URL for OpenShift CRDs, I'll also push this as a commit. |
Add an option to overwrite the default Kubernetes schema URL with a custom one, by setting a `YAMLLS_KUBERNETES_SCHEMA_URL` environment variable. See - redhat-developer#998 - redhat-developer#824 - redhat-developer#841 - redhat-developer#962 - redhat-developer#1050 Main use case is to use https://github.com/ricoberger/kubernetes-json-schema, which contains the schema for Kubernetes and the CRDs I'm using.
|
@qvalentin I think this feature is ready for merge after the changes I made. Let me know if you have objections to anything I changed or if it doesn't work for you. I'm planning on merging this Monday otherwise. |
automatically detect the Kubernetes schema based on the document's GroupVersionKind (GVK) and retrieve the matching schema from the CRD catalog.
The CRD schema resolving logic now runs even if a custom schema provider is present, allowing this to work properly in vscode-yaml. Signed-off-by: David Thompson <davthomp@redhat.com>
The CRD schemas for OpenShift are layed out slightly different from everything else in the CRD schema repo in order to provide different sets of schemas for different versions of OpenShift. Signed-off-by: David Thompson <davthomp@redhat.com>
See redhat-developer/yaml-language-server#1050 Settings to enable/disable the feature and change the repository used to find CRD schemas. Use https://raw.githubusercontent.com/nlamirault/crd-schema-store/refs/heads/main/schemas to test an alternate store of CRD schemas. Signed-off-by: David Thompson <davthomp@redhat.com>
* Settings for "auto-detect Kubernetes crd schema" See redhat-developer/yaml-language-server#1050 Settings to enable/disable the feature and change the repository used to find CRD schemas. Use https://raw.githubusercontent.com/nlamirault/crd-schema-store/refs/heads/main/schemas to test an alternate store of CRD schemas. Signed-off-by: David Thompson <davthomp@redhat.com> * Fix wording of enable setting Co-authored-by: Morgan Chang <shin19991207@gmail.com> * Fix wording of url setting Co-authored-by: Morgan Chang <shin19991207@gmail.com> * Add a quote that somehow got lost Signed-off-by: David Thompson <davthomp@redhat.com> --------- Signed-off-by: David Thompson <davthomp@redhat.com> Co-authored-by: Morgan Chang <shin19991207@gmail.com>
|
Thanks @datho7561 |
* updated eslint version Signed-off-by: msivasubramaniaan <msivasub@redhat.com> * changelog updated for the release v1.17.0 Signed-off-by: msivasubramaniaan <msivasub@redhat.com> * add language 'github-actions-workflow' * bumped v1.18.0 Signed-off-by: msivasubramaniaan <msivasub@redhat.com> * updated chagelog for the release 1.18.0 Signed-off-by: msivasubramaniaan <msivasub@redhat.com> * bump version 1.19.0 Signed-off-by: msivasubramaniaan <msivasub@redhat.com> * Fix build failure due to prettier error Fixes redhat-developer#1117 Signed-off-by: David Thompson <davthomp@redhat.com> * Stabilize UI tests This PR uses delays in order to get the "YAML custom tags works as expected" test passing consistently. My best guess as to why this works is that it takes bit of time for the setting change to propagate to the language server. Fixes redhat-developer#1119 Signed-off-by: David Thompson <davthomp@redhat.com> * Add formatting option for trailing comma See redhat-developer/yaml-language-server#1063, this PR requires that PR in order to work. Fixes redhat-developer#1112 Signed-off-by: David Thompson <davthomp@redhat.com> * update settings.json as default formatter as yaml Signed-off-by: msivasubramaniaan <msivasub@redhat.com> * send vscode configured language to LS * upstream node * handled locale on YAML LS side * removed yarn dependency and added npm * fixed build issue * removed .yarnrc * check yarn availability * removed yarn * removed yarn completely * removed yarn.lock * check yarn * removed the yarn * add -force instead of -f * removed yarn based on OS * fix winows CI build issue * removed -y(yarn) in the script * CHANGELOG for 1.19.0 Signed-off-by: David Thompson <davthomp@redhat.com> * l10n missing from .vsix 1. The `l10n` folder, which contains all the translations, was not present anywhere in the built .vsix. The translations are required in order to run yaml-language-server, so I copied from yaml-language-server to `dist/l10n`. 2. The location of l10n was listed as `../yaml-language-server/l10n`. This doesn't work if yaml-language-server isn't cloned locally, and doesn't work when running as a .vsix, so I changed it to `./dist/l10n`. 3. During the CI run, npm isn't installing the latest prerelease of yaml-language-server. I added a step to force it to use the latest yaml-language-server. This is important, because the above bugs could have been caught earlier if we were actually testing against the newest yaml-language-server. Fixes redhat-developer#1152 Signed-off-by: David Thompson <davthomp@redhat.com> * Upversion to 1.20.0 * Publish prereleases Fixes redhat-developer#1155 Signed-off-by: David Thompson <davthomp@redhat.com> * Revert "update settings.json as default formatter as yaml" This reverts commit 84d4516. Fixes redhat-developer#1157 Adjusting the user settings automatically was annoying to some users. The main bug fix, which consisted of setting the document selector to include Docker Compose and GitHub Action files, will not be reverted. * Fix prerelease GitHub Action I had the wrong path for the script to bump the version. Signed-off-by: David Thompson <davthomp@redhat.com> * Don't tag prereleases Fixes redhat-developer#1162 Signed-off-by: David Thompson <davthomp@redhat.com> * Add languages `yaml-textmate` & `yaml-tmlanguage` (redhat-developer#1093) * Add languages `yaml-textmate` & `yaml-tmlanguage` * Add to `activationEvents` * Fix scheduled prerelease skipping publish step The prerelease build skipped the publishing step. (See https://github.com/redhat-developer/vscode-yaml/actions/runs/18187139533/job/51773763520) This should prevent the publishing step from being skipped. Signed-off-by: David Thompson <davthomp@redhat.com> * Another fix for prereleases In order to skip tagging the prerelease version, we need to entirely skip the publish to GitHub Releases step. Passing a null tag causes the step to error, which ends the build. Signed-off-by: David Thompson <davthomp@redhat.com> * CHANGELOG for 1.19.1 Signed-off-by: David Thompson <davthomp@redhat.com> * Upversion to 1.20.0 Signed-off-by: David Thompson <davthomp@redhat.com> * feat: Register `ansible` and `ansible-jinja` languages for formatter (redhat-developer#1179) * feat: Register `ansible` and `ansible-jinja` languages for formatter * docs: Add changelog after getting issue number * refactor: Change yaml pattern and release tag to follow upstream * docs: Remove version entry in changelog * docs: Improve changelog * docs: Clear update entry * Don't syntax highlight `1.4.0` as a float Modify the TextMate grammar so that it doesn't count `1.4.0` as a floating point number. The current regular expression was taken from the YAML 1.1 spec, but the regular expression contradicts the examples and explanation and was likely an error. The new regular expression was taken from the YAML 1.2.2 spec. Notably, I believe that `1.4.0` is treated as a string in 1.1 and 1.2 YAML deserializers. It doesn't seem like we'll be able to upstream this change into TextMate, given there's an existing upstream PR to make this change that hasn't been merged. Fixes redhat-developer#901 Signed-off-by: David Thompson <davthomp@redhat.com> * Update CODEOWNERS Muthu is working on other tasks and unfortunately doesn't have time to work on yaml-language-server. For the time being, I've been handling PR reviews and managing issues. Updating the CODEOWNERS to reflect this will automatically request for me to review any new PRs that come in, and will skip pinging Muthu. Signed-off-by: David Thompson <davthomp@redhat.com> * Fix macOS CI UI tests (redhat-developer#1186) * await for progress bar to disappear * Update customTagsTest.ts * Update extensionUITest.ts * Update extensionUITest.ts * Update extensionUITest.ts * Update extensionUITest.ts * Update extensionUITest.ts * increase timeout * Update extensionUITest.ts * Improve documentation for the `yaml.schemas` setting Fixes redhat-developer#1147 Signed-off-by: David Thompson <davthomp@redhat.com> * Settings for "auto-detect Kubernetes crd schema" (redhat-developer#1192) * Settings for "auto-detect Kubernetes crd schema" See redhat-developer/yaml-language-server#1050 Settings to enable/disable the feature and change the repository used to find CRD schemas. Use https://raw.githubusercontent.com/nlamirault/crd-schema-store/refs/heads/main/schemas to test an alternate store of CRD schemas. Signed-off-by: David Thompson <davthomp@redhat.com> * Fix wording of enable setting Co-authored-by: Morgan Chang <shin19991207@gmail.com> * Fix wording of url setting Co-authored-by: Morgan Chang <shin19991207@gmail.com> * Add a quote that somehow got lost Signed-off-by: David Thompson <davthomp@redhat.com> --------- Signed-off-by: David Thompson <davthomp@redhat.com> Co-authored-by: Morgan Chang <shin19991207@gmail.com> * Add new setting to enable/disable anchor hover support See redhat-developer/yaml-language-server#1150 Draft PR until it gets merged. Signed-off-by: David Thompson <davthomp@redhat.com> * Fix running as a web extension, add smoke tests to prevent another regression (redhat-developer#1193) * On web, send a message to the worker containing the translations Translations weren't working at all when running the language server in a web worker. In fact, the server was crashing immediately, since the server was trying to load them from the filesystem. This PR provides the translations to the server using the webworker message passing mechanism. Requires redhat-developer/yaml-language-server#1165 Signed-off-by: David Thompson <davthomp@redhat.com> * Fix web support, add smoke tests 1. Fix schema content resolving when running on web (referencing a schema with a relative path wasn't working) 2. Add smoke tests for desktop using `@vscode/test-cli` 3. Add smoke tests for web using `@vscode/test-web` The smoke tests are very minimal; it loads a YAML file that references a schema using a relative path and verifies the expected diagnostics appear. Signed-off-by: David Thompson <davthomp@redhat.com> --------- Signed-off-by: David Thompson <davthomp@redhat.com> * Fix resolving schema content on web Use a new extension point, which requests schema content by URI instead of filepath. This is helpful, since web instances may use URIs with schemas other than `file://` to refer to the files in the workspace. Add a smoke test. Fixes redhat-developer#1194 Signed-off-by: David Thompson <davthomp@redhat.com> * Remove debug notification Missed removing it before I pushed the code, and it unfortunately got missed in review too. Signed-off-by: David Thompson <davthomp@redhat.com> * CHANGELOG for 1.20.0 Signed-off-by: David Thompson <davthomp@redhat.com> * Upversion to 1.21.0 Signed-off-by: David Thompson <davthomp@redhat.com> * Switch to 1.20.1 Given we're doing a release to address bugs introduced in 1.20.1, I think that we should mark the nexte version as a patch release. Signed-off-by: David Thompson <davthomp@redhat.com> * Fix documentation of `yaml.schemas` setting You can use a single glob or an array of globs. I've updated the documentation and JSON Schema to reflect this. Fixes redhat-developer#1207 Signed-off-by: David Thompson <davthomp@redhat.com> * Replaced the asterisk based file matching with explicit YAML language IDs (redhat-developer#1209) * replaced the asterisk based file matching with explicit YAML language IDs Signed-off-by: Morgan Chang <shin19991207@gmail.com> * remove unsupported language IDs Signed-off-by: Morgan Chang <shin19991207@gmail.com> --------- Signed-off-by: Morgan Chang <shin19991207@gmail.com> * make yaml.validate and yaml.format.enable language overridable Signed-off-by: Morgan Chang <shin19991207@gmail.com> * switch back to 1.21.0 * Add yaml-lint-disable comment to suppress diagnostics per-line Add support for suppressing linter diagnostics on a per-line basis using a yaml-lint-disable comment placed on the line immediately before the one producing the diagnostic. The comment supports three forms: suppress all diagnostics, suppress by message substring, or suppress by multiple comma-separated substrings (case-insensitive). Implemented via LSP client middleware that intercepts diagnostics from the yaml-language-server and filters out suppressed ones before they reach VS Code. Co-authored-by: Cursor <cursoragent@cursor.com> * Refine tests and JSDocs * Remove code * Rename yaml-lint-disable to yaml-language-server-disable * CHANGELOG for 1.21.0 (redhat-developer#1215) Signed-off-by: David Thompson <davthomp@redhat.com> Co-authored-by: Morgan Chang <shin19991207@gmail.com> * Upversion to 1.22.0 Signed-off-by: David Thompson <davthomp@redhat.com> * feat(yaml): add YAML language service features - Implemented YamlLinks service for finding document links in YAML files. - Added YamlOnTypeFormatting service for handling on-the-fly formatting in YAML. - Created YamlRename service to support renaming anchors in YAML documents. - Introduced YamlSelectionRanges service to provide selection ranges for YAML nodes. - Enhanced YamlSchemaService with new schema modification actions. - Added diagnostic filtering capabilities for suppressing specific diagnostics in YAML. - Implemented utility functions for handling YAML AST and scalar types. - Updated telemetry interface for error reporting. - Refactored text buffer utilities for better line handling. - Updated Kubernetes schema URL to the latest version. - Adjusted schema status bar item to work with eBuilder YAML language. * fix: outdate config * feat: add support for .eyaml and .eyml file extensions --------- Signed-off-by: msivasubramaniaan <msivasub@redhat.com> Signed-off-by: David Thompson <davthomp@redhat.com> Signed-off-by: Morgan Chang <shin19991207@gmail.com> Co-authored-by: msivasubramaniaan <msivasub@redhat.com> Co-authored-by: Muthurajan Sivasubramanian <93245779+msivasubramaniaan@users.noreply.github.com> Co-authored-by: Alberto Cavalcante <54247214+albertocavalcante@users.noreply.github.com> Co-authored-by: David Thompson <davthomp@redhat.com> Co-authored-by: RedCMD <33529441+RedCMD@users.noreply.github.com> Co-authored-by: Lanqing Huang <lqhuang@outlook.com> Co-authored-by: Morgan Chang <shin19991207@gmail.com> Co-authored-by: Simon Heather <simon.heather@yulife.com> Co-authored-by: Cursor <cursoragent@cursor.com>
What does this PR do?
This PR adds an option to automatically detect the Kubernetes schema by parsing the contents of the document. In case a Kubernetes GroupVersionKind (GVK) is detected, the matching schema is retrieved from the CRD catalog.
What issues does this PR fix or reference?
#605
Is it tested? How?
Tests for parsing the Kubernetes Group Version Kind have been added
Updated version of #962 including the following changes:
TODO: