Skip to content

Add yaml support#227

Merged
ahl merged 1 commit into
oxidecomputer:mainfrom
jayvdb:support-yaml
Jan 14, 2023
Merged

Add yaml support#227
ahl merged 1 commit into
oxidecomputer:mainfrom
jayvdb:support-yaml

Conversation

@jayvdb

@jayvdb jayvdb commented Nov 7, 2022

Copy link
Copy Markdown
Contributor

No description provided.

@jclulow

jclulow commented Nov 7, 2022

Copy link
Copy Markdown
Collaborator

I think it would be better to make the format choice explicit using a flag of some kind (e.g., -f, --format <FORMAT>) rather than trying to guess by looking in the file, so that you would then invoke:

$ progenitor -f yaml -i your/openapi/document ...

The default value for -f would be json if not specified, for compatibility.

@ahl

ahl commented Nov 8, 2022

Copy link
Copy Markdown
Collaborator

My understanding is that yaml is a superset of json--why would we require users to specify the file type if we don't have to?

@jclulow

jclulow commented Nov 8, 2022

Copy link
Copy Markdown
Collaborator

I think it is better to be strict. If what I believe that I have is a JSON file, I don't want it to be accidentally mis-parsed as YAML if it is not valid for some reason.

@jayvdb

jayvdb commented Nov 8, 2022

Copy link
Copy Markdown
Contributor Author

Note other readers often use some sort of trick to automate this, either file extension, looking for {, and HTTP content-type.
e.g. https://github.com/paperclip-rs/paperclip/blob/43cfea1/src/v2/mod.rs#L110
https://github.com/kstasik/schema-tools/blob/aadd4b5/src/schema.rs#L70
https://github.com/maxdeviant/sdkgen/blob/58db7b41/src/sdkgen/src/main.rs#L56

Having an automated approach becomes more important when implementing #228 , as the external files may be using a different format than the initial file/url specified on the CLI. e.g. a YAML file may refer to a JSON file.

I suggest the following order for determining the format:

  1. an explicit CLI arg,
  2. content-type (would exist at this level when Support external $ref #228 is implemented)
  3. file extension,
  4. detect using {

i.e. the CLI arg would not default to json, but if supplied it overrides all.

@ahl

ahl commented Nov 20, 2022

Copy link
Copy Markdown
Collaborator

I think it is better to be strict.

I think it it sometimes better to be string and sometimes better to be relaxed. Admittedly this is a harder position to navigate.

If what I believe that I have is a JSON file, I don't want it to be accidentally mis-parsed as YAML if it is not valid for some reason.

What negative consequence would you imagine happening?

@ahl ahl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're doing this for the command, we should do this for the macro.

Off the cuff, I think my preferences for inferring the type of the file would be

  1. the file name
  2. try it as json, fall back to yaml

I don't love option 1. It's kind of gross, and I have plans for the macro that means that the file name may not be as readily available.

I do like option 2 even though it's inefficient:

  • If the file is JSON but there's an error, the serde_yaml error will be basically the same
  • serde_json is a lot faster so I don't want to just use serde_yaml for everything even though it would work
  • Slowing down the error case for a badly formatted file would be fine.

Comment thread progenitor/src/main.rs Outdated
let mut f = File::open(p)?;

let mut buf = [b' '];
while buf[0].is_ascii_whitespace() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't seem right: a json file could start with whitespace, couldn't it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loops ignores all leading whitespace in the file, so that the if below is checking whether the first non-whitespace character is {.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right! fair enough. Sounds like you prefer this to the options I mentioned?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, IMO we need some auto-detection to handle #228 in order to still do the right thing if the content-type isnt recognised , and often raw json & yaml files have strange content-types provided by the server.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, agreed. I suggested that we try to parse as json and then, failing that, parse as yaml. Would that not suffice?

@jayvdb jayvdb Jan 8, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I like that, and will do that next time I revisit this PR - should be later this week.

@ahl ahl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather have some non-keeper yaml file since otherwise it will be annoying to sync up the json and yaml files as folks add test cases to them

Comment thread progenitor-impl/tests/test_output.rs Outdated
Comment on lines +13 to +17
if openapi_file.contains('.') {
in_path.push(openapi_file);
} else {
in_path.push(format!("{}.json", openapi_file));
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just add .json to the callers

Comment thread progenitor-macro/src/lib.rs Outdated
Comment on lines +37 to +38
/// The `spec` key is required; it is the OpenAPI document path from which the
/// client is derived, and may be either JSON or YAML.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// The `spec` key is required; it is the OpenAPI document path from which the
/// client is derived, and may be either JSON or YAML.
/// The `spec` key is required; it is the path of the OpenAPI document (JSON or YAML) from which the client is derived.

Comment thread progenitor/src/lib.rs Outdated
//! README](https://github.com/oxidecomputer/progenitor/README.md)

pub use progenitor_client;
pub use progenitor_impl::util::load_api;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be in the progenitor public API

Comment thread progenitor-impl/src/lib.rs Outdated
Comment on lines +29 to +32
#[error("invalid file location {0}")]
InvalidFileLocation(String),
#[error("unrecognised file contents {0}")]
InvalidFileContents(String),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if load_api isn't public we don't need these error cases. I'd prefer copying the code in both progenitor/src/main.rs and progenitor-macro

At some point, progenitor-macro will switch to something that uses include_str!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, ok 👍

@ahl ahl merged commit 1ef131a into oxidecomputer:main Jan 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants