[PRFC] OpenAPI Typings for Routers by sennyeya · Pull Request #15667 · backstage/backstage

sennyeya · 2023-01-10T18:23:59Z

Signed-off-by: Aramis Sennyey sennyeya@amazon.com

Hey, I just made a Pull Request!

PoC work to create OpenAPI documentation and integration with express-openapi library using a schema-first approach. There are a few issues with this initial work that are pretty clear around dependent services being passed into the catalog and parsing input formats.

Update: Attempted to step back from express-openapi as typings are non-existent in the library. Their validators will be useful but the primary library has minimal use wrt developer experience. Found this repository https://github.com/varanauskas/oatx which handles automatic OpenAPI 3.x spec definitions in types and adapted it to a custom express router allowing for type-verified request bodies and response bodies. There will be more work to verify params and headers.
Basic workflow would be,

Write/update OpenAPI spec file.
Convert file to JSON and update/create typescript file to export the definition as const.
Import definition from the TS file and use to type the router.

Demo: https://drive.google.com/file/d/1uE2fAE9kjTezELFqwS_NUwThbWxmbJJ8/view?usp=share_link.

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

backstage-goalie · 2023-01-10T18:24:24Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/backend-openapi-utils	packages/backend-openapi-utils	patch	`v0.0.0`
@backstage/repo-tools	packages/repo-tools	minor	`v0.1.4-next.1`
@backstage/plugin-catalog-backend	plugins/catalog-backend	patch	`v1.8.1-next.2`

taras · 2023-01-10T20:16:15Z

@sennyeya nice! Did you write that OpenApi Spec file?

sennyeya · 2023-01-10T21:14:57Z

@taras Not for this iteration, I pulled the one I generated with #15038 and fixed a bunch of the invalid fields that it created. Wanted to highlight the issues with dynamic options before committing to writing a full spec.

taras · 2023-01-11T20:14:22Z

What does validateDependency do?

A small nitpick - is it possible to keep the typescript checks the same if (refreshService) { ... } instead of refreshService!. It makes the diff look bigger than it is and makes the comparison more difficult.

Which of the tools do you think we can use to generate typescript types?

sennyeya · 2023-01-11T21:26:50Z

is it possible to keep the typescript checks the same if (refreshService) { ... } instead of refreshService!. It makes the diff look bigger than it is and makes the comparison more difficult.

This is related to validateDependency. Because we can't use an if(refreshService) to remove routes, we would need to do if(refreshService) for each operation. validateDependency does this work and throws if refreshService is falsy.

I'm taking a look at the typescript type generation, I think I want to use quicktypes to generate from an OpenAPI schema, ie pulling the components list from an OpenAPI file. It'll add the types, for now duplicating as needed from catalog-model, but won't have any access/type safety with the API methods.

taras · 2023-01-11T21:30:44Z

API methods are useful types, IMO. It's also not guaranteed that components will be used. I think we need a tool that generates types starting from the operations.

sennyeya · 2023-01-11T22:14:29Z

Definitely agree API methods are useful typings. As a dev, I would definitely prefer typed request arguments as opposed to the casting/AJV validation we're currently doing manually.

However, I don't think schema-first is the right starting place for this PoC, especially as there are 7 distinct different schemas (plus however many permutations) we should be presenting, ie 1 when the catalog is in readonly mode and 6 for various dependent services being falsy when passed in. We could also make the schema as general as possible, ie have return values that include 404, but that seems overly broad and wouldn't help consumers make simple decisions like is the POST analyze-location or the DELETE /locations/:id available by just looking at the spec.

freben · 2023-01-17T15:47:17Z

I would completely ignore the concept of optionality, it's just a distraction. Just make it the full interface no matter what. The server can choose to return 501 Not Implemented if they want to. And end users can choose to have an export type CatalogClient = Omit<OriginalCatalogClient, 'deleteLocation'> or whatnot.

freben · 2023-01-17T15:48:55Z

or if it's possible to split apart the large catalog api into e.g. an entities part, a locations part etc (possibly separate schemas) and then generating stubs and types separately for them, and then by hand doing export type CatalogApi = EntitiesApi & LocationsApi & ...

sennyeya · 2023-01-26T17:29:53Z

@freben Apologies for the late response. Had to think through this some more.

I think the question I should be asking is whether we want to generate an SDK that users can consume that's published to PyPI/Maven/other package managers that has all available methods or if we want adopters to generate an SDK from the CLI that they can distribute themselves? My thinking was towards the second option.

If the second option, then I think we'd want to consider optionality.

github-actions · 2023-02-02T17:36:00Z

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

github-actions · 2023-02-16T16:18:40Z

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

freben · 2023-02-22T10:55:33Z

As a hint, you generally don't need to do much with the yarn file by hand since it's auto generated and deduping should happen in precommit. Most commonly, especially when rebasing, I just checkout the yarn.lock file from its master state before the head of the branch (do not delete it outright!) and run yarn install, letting yarn rebuild it properly for me.

github-actions · 2023-02-22T22:13:23Z

Uffizzi Preview deployment-17137 was deleted.

sennyeya · 2023-02-22T22:38:19Z

        content:
          application/json:
            schema:
-              $ref: '#/components/schemas/RefreshOptions'


Request bodies shouldn't generally be refs unless they are reused in multiple places. If they're reused they should be keyed to a requestBodies key under components not just in the components.schema key.

https://swagger.io/docs/specification/describing-request-body/

sennyeya · 2023-02-22T22:40:48Z

      required:
        - url
      description: A link to external information that is related to the entity.
+      additionalProperties: false


JSON Schema doesn't restrict additional properties by default, but I think this makes sense here as we're expecting types to match the defined interfaces and whatnot when interacting with express handlers. Without this, the json schema library we're using adds a [x: string]: unknown key to all objects it parses which causes type issues with interfaces.

This may have issues further down the line with validator middleware, but that's pretty speculative.

sennyeya · 2023-02-22T22:49:16Z

@freben That took waaay more tries than I thought it would 😅 . Hopefully, it's in the right dedup'ed state now?

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

freben · 2023-04-05T12:44:38Z

(just rebasing and ensuring that yarn.lock doesn't have conflicts)

github-actions · 2023-04-06T13:03:39Z

Thank you for contributing to Backstage! The changes in this pull request will be part of the 1.13.0 release, scheduled for Tue, 18 Apr 2023.

github-actions Bot added the area:catalog Related to the Catalog Project Area label Jan 10, 2023

backstage-goalie Bot assigned freben and UsainBloot Jan 10, 2023

sennyeya mentioned this pull request Jan 10, 2023

OpenAPI spec for the REST API #2566

Closed

RobotSail mentioned this pull request Feb 1, 2023

🚀 Feature: Support for tRPC as an API type #15877

Closed

2 tasks

github-actions Bot added the stale label Feb 2, 2023

freben removed the stale label Feb 3, 2023

Rugvip requested a review from freben February 9, 2023 16:13

github-actions Bot added the stale label Feb 16, 2023

sennyeya force-pushed the express-openapi-test branch from 6d49081 to f423eb5 Compare February 16, 2023 21:34

sennyeya changed the title ~~express-openapi Integration for catalog-backend~~ [DX] OpenAPI Router Integration for catalog-backend Feb 16, 2023

github-actions Bot removed the stale label Feb 16, 2023

sennyeya force-pushed the express-openapi-test branch from d4d2390 to f307f37 Compare February 22, 2023 17:54

sennyeya commented Feb 22, 2023

View reviewed changes

sennyeya and others added 22 commits April 5, 2023 14:42

Add changeset.

62fe726

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Add role checks.

be5aca5

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Remove generate and verify from next steps.

4d72efb

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Remove unused dependencies.

8bdc62e

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Fix yarn.lock file.

48776ad

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Update schema to get rid of nullable which isn't valid 3.1.

759ba30

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Fix schema.

3be54cd

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Update spec with better parameters and fix conflict in yarn.lock.

577f36c

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Update schema.

4efe0a0

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Update api report and shift openapi-utils to a devDependency.

abd4ff4

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Add CLI command for full repo check.

6a98560

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Update yarn.lock.

c62081e

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Update to devdependency.

53e6c46

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Move CLI tooling to repo-tools.

167434d

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Need to escape oatx.

f14d03d

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Rename to package/backend-openapi-utils.

1f5800a

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Lint on generate success. Move YAML file to src/schema/

ef77092

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Add export for core as a whole type.

29a44b8

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Fix import.

8b1e62a

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Fix API report.

d1a9e11

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

Fix dependency issue.

d7d059e

Signed-off-by: Aramis Sennyey <sennyeya@amazon.com>

minor fixup

cd9296f

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

freben force-pushed the express-openapi-test branch from e16a64f to cd9296f Compare April 5, 2023 12:43

freben merged commit 7f0dd6f into backstage:master Apr 6, 2023

jhaals mentioned this pull request Apr 18, 2023

Version Packages #17329

Merged

freben mentioned this pull request Apr 20, 2023

🚀 Feature: Provide Backstage API documentation #17418

Closed

2 tasks

This was referenced Apr 21, 2023

[Meta Issue]: OpenAPI Support #17482

Open

Org Member: sennyeya backstage/community#92

Closed

Conversation

sennyeya commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Hey, I just made a Pull Request!

✔️ Checklist

Uh oh!

backstage-goalie Bot commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changed Packages

Uh oh!

taras commented Jan 10, 2023

Uh oh!

sennyeya commented Jan 10, 2023

Uh oh!

taras commented Jan 11, 2023

Uh oh!

sennyeya commented Jan 11, 2023

Uh oh!

taras commented Jan 11, 2023

Uh oh!

sennyeya commented Jan 11, 2023

Uh oh!

freben commented Jan 17, 2023

Uh oh!

freben commented Jan 17, 2023

Uh oh!

sennyeya commented Jan 26, 2023

Uh oh!

github-actions Bot commented Feb 2, 2023

Uh oh!

github-actions Bot commented Feb 16, 2023

Uh oh!

freben commented Feb 22, 2023

Uh oh!

github-actions Bot commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sennyeya Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sennyeya Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sennyeya commented Feb 22, 2023

Uh oh!

freben commented Apr 5, 2023

Uh oh!

github-actions Bot commented Apr 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sennyeya commented Jan 10, 2023 •

edited

Loading

backstage-goalie Bot commented Jan 10, 2023 •

edited

Loading

github-actions Bot commented Feb 22, 2023 •

edited

Loading

sennyeya Feb 22, 2023 •

edited

Loading

sennyeya Feb 22, 2023 •

edited

Loading