Detect openapi definitions from hono-zod-openapi apps #369

brettimus · 2024-11-19T19:55:24Z

UPDATE TYPES PACKAGE AFTER MERGE
UPDATE HONO-OTEL AFTER MERGE

Description

Automatically report openapi spec from the client library to the studio api.

Whenever a route probe response is received, Studio will try to match any incoming routes with a corresponding openapi definition for that route.

If one is found, a json object describing the parameters and response for the route is then saved in the database on a special column for the route. This mini-spec can be used to generate request parameters, which is useful, since the source code analysis can't do much to help with OpenApiHono apps (more on that below).

The spec for a route, if present, also will render a Docs tab for that route.

Feature Flagged

Adds a section in the settings page where you can point to a route that hosts a json openapi definition.
This is a more flexible approach than auto-detecting the spec from the Hono-Zod-OpenAPI integration.

Fun things along the way

OpenApiHono is not picked up by source code analysis (makes sense, since it is not of type Hono)
Had to add a special header that allows the client library to skip instrumentation for an incoming request. Otherwise we'd enter an infinite loop when explicitly looking up a spec from the target api ((_note that the "custom spec url" setting is behind a feature flag for now))
Because of said infinite loop, had to finally add a limit to the number of traces we return from the api. (My infinite loop produced close to 90K new spans, which ground the UI to a halt
Basic auth is wonky in studio

Considerations

Manually pulling the spec versus automatically receiving it — My first implementation relied on the user explicitly telling us where their spec was located, then we would try to pull it in to find matching specs for routes. However, things are a lot smoother if we only support apps built with HonoOpenAPI for now. In that case, we can ignore the URL specified on the settings page. Also, if we want to support "give us a URL and we'll try syncing", then we need to be mindful of things like the ALL method. It's a lot more work to support a link to a spec instead of relying on the Hono OpenAPI integration to report it to us.

Autopopulated parameters — This is tricky given that when you click on a Route, we might end up redirecting you to the last-made-request for that route. In which case the expected behavior is not clear. Should we merge expected query parameters from the OpenAPI spec with the actual query parameters made in the last request?

Screenshots

TODO

Test headers
Show documentation for route somewhere
~~Handle .all method~~
~~Only allow openapi integration for versions of client library past 0.4.1~~
Expand schema definitions when possible into openApiSpec
Explain OpenAPI integration on the settings page
Auto-enable openapi integration when we can

pkg-pr-new · 2024-11-19T19:58:30Z

Open in Stackblitz

npm i https://pkg.pr.new/fiberplane/fpx/@fiberplane/studio@369

npm i https://pkg.pr.new/fiberplane/fpx/@fiberplane/hono-otel@369

commit: 00cb524

This reverts commit c24788a.

…from-hono-middleware

brettimus · 2024-11-21T07:33:59Z

api/src/lib/openapi/fetch.ts

+  const response = await fetch(resolvedSpecUrl, {
+    headers: {
+      // NOTE - This is to avoid infinite loops when the OpenAPI spec is fetched from Studio
+      //        We need to make sure that the user has instrumented their app with @fiberplane/hono-otel >= 0.4.1


this part is really crucial - we need to validate that the user has the correct version of hono-otel

…spec from app that does not have hono-otel 0.4.1

brettimus · 2024-11-21T08:28:28Z

api/src/routes/traces.ts

@@ -35,6 +35,7 @@ app.get("/v1/traces", async (ctx) => {
  const spans = await db.query.otelSpans.findMany({
    where: sql`inner->>'scope_name' = 'fpx-tracer'`,
    orderBy: desc(sql`inner->>'end_time'`),
+    limit: 1000,


this is a little left field but i finally hit the issue where i had so many spans it effectively broke the UI. i put a limit of 1000 here which seems okay but... who knows

brettimus · 2024-11-21T08:32:57Z

examples/openapi-zod/src/db/schema.ts

+  updatedAt: text("updated_at").notNull().default(sql`(CURRENT_TIMESTAMP)`),
+});
+
+// TODO - Figure out how to use drizzle with "@hono/zod-openapi"


worth noting: i wanted to use drizzle-zod to share schemas between the database and the api, in order to reuse a database schema as, e.g., the payload for a route

however, the openapi integration requires that you use a special version of zod that they export, which provides some additional fields that are necessary for the schema to be converted to an openapi schema

this forces you to redefine (as far as i can tell) the schemata, which is kind of good practice to separate api schemas from database schemas, but was a little bit of a pain for me considering i didn't care about that separation at this step

This reverts commit 3ee1a0d.

…i-definitions-directly-from-hono-middleware

…from-hono-middleware

flenter

lgtm

flenter · 2024-12-03T09:31:03Z

.gitignore

@@ -4,6 +4,9 @@ shared/dist
 .envrc
 .cursorrules

+# Cursor
+.cursorrules


This line already exists in this file

flenter · 2024-12-03T09:52:22Z

api/src/lib/openapi/openapi.ts

+ * This function handles both single routes and arrays of routes.
+ *
+ * @param db - LibSQL database instance containing the OpenAPI specifications. Used to fetch the latest spec.
+ * @param routes - Single route or array of routes to be enriched. Each route should contain path, method,


Maybe i'm reading this text the wrong way, but the way I understand this doc is as follows:

route: Route | Array<Route>

However it's always an array (though maybe with a singular route)

And that makes me think, I've seen in other prs, that the JSDocs are out of date even in the PR's that introduce them. Maybe it's better to leave the types out for parameters/return values? I do like the part of the JSDoc where you describe what a function is intended for but types/parameters/return values to me seems like we need to maintain an additional thing (like for instance if you refactor/rename a parameter you'd need to manually also update the docs).

flenter · 2024-12-03T09:56:26Z

examples/openapi-zod/src/index.ts

+const UserIdPathParamSchema = z.object({
+  id: z
+    // Path params are always strings
+    .string()


You can also do:
z.number({ coerce: true}) and the id will be converted to a number

wasn't sure how that'd play with validation — worth toying around with it

studio/src/pages/RequestorPage/CommandBar/CommandBar.tsx

flenter · 2024-12-03T09:58:51Z

studio/src/pages/RequestorPage/RequestPanel/RouteDocumentation/RouteDocumentation.tsx

+  return (
+    <ScrollArea className="h-full pb-8">
+      <div className="p-2">
+        {/* Description Section - Updated styling */}


Is this still relevant? I think it can go, not?

flenter · 2024-12-03T10:08:33Z

examples/openapi-zod/package.json

+    "db:studio": "drizzle-kit studio",
+    "db:touch": "wrangler d1 execute zod-openapi --local --command='SELECT 1'",
+    "db:migrate:prod": "GOOSIFY_ENV=production drizzle-kit migrate",
+    "db:studio:prod": "GOOSIFY_ENV=production drizzle-kit studio"


I would like it if we'd had some type checking/linting enabled also for our example projects.

…from-hono-middleware

brettimus added 4 commits November 19, 2024 19:21

Add simple openapi example app

23efcd5

Add settings page for openapispecurl

318eff7

Add openapi lib file

1c47ddf

Format

98de114

brettimus added 13 commits November 19, 2024 22:05

Hack in a way to save a simple openapi description for a route

93c8892

Fix bad import

4caab50

Add an openapi route for creating a user in the example

600e95e

Implement automagical query params for openapi routes

3fee795

Filter dead query params when changing routes

03c23d8

Dereference schema entries into route openapi definition used by studio

73a7bea

First pass at generating sample data from openapi schema for json bodies

b1ff630

Comment out automagical bodies

a0dadfe

Add triple dot on hover

c24788a

Revert "Add triple dot on hover"

24699ad

This reverts commit c24788a.

Merge branch 'main' into fp-3897-detect-openapi-definitions-directly-…

e678374

…from-hono-middleware

Write documentation for openapi dereferencing code

680c982

Specify in UI that we only support JSON specs

77b6716

brettimus force-pushed the fp-3897-detect-openapi-definitions-directly-from-hono-middleware branch from 5b8a9a8 to 77b6716 Compare November 21, 2024 07:30

brettimus commented Nov 21, 2024

View reviewed changes

brettimus added 4 commits November 21, 2024 08:41

Add error handler to the fetch to the openapi spec

ff791fe

Add tests for opneapi module

e6014ad

Add some basic caching to avoid infinite loops when fetching openapi …

5071b9d

…spec from app that does not have hono-otel 0.4.1

Remove comments

0b10f0b

brettimus commented Nov 21, 2024

View reviewed changes

brettimus added 4 commits November 21, 2024 09:43

Add small changelog entry

80c1b10

Fix logic that adds openapi specs to routes (fallback to routes)

7e59890

Fix openapi params that specified numeric path param

3ee1a0d

Revert "Fix openapi params that specified numeric path param"

b4bb733

This reverts commit 3ee1a0d.

brettimus added 9 commits November 29, 2024 13:32

Add descriptions to openApiSpec fields in schemaProbedRoutes

99c3085

Improve documentation and typing of the spec fetching

9ce40d4

Remove dead code

5aab896

Remove dead code

c23c52f

Update the hono-zod-openapi example

f0ce6e8

Update comments about openapi types in the types pacakge

09ade02

Update changelog

e03db0b

Put the custom spec url feature behind a feature flag

0a23055

Fix visible tabs logic

d78c7dd

brettimus force-pushed the fp-3897-detect-openapi-definitions-directly-from-hono-middleware branch from 0bf9c98 to d78c7dd Compare November 29, 2024 13:32

brettimus added 6 commits November 29, 2024 15:24

Stylize the docs tab a little tiny bit better

b5f11ee

more more styles

7a7d74b

Add support for arrays to the routeDocumentation tab

da59ce6

Update README for openapi example

1c50ca9

Fix accessibility warnings in the command bar

133e1b4

Minor tweaks

3732bc6

brettimus changed the title ~~WIP: Detect openapi definitions~~ Detect openapi definitions from hono-zod-openapi apps Nov 29, 2024

brettimus marked this pull request as ready for review November 29, 2024 17:00

brettimus added 3 commits November 29, 2024 18:01

Merge remote-tracking branch 'origin/main' into fp-3897-detect-openap…

4954572

…i-definitions-directly-from-hono-middleware

Merge branch 'main' into fp-3897-detect-openapi-definitions-directly-…

94faf88

…from-hono-middleware

Update package json version for client lib

44c91d7

flenter approved these changes Dec 3, 2024

View reviewed changes

flenter reviewed Dec 3, 2024

View reviewed changes

brettimus added 7 commits December 3, 2024 15:11

Merge branch 'main' into fp-3897-detect-openapi-definitions-directly-…

66a00c8

…from-hono-middleware

Update openapi description for new user route

9aa2d9c

Update styles for docs tab

1c84cc2

Remove dead code

961b315

Fix logic for opening ai prompt guidance txt input

df590aa

Validate openapi operation in the requestpanel

34b5105

Bump api package.json

00cb524

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect openapi definitions from hono-zod-openapi apps #369

Detect openapi definitions from hono-zod-openapi apps #369

brettimus commented Nov 19, 2024 •

edited

Loading

pkg-pr-new bot commented Nov 19, 2024 •

edited

Loading

brettimus Nov 21, 2024

brettimus Nov 21, 2024

brettimus Nov 21, 2024

flenter left a comment

flenter Dec 3, 2024

flenter Dec 3, 2024

flenter Dec 3, 2024

brettimus Dec 3, 2024

flenter Dec 3, 2024

flenter Dec 3, 2024

Detect openapi definitions from hono-zod-openapi apps #369

Are you sure you want to change the base?

Detect openapi definitions from hono-zod-openapi apps #369

Conversation

brettimus commented Nov 19, 2024 • edited Loading

Description

Feature Flagged

Fun things along the way

Considerations

Screenshots

TODO

pkg-pr-new bot commented Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flenter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brettimus commented Nov 19, 2024 •

edited

Loading

pkg-pr-new bot commented Nov 19, 2024 •

edited

Loading