New Render API #224

NickHeiner · 2023-07-31T14:38:28Z

Our current Render API has a few challenges.

We need to list some constraints that may not be immediately obvious to people.
Some people are less familiar with generator semantics.
Is it weird to have a value that's both a promise and an async generator?
AI.createRenderContext().render is odd because wtf is a render context?
Other output methods (e.g. toTextStream, toReactStream, etc) are scattered.
If you want deltas, you need to explicitly do the frame.slice(lastValue.length) pattern.

To address this, I started sketching a new API that wraps the old one.

Downsides of the new approach:

With AI.createRenderContext().render, if you want to render within a component, it's more natural to use componentContext.render, rather than reimporting the top-level API. If I simplify that API, that natural barrier erodes.

I think this may be valuable as the top-level render API. For the "within components" rendering – maybe? If we move forward with this, I suspect we'd wrap the current top-level render API, rather than trying to use this for all render internals.

vercel · 2023-07-31T14:38:34Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
ai-jsx-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 31, 2023 3:34pm
ai-jsx-nextjs-demo	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 31, 2023 3:34pm
ai-jsx-tutorial-nextjs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 31, 2023 3:34pm

packages/ai-jsx/src/lib/openai.tsx

NickHeiner · 2023-07-31T14:39:24Z

packages/ai-jsx/src/lib/openai.tsx

+  iteration++;
+  await sleep();
+  yield `first ${iteration} `
+  await sleep();
+  yield `second ${iteration} `
+  await sleep();
+  yield `third ${iteration}`
+  return AI.AppendOnlyStream;


This is because I was working on this on the plane with no wifi.

NickHeiner · 2023-07-31T14:39:30Z

packages/ai-jsx/src/lib/openai.tsx

@@ -446,6 +470,7 @@ export async function* OpenAIChatModel(

  let delta = await advance();
  while (delta !== null) {
+    // @ts-ignore


This will be removed.

NickHeiner · 2023-07-31T14:39:59Z

packages/examples/src/simple-chat.tsx

This is just a hack to show the idea. If we move forward, I'll clean it up / move it somewhere more sensible.

NickHeiner · 2023-07-31T14:41:50Z

packages/examples/src/simple-chat.tsx

+
+// This all needs to be validate with genuine async.
+
+const renderable = <App />


This section and below demonstrate how this API would be used.

NickHeiner · 2023-07-31T14:50:24Z

packages/examples/tsconfig.json

@@ -1,13 +1,14 @@
 {
  "extends": "@tsconfig/node18/tsconfig.json",
-  "include": ["src/**/*", ".eslintrc.cjs"],
+  "include": ["src/simple-chat.tsx", ".eslintrc.cjs"],


This was just for local testing.

petersalas

This seems like a mix of:

Grouping what are effectively existing "entry-point" type APIs
Tweaking of semantics for the existing render
Facilitating consumption of the same render stream in multiple formats

But 1 and 2 feel like they can be done directly (e.g. refactoring and/or organizing imports, changing the existing implementation), and I'm not sure what the motivation for 3 is.

At a high level, I agree there are pain points here, but I'd rather solve them more directly than by adding another layer on top of the existing stuff, if that makes sense.

petersalas · 2023-07-31T15:57:46Z

packages/examples/src/simple-chat.tsx

+      }
+      done = true;
+      resolveFinalResult!(lastFrame);
+    })();


All things equal I think it's preferable to let consumers pull from the stream rather than proactively flushing it to an EventEmitter

petersalas · 2023-07-31T15:58:30Z

packages/examples/src/simple-chat.tsx

+    const finalResult = new Promise((resolve) => {
+      resolveFinalResult = resolve;
+    });
+    const eventEmitter = new EventEmitter();


With EventEmitter if there are no listeners I assume events are dropped, is that correct?

petersalas · 2023-07-31T16:00:26Z

packages/examples/src/simple-chat.tsx

+  await Promise.all([
+    streamToValues(rendered.treeStream()),
+    streamToValues(rendered.treeStream()),
+    streamToValues(rendered.deltaStream()),
+  ])


What's the scenario where one wants to consume the same stream in multiple formats/multiple times? That feels like it should inform what the right semantics are, i.e. should they operate independently or in sync?

petersalas · 2023-07-31T16:00:48Z

packages/examples/src/simple-chat.tsx

+
+console.log();
+console.log('=== Second Render ===');
+// Pass a single map function, and it's applied to intermediate and the final results.


This could (should?) just be a change to the existing render?

petersalas · 2023-07-31T16:01:25Z

packages/examples/src/simple-chat.tsx

+    streamToValues(rendered.deltaStream()),
+  ])
+);
+// If you consume a stream after the render is complete, you just get one chunk with the final result.


This could also just be a change to the existing render?

NickHeiner · 2023-08-02T22:28:08Z

This seems like a mix of:

Grouping what are effectively existing "entry-point" type APIs

Tweaking of semantics for the existing render

Facilitating consumption of the same render stream in multiple formats

But 1 and 2 feel like they can be done directly (e.g. refactoring and/or organizing imports, changing the existing implementation), and I'm not sure what the motivation for 3 is.

At a high level, I agree there are pain points here, but I'd rather solve them more directly than by adding another layer on top of the existing stuff, if that makes sense.

I'd be happy to address some of the issues by changing the existing render semantics. I did this separately in part to make it easier to demo, but also because I wasn't sure if changing the existing semantics would break some other cases. (Like, is there a reason that map only applies to intermediate results?)

For (3), I don't expect it to be a common case, but I also felt like it needed to be called out in the docs as footguns, so I would just as soon avoid it.

NickHeiner added 3 commits July 29, 2023 17:38

Hackin

45f5aed

Expanding to tree/delta streaming

c6d31ae

Hackin

1e67296

vercel bot had a problem deploying to Preview – ai-jsx-nextjs-demo July 31, 2023 14:38 Failure

NickHeiner commented Jul 31, 2023

View reviewed changes

packages/ai-jsx/src/lib/openai.tsx Outdated Show resolved Hide resolved

NickHeiner commented Jul 31, 2023

View reviewed changes

vercel bot had a problem deploying to Preview – ai-jsx-tutorial-nextjs July 31, 2023 14:41 Failure

NickHeiner commented Jul 31, 2023

View reviewed changes

Self-review

552efac

vercel bot had a problem deploying to Preview – ai-jsx-docs July 31, 2023 14:45 Failure

vercel bot had a problem deploying to Preview – ai-jsx-tutorial-nextjs July 31, 2023 14:45 Failure

vercel bot had a problem deploying to Preview – ai-jsx-nextjs-demo July 31, 2023 14:45 Failure

[Automated] Format

7a3e66a

NickHeiner commented Jul 31, 2023

View reviewed changes

vercel bot deployed to Preview – ai-jsx-docs July 31, 2023 14:55 View deployment

vercel bot deployed to Preview – ai-jsx-nextjs-demo July 31, 2023 14:55 View deployment

vercel bot deployed to Preview – ai-jsx-tutorial-nextjs July 31, 2023 14:55 View deployment

NickHeiner and others added 6 commits July 31, 2023 11:20

Fixing issues and cleanup

9626741

Fixing issues and cleanup

faea4ff

More fixes

daf5d08

Reset to main

00860cc

Enable more "test cases"

02faa1f

Merge branch 'main' into nth/new-render-api

92d1d1a

NickHeiner requested a review from petersalas July 31, 2023 15:24

vercel bot deployed to Preview – ai-jsx-tutorial-nextjs July 31, 2023 15:33 View deployment

vercel bot deployed to Preview – ai-jsx-docs July 31, 2023 15:33 View deployment

vercel bot deployed to Preview – ai-jsx-nextjs-demo July 31, 2023 15:33 View deployment

petersalas reviewed Jul 31, 2023

View reviewed changes

NickHeiner closed this Sep 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Render API #224

New Render API #224

NickHeiner commented Jul 31, 2023 •

edited

Loading

vercel bot commented Jul 31, 2023 •

edited

Loading

NickHeiner Jul 31, 2023

NickHeiner Jul 31, 2023

NickHeiner Jul 31, 2023

NickHeiner Jul 31, 2023

NickHeiner Jul 31, 2023

petersalas left a comment

petersalas Jul 31, 2023

petersalas Jul 31, 2023

petersalas Jul 31, 2023

petersalas Jul 31, 2023

petersalas Jul 31, 2023

NickHeiner commented Aug 2, 2023


		// This all needs to be validate with genuine async.

		const renderable = <App />

New Render API #224

New Render API #224

Conversation

NickHeiner commented Jul 31, 2023 • edited Loading

vercel bot commented Jul 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petersalas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickHeiner commented Aug 2, 2023

NickHeiner commented Jul 31, 2023 •

edited

Loading

vercel bot commented Jul 31, 2023 •

edited

Loading