Add explicit prompt/completion logging/attributes, skip tracing for synchronous components

[petersalas]

1. Add explicit (conversational) prompt/completion logging and span attributes, including token counts
2. Skip OpenTelemetry tracing for non-root synchronous components, which drastically simplifies the traces
3. Remove universal ai.jsx.result.tokenCount span attribute, which doesn't have the opportunity to handle conversational messages correctly

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
ai-jsx-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 22, 2023 4:20pm
ai-jsx-nextjs-demo	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 22, 2023 4:20pm
ai-jsx-tutorial-nextjs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 22, 2023 4:20pm

[petersalas]

@NickHeiner do you think this tracing/logging is good/comprehensive enough to remove the full render logging/token count attribute? Here's an example trace from the use-tools example

[NickHeiner]

@NickHeiner do you think this tracing/logging is good/comprehensive enough to remove the full render logging/token count attribute? Here's an example trace from the use-tools example

Maybe I'm missing something, but where do we see how many tokens UseTools resolved to?

[petersalas]

Maybe I'm missing something, but where do we see how many tokens UseTools resolved to?

You can't -- my claim is that token counts are only meaningful in the context of a single model, so the token counts are only available on the <OpenAIChatModel> spans. In particular with things like context-window-trimming, the relationship between what <UseTools> renders to and the underlying model calls is non-trivial. (Also, which tokenizer should be used? What if different models are used?)

That said, a more well-defined thing we could do is aggregate token usage as reported by the descendant model calls, but that's a different beast. (My inclination is also to do that aggregation downstream rather than in the code itself.)

[NickHeiner]

Maybe I'm missing something, but where do we see how many tokens UseTools resolved to?

You can't -- my claim is that token counts are only meaningful in the context of a single model, so the token counts are only available on the <OpenAIChatModel> spans. In particular with things like context-window-trimming, the relationship between what <UseTools> renders to and the underlying model calls is non-trivial. (Also, which tokenizer should be used? What if different models are used?)

That said, a more well-defined thing we could do is aggregate token usage as reported by the descendant model calls, but that's a different beast. (My inclination is also to do that aggregation downstream rather than in the code itself.)

Ok, great point that the tokenizer means the count is only meaningful within the context of a single model.

Broadly, the thing I'm trying to support is the ability to ask "how many tokens did I spend on docs vs API responses". My original solution was to give an output size for every component, but I agree with your points that there are some problems with that. Could we address that with your downstream aggregation suggestion?

[NickHeiner]

I disagree with some aspects of how this is being done but don't want to block on it.

Can you update https://docs.ai-jsx.com/guides/observability to reflect these changes?

And perhaps add one or two more unit tests?

[petersalas]

Broadly, the thing I'm trying to support is the ability to ask "how many tokens did I spend on docs vs API responses". My original solution was to give an output size for every component, but I agree with your points that there are some problems with that. Could we address that with your downstream aggregation suggestion?

IIUC I think that scenario is already addressed with the newly added ai.jsx.prompt attribute, which shows each conversation message given to the LLM and its token count.