Token Level Log

[yuhongsun96]

Description

Add option to log individual tokens. Needed for tracking why some citations fail to process

How Has This Been Tested?

Ran app

Accepted Risk (provide if relevant)

N/A

Related Issue(s) (provide if relevant)

N/A

Mental Checklist:

• All of the automated tests pass
• All PR comments are addressed and marked resolved
• If there are migrations, they have been rebased to latest main
• If there are new dependencies, they are added to the requirements
• If there are new environment variables, they are added to all of the deployment methods
• If there are new APIs that don't require auth, they are added to PUBLIC_ENDPOINT_SPECS
• Docker images build and basic functionalities work
• Author has done a final read through of the PR right before merge

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Nov 24, 2024 2:40am