Release 1.5.1 (#145)

* Pagination bug * Bug fix * chore: add docker cmd * Compatibility fixes for SDK version 2.0.0 (#69) * Pagination bug * Bug fix * Fix for schema changes * Render tool calling * Support for Langgraph, Qdrant & Groq (#73) * Pagination bug * Bug fix * Add langgraph support * QDrant support * Add Groq support * update README * update README * feat: optimise docker image for self host setup * adding api access to traces endpoint * clean up * refactor * feat: add clickhouse db create on app start (#79) * docs: add railway deploy, fix sdk badges (#81) * untrack .env * Revert "untrack .env" This reverts commit 4551d7e. * Playground and Prompt Management (#83) * Pagination bug * Bug fix * Playground - basic implementation * Playground - streaming and nonstreaming * Move playground inside project * API key flow * Api key * Playground refactor * Add chat hookup * anthropic streaming support * Bug fixes to openai playground * Anthropic bugfixes * Anthropic bugfix * Cohere first iteration * Cohere role fixes * Cohere api fix * Parallel running * Playground cost calculation non streaming * playground - streaming token calculation * latency and cost * Support for Groq * Add model name * Prompt management views * Remove current promptset flow * Prompt management API hooks * Prompt registry final * Playground bugfixes * Bug fix playground * Rearrange project nav * Fix playground * Fix prompts * Bugfixes * Minor fix * Prompt versioning bugfix * Bugfix * fix: clickhouse table find queries (#82) * Fix to surface multiple LLM requests inside LLM View (#84) * Pagination bug * Bug fix * Fix for surfacing multiple LLM requests in LLMView * Minor bugfixes (#86) * Pagination bug * Bug fix * Bugfixes * api to fetch promptset with prompt filters * bug fixes * fix invalid redirect * fix invalid status code * Project Switcher (#90) * Pagination bug * Bug fix * Project switcher * Feat: dataset download (#60) * API: download dataset * API: Download dataset * updated download-dataset api * Updated: download_dataset api * Updated download dataset API * Updated Download API: changed Response to Next Response, add a condition to ensure max page size is 500 * updated the download-dataset API: fixed the format and removed redundant lines of code * Updated download_daatset API: file name and removed 'id' param * Added the Download dataset button. * Merged developemnt into my branch * Updated button size * Fixes --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * Update prompt registry with instructions to fetch prompts (#91) * Pagination bug * Bug fix * Update prompt registry * Minor bugfix (#94) * Pagination bug * Bug fix * Minor bugfix * chore: update github repo badges * optimizing token count function * Add GPT4-O Pricing and Playground (#98) * Pagination bug * Bug fix * Add GPT4-O support * Add GPT4-O support * Update cost * Dylan/s3en 2234 add perplexity support to playground (#89) * adding perplexity to playground types * adding ui stuff:' * adding perplexity chat api * fixing perplexity model dropdown --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * api changes * add api access to get api and fix all bugs * bug fix * bug fix * updating descriptions to optional * prio python * cleanup and fixes * more bug fixes * more fixes * remove console log * updating trace_service functions * add migration * add format function, updating from day to hour * adding dropwdown menu * updating query key * updating query keys v2 * clean up * fix bug * Minor bugfix (#102) * Pagination bug * Bug fix * Minor bugfix * fixes * Bug fixes to Evaluations (#111) * Pagination bug * Bug fix * Bug fixes to evals v2 * Minor fix * upgrading tiktoken, adding omni to pricing (#110) Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * Update package * Dataset bugfix (#113) * Pagination bug * Bug fix * Dataset bugfix * Minor linking fix (#114) * Pagination bug * Bug fix * Dataset bugfix * Minor fix * api bug fix (#116) * fixing dropdown color (#117) * Support for Image inputs (#119) * Pagination bug * Bug fix * Support image inputs * Bug fix Spans with status_code = "ERROR" (#120) * Pagination bug * Bug fix * Support image inputs * Bug fixes * Dylan/s3en 2265 save prompt and model settings directly from playground (#99) * bug fixes, adding save prompt button in playground * adding save prompt dialog * adding create registry dialog * working registry selection * Adding new prompt to exist registry * storing * button padding fix * refactor, adding combobox * increasing dropdwon width --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * adding image display * adding property filtering (#123) * optimizing cost function * optimizing per cost per hour function * removing console statements * updating queryKey for datasets * init advanced filter dialog * adding operator combobox * updating ui * adding attribute type * bug fixes * cleanup * remove console * updating queryKeys * adding missing table checks * adding more checks * chore: update clickhouse docker for user management * style: remove blank space * updating save prompt button * Redesign of Manual Evaluations (#134) * Pagination bug * Bug fix * Schema changes for evals * Fix APIs for evals * Minor bugfixes * Bugfix * Evaluation changes * Manual eval simplification * Minor fix * Manual evaluations UX changes * Chart title fix * Disable experiments * fix * Dylan/s3en 2318 add the ability to import a conversation from the traces to (#133) * adding import conversation button * init import conversation dialog * adding trace rows * fixing scroll bug * adding import prompt to playground * fixes * Minor * Bugfixes --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * storing * remove import for images * importing attribute stuff * fixing operations display * cleaning up advanced filters * cleanup * docs: update docker instructions * removing console, fixing button * add stars history (#139) * Dylan/s3en 2401 ability to filter by user for metrics and traces (#138) * adding user id combobox, api * fixing sql query * removing user filter on deselect{ * moving user combobox into filter dialog * adding filtering by user for token charts * bug fixes, more userId route, function updates * latency filtering * updating to alphanumeric * removing console, comment * updating default filter operation to and * adding clear all button, cleanup * Dylan/s3en 2350 aggregate metrics by models v2 (#141) * adding user id combobox, api * fixing sql query * removing user filter on deselect{ * moving user combobox into filter dialog * adding filtering by user for token charts * bug fixes, more userId route, function updates * latency filtering * updating to alphanumeric * removing console, comment * updating default filter operation to and * adding clear all button, cleanup * init commit model combobox * adding api, updating function, adding to metrics page * adding model to charts, apis, service * remove consoles * Automated Evaluations - Support for Inspect AI (#142) * Pagination bug * Bug fix * Experiments view * re-order * Comparison view * Experiment report * Experiments - hook up APIs * Experiments pagination * deduplicate paginated data * Fix loading and error states - experiments * Rename experiments and evaluations * Add loading spinner * Link evaluation * Link to evals * Fixes * Minor UI fixes * Bugfixes * DSPy support (#144) * Pagination bug * Bug fix * Bugfixes * DSPy support * Update README --------- Co-authored-by: Darshit Suratwala <darshit@scale3labs.com> Co-authored-by: darshit-s3 <119623510+darshit-s3@users.noreply.github.com> Co-authored-by: dylan <dylan@scale3labs.com> Co-authored-by: dylanzuber-scale3 <116033320+dylanzuber-scale3@users.noreply.github.com> Co-authored-by: Rohit Kadhe <rohit@scale3labs.com> Co-authored-by: Rohit Kadhe <113367036+rohit-kadhe@users.noreply.github.com> Co-authored-by: MayuriS24 <159064413+MayuriS24@users.noreply.github.com> Co-authored-by: yemiscale3 <111706151+yemiadej@users.noreply.github.com>
Scale3-Labs · Jun 17, 2024 · e16d627 · e16d627
1 parent fa4ea33
commit e16d627
Show file tree

Hide file tree

Showing 8 changed files with 119 additions and 37 deletions.
diff --git a/README.md b/README.md
@@ -132,6 +132,7 @@ Langtrace automatically captures traces from the following vendors:
 | Groq | LLM | :x: | :white_check_mark: |
 | Langchain | Framework | :x: | :white_check_mark: |
 | LlamaIndex | Framework | :white_check_mark: | :white_check_mark: |
+| DSPy | Framework | :x: | :white_check_mark: |
 | Pinecone | Vector Database | :white_check_mark: | :white_check_mark: |
 | ChromaDB | Vector Database | :white_check_mark: | :white_check_mark: |
 | QDrant | Vector Database | :x: | :white_check_mark: |

diff --git a/components/project/traces/trace-row.tsx b/components/project/traces/trace-row.tsx
@@ -6,7 +6,7 @@ import {
  convertTracesToHierarchy,
  correctTimestampFormat,
 } from "@/lib/trace_utils";
-import { calculatePriceFromUsage, formatDateTime } from "@/lib/utils";
+import { calculatePriceFromUsage, cn, formatDateTime } from "@/lib/utils";
 import { ChevronDown, ChevronRight } from "lucide-react";
 import { useState } from "react";
 import { JsonView, allExpanded, defaultStyles } from "react-json-view-lite";
@@ -104,11 +104,10 @@ export const TraceRow = ({
  return (
  <div className="flex flex-col gap-3">
  <div
- className={
- importTrace
- ? "grid grid-cols-12 items-center gap-6 cursor-pointer"
- : "grid grid-cols-11 items-center gap-6 cursor-pointer"
- }
+ className={cn(
+ importTrace ? "grid-cols-12" : "grid-cols-11",
+ "grid items-center gap-6 cursor-pointer"
+ )}
  onClick={() => setCollapsed(!collapsed)}
  >
  <div className="flex flex-row items-center gap-2">
@@ -149,14 +148,22 @@ export const TraceRow = ({
  </p>
  </div>
  <p className="text-xs font-semibold">{model}</p>
- <HoverCell
- values={prompts?.length > 0 ? JSON.parse(prompts[0]) : []}
- className="flex items-center max-w-fit text-xs h-10 truncate overflow-y-scroll font-semibold col-span-2"
- />
- <HoverCell
- values={responses?.length > 0 ? JSON.parse(responses[0]) : []}
- className="flex items-center max-w-fit text-xs h-10 truncate overflow-y-scroll font-semibold col-span-2"
- />
+ {prompts?.length > 0 ? (
+ <HoverCell
+ values={JSON.parse(prompts[0])}
+ className="flex items-center max-w-fit text-xs h-10 truncate overflow-y-scroll font-semibold col-span-2"
+ />
+ ) : (
+ <p className="text-xs font-semibold col-span-2"></p>
+ )}
+ {responses?.length > 0 ? (
+ <HoverCell
+ values={JSON.parse(responses[0])}
+ className="flex items-center max-w-fit text-xs h-10 truncate overflow-y-scroll font-semibold col-span-2"
+ />
+ ) : (
+ <p className="text-xs font-semibold col-span-2"></p>
+ )}
  <p className="text-xs font-semibold">{userId}</p>
  <p className="text-xs">
  {tokenCounts?.input_tokens || tokenCounts?.prompt_tokens}
@@ -199,6 +206,25 @@ export const TraceRow = ({
  )}
  </Button>
  )}
+ <Button
+ disabled={prompts.length === 0 || responses.length === 0}
+ onClick={() => setSelectedTab("llm")}
+ variant={"ghost"}
+ className="flex flex-col justify-between pb-0"
+ >
+ <p
+ className={
+ selectedTab === "llm"
+ ? "text-xs text-primary font-medium"
+ : "text-xs text-muted-foreground font-medium"
+ }
+ >
+ LLM Requests
+ </p>
+ {selectedTab === "llm" && (
+ <Separator className="bg-primary h-[2px]" />
+ )}
+ </Button>
  {!importTrace && (
  <Button
  onClick={() => setSelectedTab("logs")}
@@ -212,7 +238,7 @@ export const TraceRow = ({
  : "text-xs text-muted-foreground font-medium"
  }
  >
- Logs
+ {`Logs (${trace?.length ?? 0} total)`}
  </p>
  {selectedTab === "logs" && (
  <Separator className="bg-primary h-[2px]" />
@@ -233,32 +259,13 @@ export const TraceRow = ({
  : "text-xs text-muted-foreground font-medium"
  }
  >
- Events
+ Events/Errors
  </p>
  {selectedTab === "events" && (
  <Separator className="bg-primary h-[2px]" />
  )}
  </Button>
  )}
- <Button
- disabled={prompts.length === 0 || responses.length === 0}
- onClick={() => setSelectedTab("llm")}
- variant={"ghost"}
- className="flex flex-col justify-between pb-0"
- >
- <p
- className={
- selectedTab === "llm"
- ? "text-xs text-primary font-medium"
- : "text-xs text-muted-foreground font-medium"
- }
- >
- LLM Requests
- </p>
- {selectedTab === "llm" && (
- <Separator className="bg-primary h-[2px]" />
- )}
- </Button>
  {langgraph && !importTrace && (
  <Button
  onClick={() => setSelectedTab("langgraph")}

diff --git a/components/shared/vendor-metadata.tsx b/components/shared/vendor-metadata.tsx
@@ -46,8 +46,16 @@ export function vendorBadgeColor(vendor: string) {
  return "bg-grey-500";
  }
 
- if (vendor.includes("qdrant")) {
- return "bg-grey-500";
+ if (vendor.includes("dspy")) {
+ return "bg-red-500";
+ }
+
+ if (vendor.includes("weaviate")) {
+ return "bg-green-500";
+ }
+
+ if (vendor.includes("pg")) {
+ return "bg-blue-500";
  }
 
  return "bg-gray-500";
@@ -98,6 +106,18 @@ export function vendorColor(vendor: string) {
  return "bg-slate-200";
  }
 
+ if (vendor.includes("dspy")) {
+ return "bg-red-200";
+ }
+
+ if (vendor.includes("weaviate")) {
+ return "bg-green-200";
+ }
+
+ if (vendor.includes("pg")) {
+ return "bg-blue-200";
+ }
+
  return "bg-gray-800";
 }
 
@@ -279,6 +299,54 @@ export function VendorLogo({
  );
  }
 
+ if (vendor.includes("dspy")) {
+ const color = vendorColor("dspy");
+ return (
+ <Image
+ alt="DSPy Logo"
+ src="/dspy.png"
+ width={50}
+ height={50}
+ className={cn(
+ `${color} p-[3px]`,
+ variant === "circular" ? "rounded-full" : "rounded-md"
+ )}
+ />
+ );
+ }
+
+ if (vendor.includes("weaviate")) {
+ const color = vendorColor("weaviate");
+ return (
+ <Image
+ alt="Weaviate Logo"
+ src="/weaviate.png"
+ width={50}
+ height={50}
+ className={cn(
+ `${color} p-[3px]`,
+ variant === "circular" ? "rounded-full" : "rounded-md"
+ )}
+ />
+ );
+ }
+
+ if (vendor.includes("pg")) {
+ const color = vendorColor("pg");
+ return (
+ <Image
+ alt="Postgres Logo"
+ src="/pg.png"
+ width={50}
+ height={50}
+ className={cn(
+ `${color} p-[3px]`,
+ variant === "circular" ? "rounded-full" : "rounded-md"
+ )}
+ />
+ );
+ }
+
  return (
  <div className="flex items-center bg-muted p-2 rounded-sm">
  <StackIcon

diff --git a/lib/constants.ts b/lib/constants.ts
@@ -84,6 +84,10 @@ export const OPENAI_PRICING: Record<string, CostTableEntry> = {
  input: 0.0005,
  output: 0.0015,
  },
+ "gpt-3.5-turbo-0613": {
+ input: 0.0005,
+ output: 0.0015,
+ },
  "gpt-3.5-turbo-instruct": {
  input: 0.0015,
  output: 0.002,

diff --git a/lib/utils.ts b/lib/utils.ts
@@ -572,6 +572,8 @@ export function getVendorFromSpan(span: Span): string {
  vendor = "weaviate";
  } else if (span.name.includes("pg") || serviceName.includes("pg")) {
  vendor = "pg";
+ } else if (span.name.includes("dspy") || serviceName.includes("dspy")) {
+ vendor = "dspy";
  }
  return vendor;
 }
diff --git a/public/dspy.png b/public/dspy.png
diff --git a/public/pg.png b/public/pg.png
diff --git a/public/weaviate.png b/public/weaviate.png