Skip to content

Commit

Permalink
Release 1.5.1 (#143)
Browse files Browse the repository at this point in the history 
* Pagination bug

* Bug fix

* chore: add docker cmd

* Compatibility fixes for SDK version 2.0.0 (#69)

* Pagination bug

* Bug fix

* Fix for schema changes

* Render tool calling

* Support for Langgraph, Qdrant & Groq  (#73)

* Pagination bug

* Bug fix

* Add langgraph support

* QDrant support

* Add Groq support

* update README

* update README

* feat: optimise docker image for self host setup

* adding api access to traces endpoint

* clean up

* refactor

* feat: add clickhouse db create on app start (#79)

* docs: add railway deploy, fix sdk badges (#81)

* untrack .env

* Revert "untrack .env"

This reverts commit 4551d7e.

* Playground and Prompt Management (#83)

* Pagination bug

* Bug fix

* Playground - basic implementation

* Playground - streaming and nonstreaming

* Move playground inside project

* API key flow

* Api key

* Playground refactor

* Add chat hookup

* anthropic streaming support

* Bug fixes to openai playground

* Anthropic bugfixes

* Anthropic bugfix

* Cohere first iteration

* Cohere role fixes

* Cohere api fix

* Parallel running

* Playground cost calculation non streaming

* playground - streaming token calculation

* latency and cost

* Support for Groq

* Add model name

* Prompt management views

* Remove current promptset flow

* Prompt management API hooks

* Prompt registry final

* Playground bugfixes

* Bug fix playground

* Rearrange project nav

* Fix playground

* Fix prompts

* Bugfixes

* Minor fix

* Prompt versioning bugfix

* Bugfix

* fix: clickhouse table find queries (#82)

* Fix to surface multiple LLM requests inside LLM View (#84)

* Pagination bug

* Bug fix

* Fix for surfacing multiple LLM requests in LLMView

* Minor bugfixes (#86)

* Pagination bug

* Bug fix

* Bugfixes

* api to fetch promptset with prompt filters

* bug fixes

* fix invalid redirect

* fix invalid status code

* Project Switcher (#90)

* Pagination bug

* Bug fix

* Project switcher

* Feat: dataset download (#60)

* API: download dataset

* API: Download dataset

* updated download-dataset api

* Updated: download_dataset api

* Updated download dataset API

* Updated Download API: changed Response to Next Response, add a condition to ensure max page size is 500

* updated the download-dataset API: fixed the format and removed redundant lines of code

* Updated download_daatset API: file name and removed 'id' param

* Added the Download dataset button.

* Merged developemnt into my branch

* Updated button size

* Fixes

---------

Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com>

* Update prompt registry with instructions to fetch prompts (#91)

* Pagination bug

* Bug fix

* Update prompt registry

* Minor bugfix (#94)

* Pagination bug

* Bug fix

* Minor bugfix

* chore: update github repo badges

* optimizing token count function

* Add GPT4-O Pricing and Playground (#98)

* Pagination bug

* Bug fix

* Add GPT4-O support

* Add GPT4-O support

* Update cost

* Dylan/s3en 2234 add perplexity support to playground (#89)

* adding perplexity to playground types

* adding ui stuff:'

* adding perplexity chat api

* fixing perplexity model dropdown

---------

Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com>

* api changes

* add api access to get api and fix all bugs

* bug fix

* bug fix

* updating descriptions to optional

* prio python

* cleanup and fixes

* more bug fixes

* more fixes

* remove console log

* updating trace_service functions

* add migration

* add format function, updating from day to hour

* adding dropwdown menu

* updating query key

* updating query keys v2

* clean up

* fix bug

* Minor bugfix (#102)

* Pagination bug

* Bug fix

* Minor bugfix

* fixes

* Bug fixes to Evaluations  (#111)

* Pagination bug

* Bug fix

* Bug fixes to evals v2

* Minor fix

* upgrading tiktoken, adding omni to pricing (#110)

Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com>

* Update package

* Dataset bugfix (#113)

* Pagination bug

* Bug fix

* Dataset bugfix

* Minor linking fix (#114)

* Pagination bug

* Bug fix

* Dataset bugfix

* Minor fix

* api bug fix (#116)

* fixing dropdown color (#117)

* Support for Image inputs (#119)

* Pagination bug

* Bug fix

* Support image inputs

* Bug fix Spans with status_code = "ERROR" (#120)

* Pagination bug

* Bug fix

* Support image inputs

* Bug fixes

* Dylan/s3en 2265 save prompt and model settings directly from playground (#99)

* bug fixes, adding save prompt button in playground

* adding save prompt dialog

* adding create registry dialog

* working registry selection

* Adding new prompt to exist registry

* storing

* button padding fix

* refactor, adding combobox

* increasing dropdwon width

---------

Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com>

* adding image display

* adding property filtering (#123)

* optimizing cost function

* optimizing per cost per hour function

* removing console statements

* updating queryKey for datasets

* init advanced filter dialog

* adding operator combobox

* updating ui

* adding attribute type

* bug fixes

* cleanup

* remove console

* updating queryKeys

* adding missing table checks

* adding more checks

* chore: update clickhouse docker for user management

* style: remove blank space

* updating save prompt button

* Redesign of Manual Evaluations (#134)

* Pagination bug

* Bug fix

* Schema changes for evals

* Fix APIs for evals

* Minor bugfixes

* Bugfix

* Evaluation changes

* Manual eval simplification

* Minor fix

* Manual evaluations UX changes

* Chart title fix

* Disable experiments

* fix

* Dylan/s3en 2318 add the ability to import a conversation from the traces to (#133)

* adding import conversation button

* init import conversation dialog

* adding trace rows

* fixing scroll bug

* adding import prompt to playground

* fixes

* Minor

* Bugfixes

---------

Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com>

* storing

* remove import for images

* importing attribute stuff

* fixing operations display

* cleaning up advanced filters

* cleanup

* docs: update docker instructions

* removing console, fixing button

* add stars history (#139)

* Dylan/s3en 2401 ability to filter by user for metrics and traces (#138)

* adding user id combobox, api

* fixing sql query

* removing user filter on deselect{

* moving user combobox into filter dialog

* adding filtering by user for token charts

* bug fixes, more userId route, function updates

* latency filtering

* updating to alphanumeric

* removing console, comment

* updating default filter operation to and

* adding clear all button, cleanup

* Dylan/s3en 2350 aggregate metrics by models v2 (#141)

* adding user id combobox, api

* fixing sql query

* removing user filter on deselect{

* moving user combobox into filter dialog

* adding filtering by user for token charts

* bug fixes, more userId route, function updates

* latency filtering

* updating to alphanumeric

* removing console, comment

* updating default filter operation to and

* adding clear all button, cleanup

* init commit model combobox

* adding api, updating function, adding to metrics page

* adding model to charts, apis, service

* remove consoles

* Automated Evaluations - Support for Inspect AI (#142)

* Pagination bug

* Bug fix

* Experiments view

* re-order

* Comparison view

* Experiment report

* Experiments - hook up APIs

* Experiments pagination

* deduplicate paginated data

* Fix loading and error states - experiments

* Rename experiments and evaluations

* Add loading spinner

* Link evaluation

* Link to evals

* Fixes

* Minor UI fixes

---------

Co-authored-by: Darshit Suratwala <darshit@scale3labs.com>
Co-authored-by: darshit-s3 <119623510+darshit-s3@users.noreply.github.com>
Co-authored-by: dylan <dylan@scale3labs.com>
Co-authored-by: dylanzuber-scale3 <116033320+dylanzuber-scale3@users.noreply.github.com>
Co-authored-by: Rohit Kadhe <rohit@scale3labs.com>
Co-authored-by: Rohit Kadhe <113367036+rohit-kadhe@users.noreply.github.com>
Co-authored-by: MayuriS24 <159064413+MayuriS24@users.noreply.github.com>
Co-authored-by: yemiscale3 <111706151+yemiadej@users.noreply.github.com>
  • Loading branch information
@karthikscale3 @darshit-s3
@dylanzuber-scale3 @rohit-kadhe @MayuriS24 @yemiadej
9 people authored Jun 13, 2024
1 parent e39e3c6 commit fa4ea33
Show file tree
Hide file tree
Showing 56 changed files with 2,750 additions and 346 deletions.
129 changes: 20 additions & 109 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ The traces generated by Langtrace adhere to [Open Telemetry Standards(OTEL)](htt

---

## SDK Repositories

- [Langtrace Typescript SDK](https://github.com/Scale3-Labs/langtrace-typescript-sdk)
- [Langtrace Python SDK](https://github.com/Scale3-Labs/langtrace-python-sdk)
- [Semantic Span Attributes](https://github.com/Scale3-Labs/langtrace-trace-attributes)

---

## Getting Started

### Langtrace Cloud ☁️
Expand All @@ -35,7 +43,7 @@ To use the managed SaaS version of Langtrace, follow the steps below:
4. In your application, install the Langtrace SDK and initialize it with the API key you generated in the step 3.
5. The code for installing and setting up the SDK is shown below:

### If your application is built using **typescript/javascript**
### If your application is built using typescript/javascript

```typescript
npm i @langtrase/typescript-sdk
Expand All @@ -53,7 +61,7 @@ import * as Langtrace from "@langtrase/typescript-sdk"; // Must precede any llm
LangTrace.init(); // LANGTRACE_API_KEY as an ENVIRONMENT variable
```

### If your application is built using **python**
### If your application is built using python

```python
pip install langtrace-python-sdk
Expand All @@ -80,7 +88,7 @@ To run the Langtrace locally, you have to run three services:
- Clickhouse database

> [!IMPORTANT]
> Checkout [documentation](https://docs.langtrace.ai/hosting/overview) for various deployment options and configurations.
> Checkout our [documentation](https://docs.langtrace.ai/hosting/overview) for various deployment options and configurations.
Requirements:

Expand All @@ -99,16 +107,6 @@ docker compose up

The application will be available at `http://localhost:3000`.

> [!NOTE]
> if you wish to build the docker image locally and use it, run the docker compose up command with the `--build` flag.
> [!TIP]
> to manually pull the docker image from [docker hub](https://hub.docker.com/r/scale3labs/langtrace-client/tags), run the following command:
>
> ```bash
> docker pull scale3labs/langtrace-client:latest
> ```
#### Take down the setup

To delete containers and volumes
Expand All @@ -119,100 +117,6 @@ docker compose down -v

`-v` flag is used to delete volumes

#### Useful Ops Commands

The following are some commands that may come handy during setup and debugging.

<details>
<summary>Connecting to postgres db</summary>

```bash
docker exec -it langtrace-postgres psql --dbname=langtrace --username=ltuser --password
```

</details>

<details>
<summary>Connecting to clickhouse server</summary>

```bash
docker exec -it langtrace-clickhouse clickhouse-client
```

</details>

<details>
<summary>Checking langtrace client app logs</summary>

```bash
docker logs langtrace
```

If you want to follow the logs

```bash
docker logs -f langtrace
```

</details>

<details>
<summary>Running prisma schema apply command</summary>

```bash
docker exec -it langtrace npm run create-tables
```

</details>

#### Common issues for local setup

<details>
<summary>Table not found error OR Column not found error</summary>
Its likely that schema is not applied to the database or the schema is not in sync with the database. To fix this, run the following command:

```bash
docker exec -it langtrace npm run create-tables
```

</details>

<details>
<summary>Prisma schema not in sync with database</summary>
If you have made changes to the prisma schema and want to apply the changes to the database, run the following command:

```bash
docker exec -it langtrace npm run create-tables
```

</details>

<details>
<summary>Docker compose failing to setup with `Additional property required is not allowed` errors</summary>
Its likely that you are using an older version of docker-compose. Update docker-compose to the latest version.

Certain docker compose schema used in this project are only supported in newer versions of docker-compose.

Either you **update the docker compose version** OR **remove the depends_on property** that is causing the error.

</details>

<details>
<summary>Clickhouse server not starting</summary>
If clickhouse server is not starting, it is likely that the port 8123 is already in use. You can change the port in the docker-compose file.
</details>

<br/>
Install the langtrace SDK in your application by following the same instructions under the Langtrace Cloud section above for sending traces to your self hosted setup.

---

## SDK Repositories

- [Langtrace Typescript SDK](https://github.com/Scale3-Labs/langtrace-typescript-sdk)
- [Langtrace Python SDK](https://github.com/Scale3-Labs/langtrace-python-sdk)
- [Semantic Span Attributes](https://github.com/Scale3-Labs/langtrace-trace-attributes)

---

## Supported integrations
Expand All @@ -231,6 +135,7 @@ Langtrace automatically captures traces from the following vendors:
| Pinecone | Vector Database | :white_check_mark: | :white_check_mark: |
| ChromaDB | Vector Database | :white_check_mark: | :white_check_mark: |
| QDrant | Vector Database | :x: | :white_check_mark: |
| Weaviate | Vector Database | :white_check_mark: | :white_check_mark: |

---

Expand All @@ -253,6 +158,12 @@ We welcome contributions to this project. To get started, fork this repository a

---

## Langtrace Star History

## [![Langtrace Star History Chart](https://api.star-history.com/svg?repos=Scale3-Labs/langtrace&type=Timeline)](https://star-history.com/#Scale3-Labs/langtrace&Timeline)

---

## Security

To report security vulnerabilites, email us at <security@scale3labs.com>. You can read more on security [here](https://github.com/Scale3-Labs/langtrace/blob/development/SECURITY.md).
Expand All @@ -269,7 +180,7 @@ To report security vulnerabilites, email us at <security@scale3labs.com>. You ca
## Frequently Asked Questions

**1. Can I self host and run Langtrace in my own cloud?**
Yes, you can absolutely do that. Follow the self hosting setup instructions laid out above.
Yes, you can absolutely do that. Follow the self hosting setup instructions in our [documentation](https://docs.langtrace.ai/hosting/overview).

**2. What is the pricing for Langtrace cloud?**
Currently, we are not charging anything for Langtrace cloud and we are primarily looking for feedback so we can continue to improve the project. We will inform our users when we decide to monetize it.
Expand All @@ -278,7 +189,7 @@ Currently, we are not charging anything for Langtrace cloud and we are primarily
Langtrace uses NextJS for the frontend and APIs. It uses PostgresDB as a metadata store and Clickhouse DB for storing spans, metrics, logs and traces.

**4. Can I contribute to this project?**
Absolutely! We love developers and welcome contributions. Get involved early by joining our slack workspace.
Absolutely! We love developers and welcome contributions. Get involved early by joining our [Discord Community](https://discord.langtrace.ai/).

**5. What skillset is required to contribute to this project?**
Programming Languages: Typescript and Python.
Expand Down
71 changes: 12 additions & 59 deletions ...ect/[project_id]/evaluate/page-client.tsx → .../[project_id]/annotations/page-client.tsx
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
"use client";

import { EvalChart } from "@/components/charts/eval-chart";
import LargeChartSkeleton from "@/components/charts/large-chart-skeleton";
import { CreateTest } from "@/components/evaluate/create-test";
import { EditTest } from "@/components/evaluate/edit-test";
import { CreateTest } from "@/components/annotations/create-test";
import { EditTest } from "@/components/annotations/edit-test";
import EvaluationTable, {
EvaluationTableSkeleton,
} from "@/components/evaluate/evaluation-table";
} from "@/components/annotations/evaluation-table";
import { EvalChart } from "@/components/charts/eval-chart";
import LargeChartSkeleton from "@/components/charts/large-chart-skeleton";
import { AddtoDataset } from "@/components/shared/add-to-dataset";
import { Separator } from "@/components/ui/separator";
import { Skeleton } from "@/components/ui/skeleton";
import { cn } from "@/lib/utils";
import { Button } from "@/components/ui/button";
import { Test } from "@prisma/client";
import { RabbitIcon } from "lucide-react";
import { useParams } from "next/navigation";
Expand Down Expand Up @@ -75,7 +73,7 @@ export default function PageClient({ email }: { email: string }) {
return (
<div className="w-full flex flex-col gap-4">
<div className="md:px-24 px-12 py-12 flex justify-between bg-muted">
<h1 className="text-3xl font-semibold">Evaluations</h1>
<h1 className="text-3xl font-semibold">Annotate & Measure</h1>
<div className="flex gap-2">
<CreateTest projectId={projectId} variant={"outline"} />
{tests?.tests?.length > 0 && (
Expand Down Expand Up @@ -123,57 +121,12 @@ export default function PageClient({ email }: { email: string }) {
function PageSkeleton() {
return (
<div className="w-full flex flex-col">
<div className="flex flex-row gap-4 absolute top-[14rem] w-full md:px-24 px-12">
<div className="bg-primary-foreground flex flex-col gap-0 border rounded-md w-[12rem] h-fit">
{Array.from({ length: 5 }).map((_, i) => {
return (
<div className="flex flex-col" key={i}>
<div
className={cn(
"flex flex-col gap-4 p-4 items-start cursor-pointer",
i === 0 ? "rounded-t-md" : "",
i === 4 ? "rounded-b-md" : ""
)}
>
<div
className={cn(
"text-sm text-muted-foreground font-semibold capitalize",
i === 0 ? "text-primary" : ""
)}
>
<Skeleton className="w-20 h-6" />
</div>
<span className="text-[0.6rem] text-primary font-bold">
<Skeleton className="w-20 h-6" />
</span>
</div>
<Separator />
</div>
);
})}
</div>
<div className="bg-primary-foreground flex flex-col gap-12 border rounded-md w-full p-4 mb-24">
<div className="flex flex-row">
<div className="flex flex-col gap-3 items-start w-[25rem]">
<div className="flex flex-col gap-1">
<div className="text-xl font-semibold capitalize">
<Skeleton className="w-20 h-6" />
</div>
<span className="text-xs font-semibold text-muted-foreground">
<Skeleton className="w-20 h-6" />
</span>
</div>
<span className="text-sm text-primary font-bold">
<Skeleton className="w-20 h-6" />
</span>
<div className="text-sm text-muted-foreground">
<Skeleton className="w-20 h-6" />
</div>
</div>
<LargeChartSkeleton />
</div>
<EvaluationTableSkeleton />
<div className="flex flex-col gap-12 top-[16rem] w-full md:px-24 px-12 mb-24">
<LargeChartSkeleton />
<div className="flex flex-col gap-2">
<Button variant="outline" disabled={true} />
</div>
<EvaluationTableSkeleton />
</div>
</div>
);
Expand Down
0 ...d)/project/[project_id]/evaluate/page.tsx → ...project/[project_id]/annotations/page.tsx
View file
File renamed without changes.
40 changes: 31 additions & 9 deletions app/(protected)/project/[project_id]/datasets/dataset/[dataset_id]/page.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,22 @@ import DatasetRowSkeleton from "@/components/project/dataset/dataset-row-skeleto
import { EditData } from "@/components/project/dataset/edit-data";
import { DownloadDataset } from "@/components/shared/download-dataset";
import { Spinner } from "@/components/shared/spinner";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { Separator } from "@/components/ui/separator";
import { PAGE_SIZE } from "@/lib/constants";
import { EVALUATIONS_DOCS_URL, PAGE_SIZE } from "@/lib/constants";
import { Data } from "@prisma/client";
import { ChevronLeft } from "lucide-react";
import { ArrowTopRightIcon } from "@radix-ui/react-icons";
import { ChevronLeft, FlaskConical } from "lucide-react";
import Link from "next/link";
import { useParams } from "next/navigation";
import { useState } from "react";
import { useBottomScrollListener } from "react-bottom-scroll-listener";
import { useQuery } from "react-query";
import { toast } from "sonner";

export default function Dataset() {
const projectId = useParams()?.project_id as string;
const dataset_id = useParams()?.dataset_id as string;
const [page, setPage] = useState<number>(1);
const [totalPages, setTotalPages] = useState<number>(1);
Expand Down Expand Up @@ -86,13 +90,31 @@ export default function Dataset() {
} else {
return (
<div className="w-full py-6 px-6 flex flex-col gap-4">
<div className="flex gap-4 items-center w-fit">
<Button variant="secondary" onClick={() => window.history.back()}>
<ChevronLeft className="mr-1" />
Back
</Button>
<CreateData datasetId={dataset_id} />
<DownloadDataset datasetId={dataset_id} disabled={fetchDataset.isLoading || currentData?.length === 0} />
<div className="flex justify-between items-center">
<div className="flex gap-4 items-center w-fit">
<Button variant="secondary" onClick={() => window.history.back()}>
<ChevronLeft className="mr-1" />
Back
</Button>
<CreateData datasetId={dataset_id} />
<DownloadDataset
projectId={projectId}
datasetId={dataset_id}
disabled={fetchDataset.isLoading || currentData?.length === 0}
/>
</div>
<div className="flex gap-4 items-center w-fit">
<Badge variant={"outline"} className="text-sm">
Dataset ID: {dataset_id}
</Badge>
<Link href={EVALUATIONS_DOCS_URL} target="_blank">
<Button variant="outline">
Run Evaluation
<FlaskConical className="ml-1 h-4 w-4" />
<ArrowTopRightIcon className="ml-1 h-4 w-4" />
</Button>
</Link>
</div>
</div>
<div className="flex flex-col gap-3 rounded-md border border-muted max-h-screen overflow-y-scroll">
<div className="grid grid-cols-5 items-center justify-stretch gap-3 py-3 px-4 bg-muted">
Expand Down
Loading

0 comments on commit fa4ea33

Please sign in to comment.