Skip to content

Commit

Permalink
Opts config (#2)
Browse files Browse the repository at this point in the history 
* serving init opts

* serving run pass

* serving tests passing

* handle extra args

* move message send logic

* message send

* organize

* format

* main typespecs

* other typespecs

* message tests

* unreachable code

* remove serving already started case

* finish typespecs

* name to pid

* update docs

* docs final

* changelog
  • Loading branch information
@jessedrelick
jessedrelick authored Aug 30, 2024
1 parent cb19734 commit 542cc4f
Show file tree
Hide file tree
Showing 14 changed files with 668 additions and 381 deletions.
18 changes: 18 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Changelog

## 0.1.2
This release removes [application environment configuration](https://hexdocs.pm/elixir/1.17.2/design-anti-patterns.html#using-application-configuration-for-libraries) and moves to an opts-based configuration. See [README.md](README.md#configuration) for more info.

### Features
- Configure `Agens` via `Supervisor` opts instead of `Application` environment
- Add `Agens.Agent.get_config/1`
- Add `Agens.Serving.get_config/1`
- Support sending `Agens.Message` without `Agens.Agent`
- Override default prompt prefixes with `Agens.Serving`

### Fixes
- `Agens.Job.get_config/1` now wraps return value with `:ok` tuple: `{:ok, Agens.Job.Config.t()}`
- Replaced `module() | Nx.Serving.t()` with `atom()` in `Agens.Agent.Config.t()`

## 0.1.1
Initial release
118 changes: 63 additions & 55 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,46 +28,6 @@ def deps do
end
```

## Configuration
Future versions of Agens will be configurable by providing options to `Agens.Supervisor` in order to [avoid using application configuration](https://hexdocs.pm/elixir/1.17.2/design-anti-patterns.html#using-application-configuration-for-libraries). For now, however, you can change the Agens `Registry` if needed via config:

```elixir
config :agens, registry: Agens.Registry
```

In addition, you can also override the prompt prefixes:

```elixir
config :agens, prompts: %{
prompt:
{"Agent",
"You are a specialized agent with the following capabilities and expertise"},
identity:
{"Identity",
"You are a specialized agent with the following capabilities and expertise"},
context: {"Context", "The purpose or goal behind your tasks are to"},
constraints:
{"Constraints", "You must operate with the following constraints or limitations"},
examples:
{"Examples",
"You should consider the following examples before returning results"},
reflection:
{"Reflection",
"You should reflect on the following factors before returning results"},
instructions:
{"Tool Instructions",
"You should provide structured output for function calling based on the following instructions"},
objective: {"Step Objective", "The objective of this step is to"},
description:
{"Job Description", "This is part of multi-step job to achieve the following"},
input:
{"Input",
"The following is the actual input from the user, system or another agent"}
}
```

See the [Prompting](#prompting) section below or `Agens.Message` for more information on prompt prefixes.

## Usage
Building a multi-agent workflow with Agens involves a few different steps and core entities:

Expand All @@ -77,12 +37,11 @@ Building a multi-agent workflow with Agens involves a few different steps and co
This will start Agens as a supervised process inside your application:

```elixir
Supervisor.start_link(
[
{Agens.Supervisor, name: Agens.Supervisor}
],
strategy: :one_for_one
)
children = [
{Agens.Supervisor, name: Agens.Supervisor}
]

Supervisor.start_link(children, strategy: :one_for_one)
```

See `Agens.Supervisor` for more information
Expand All @@ -96,19 +55,17 @@ A **Serving** is essentially a wrapper for language model inference. It can be a
Application.put_env(:nx, :default_backend, EXLA.Backend)
auth_token = System.get_env("HF_AUTH_TOKEN")

my_serving = fn ->
repo = {:hf, "mistralai/Mistral-7B-Instruct-v0.2", auth_token: auth_token}
repo = {:hf, "mistralai/Mistral-7B-Instruct-v0.2", auth_token: auth_token}

{:ok, model} = Bumblebee.load_model(repo, type: :bf16)
{:ok, tokenizer} = Bumblebee.load_tokenizer(repo)
{:ok, generation_config} = Bumblebee.load_generation_config(repo)
{:ok, model} = Bumblebee.load_model(repo, type: :bf16)
{:ok, tokenizer} = Bumblebee.load_tokenizer(repo)
{:ok, generation_config} = Bumblebee.load_generation_config(repo)

Bumblebee.Text.generation(model, tokenizer, generation_config)
end
serving = Bumblebee.Text.generation(model, tokenizer, generation_config)

serving_config = %Agens.Serving.Config{
name: :my_serving,
serving: my_serving()
serving: serving
}

{:ok, pid} = Agens.Serving.start(serving_config)
Expand Down Expand Up @@ -165,10 +122,61 @@ See `Agens.Job` for more information

---

## Configuration
Additional options can be passed to `Agens.Supervisor` in order to override the default values:

```elixir
opts = [
registry: Agens.MyCustomRegistry,
prompts: custom_prompt_prefixes
]

children = [
{Agens.Supervisor, name: Agens.Supervisor, opts: opts}
]

Supervisor.start_link(children, strategy: :one_for_one)
```

The following default prompt prefixes can be copied, customized and used for the `prompts` option above:

```elixir
%{
prompt:
{"Agent",
"You are a specialized agent with the following capabilities and expertise"},
identity:
{"Identity",
"You are a specialized agent with the following capabilities and expertise"},
context: {"Context", "The purpose or goal behind your tasks are to"},
constraints:
{"Constraints", "You must operate with the following constraints or limitations"},
examples:
{"Examples",
"You should consider the following examples before returning results"},
reflection:
{"Reflection",
"You should reflect on the following factors before returning results"},
instructions:
{"Tool Instructions",
"You should provide structured output for function calling based on the following instructions"},
objective: {"Step Objective", "The objective of this step is to"},
description:
{"Job Description", "This is part of multi-step job to achieve the following"},
input:
{"Input",
"The following is the actual input from the user, system or another agent"}
}
```

See the [Prompting](#prompting) section below or `Agens.Message` for more information on prompt prefixes.

You can also see `Agens.Supervisor` for more information on configuration options.

## Prompting
Agens provides a variety of different ways to customize the final prompt sent to the language model (LM) or Serving. A natural language string can be assigned to the entity's specialized field (see below), while `nil` values will omit that field from the final prompt. This approach allows for precise control over the prompt’s content.

All fields with values, in addition to user input, will be included in the final prompt !!!!using the [in-context learning]() method!!!!. The goal should be to balance detailed prompts with efficient token usage by focusing on relevant fields and using concise language. This approach will yield the best results with minimal token usage, keeping costs low and performance high.
All fields with values, in addition to user input, will be included in the final prompt. The goal should be to balance detailed prompts with efficient token usage by focusing on relevant fields and using concise language. This approach will yield the best results with minimal token usage, keeping costs low and performance high.

### User/Agent
The `input` value is the only required field for building prompts. This value can be the initial value provided to `Agens.Job.run/2`, or the final result of a previous step (`Agens.Job.Step`). Both the `input` and `result` are stored in `Agens.Message`, which can also be used to send messages directly to `Agens.Agent` or `Agens.Serving` without being part of an `Agens.Job`.
Expand Down
22 changes: 0 additions & 22 deletions config/config.exs
View file

This file was deleted.

160 changes: 17 additions & 143 deletions lib/agens.ex
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,157 +21,31 @@ defmodule Agens do
- `Agens.Serving` - used to interact with language models
- `Agens.Agent` - used to interact with servings in a specialized manner
- `Agens.Job` - used to define multi-agent workflows
- `Agens.Message` - used to facilitate communication between agents, jobs, and servings
"""

defmodule Message do
@moduledoc """
The Message struct defines the details of a message passed between Agents, Jobs and Servings.
## Fields
* `:parent_pid` - The process identifier of the parent/caller process.
* `:input` - The input string for the message.
* `:prompt` - The prompt string or `Agens.Agent.Prompt` struct for the message.
* `:result` - The result string for the message.
* `:agent_name` - The name of the `Agens.Agent`.
* `:serving_name` - The name of the `Agens.Serving`.
* `:job_name` - The name of the `Agens.Job`.
* `:job_description` - The description of the `Agens.Job` to be added to the LM prompt.
* `:step_index` - The index of the `Agens.Job.Step`.
* `:step_objective` - The objective of the `Agens.Job.Step` to be added to the LM prompt.
"""

@type t :: %__MODULE__{
parent_pid: pid() | nil,
input: String.t() | nil,
prompt: String.t() | Agens.Agent.Prompt.t() | nil,
result: String.t() | nil,
agent_name: atom() | nil,
serving_name: atom() | nil,
job_name: atom() | nil,
job_description: String.t() | nil,
step_index: non_neg_integer() | nil,
step_objective: String.t() | nil
}

@enforce_keys []
defstruct [
:parent_pid,
:input,
:prompt,
:result,
:agent_name,
:serving_name,
:job_name,
:job_description,
:step_index,
:step_objective
]

alias Agens.{Agent, Serving}

@registry Application.compile_env(:agens, :registry)
@fields Application.compile_env(:agens, :prompts)

@doc """
Sends an `Agens.Message` to an `Agens.Agent`
"""
@spec send(__MODULE__.t()) :: __MODULE__.t() | {:error, :agent_not_running}
def send(%__MODULE__{} = message) do
case Registry.lookup(@registry, message.agent_name) do
[{_, {agent_pid, config}}] when is_pid(agent_pid) ->
base = build_prompt(config, message)
prompt = "<s>[INST]#{base}[/INST]"

result =
message
|> Map.put(:serving_name, config.serving)
|> Map.put(:prompt, prompt)
|> Serving.run()

message = Map.put(message, :result, result)
maybe_use_tool(message, config.tool)

[] ->
{:error, :agent_not_running}
end
end

@spec build_prompt(Agent.Config.t(), t()) :: String.t()
defp build_prompt(%Agent.Config{prompt: prompt, tool: tool}, %__MODULE__{} = message) do
%{
objective: message.step_objective,
description: message.job_description
}
|> maybe_add_prompt(prompt)
|> maybe_add_tool(tool)
|> maybe_prep_input(message.input, tool)
|> Enum.reject(&filter_empty/1)
|> Enum.map(&field/1)
|> Enum.map(&to_prompt/1)
|> Enum.join("\n\n")
end

defp filter_empty({_, value}), do: value == "" or is_nil(value)

defp field({key, value}) do
{Map.get(@fields, key), value}
end

defp to_prompt({{heading, detail}, value}) do
"""
## #{heading}
#{detail}: #{value}
"""
end

defp maybe_add_prompt(map, %Agent.Prompt{} = prompt),
do: prompt |> Map.from_struct() |> Map.merge(map)

defp maybe_add_prompt(map, prompt) when is_binary(prompt), do: Map.put(map, :prompt, prompt)
defp maybe_add_prompt(map, _prompt), do: map

defp maybe_add_tool(map, nil), do: map
defp maybe_add_tool(map, tool), do: Map.put(map, :instructions, tool.instructions())

defp maybe_prep_input(map, input, nil), do: Map.put(map, :input, input)
defp maybe_prep_input(map, input, tool), do: Map.put(map, :input, tool.pre(input))

@spec maybe_use_tool(__MODULE__.t(), module() | nil) :: __MODULE__.t()
defp maybe_use_tool(message, nil), do: message

defp maybe_use_tool(%__MODULE__{} = message, tool) do
send(
message.parent_pid,
{:tool_started, {message.job_name, message.step_index}, message.result}
)

raw =
message.result
|> tool.to_args()
|> tool.execute()

send(message.parent_pid, {:tool_raw, {message.job_name, message.step_index}, raw})

result = tool.post(raw)

send(message.parent_pid, {:tool_result, {message.job_name, message.step_index}, result})
use DynamicSupervisor

Map.put(message, :result, result)
end
@doc false
@spec start_link(keyword()) :: Supervisor.on_start()
def start_link(args) do
opts = Keyword.fetch!(args, :opts)
DynamicSupervisor.start_link(__MODULE__, opts, name: __MODULE__)
end

use DynamicSupervisor

@doc false
@spec start_link(any()) :: :ignore | {:error, any()} | {:ok, pid()}
def start_link(_) do
DynamicSupervisor.start_link(__MODULE__, :ok, name: __MODULE__)
@impl true
@spec init(keyword()) :: {:ok, DynamicSupervisor.sup_flags()}
def init(opts) do
DynamicSupervisor.init(strategy: :one_for_one, extra_arguments: [opts])
end

@doc false
@spec init(any()) :: {:ok, any()}
def init(:ok) do
DynamicSupervisor.init(strategy: :one_for_one)
@spec name_to_pid(atom(), {:error, term()}, (pid() -> any())) :: any()
def name_to_pid(name, err, cb) do
case Process.whereis(name) do
nil -> err
pid when is_pid(pid) -> cb.(pid)
end
end
end
Loading

0 comments on commit 542cc4f

Please sign in to comment.