diff --git a/LICENSE.md b/LICENSE.md index 388bda13d5..405cc0eb38 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -108,5 +108,12 @@ NOTE: This license applies to all parts of this repository except for the datase - **License**: Creative Commons Attribution 4.0 International: https://creativecommons.org/licenses/by/4.0/ - **Source**: https://allenai.org/data/socialiqa +#### Already Said That -Please note: While efforts have been made to accurately represent the licenses associated with each dataset, users should consult the original source of the dataset to ensure compliance with any licensing terms and conditions. +- **Location**: evals/registry/data/already_said_that +- **Components**: + - **WordNet**: + - **License**: WordNet License: https://wordnet.princeton.edu/license-and-commercial-use + - **Source**: https://wordnet.princeton.edu/ + +Please note: While efforts have been made to accurately represent the licenses associated with each dataset, users should consult the original source of the dataset to ensure compliance with any licensing terms and conditions. \ No newline at end of file diff --git a/evals/elsuite/already_said_that/README.md b/evals/elsuite/already_said_that/README.md new file mode 100644 index 0000000000..bdb5274b1e --- /dev/null +++ b/evals/elsuite/already_said_that/README.md @@ -0,0 +1,185 @@ +# Already Said That + +This eval measures how robust models are to distractors when performing +sequential tasks. We construct a toy task where the model needs to determine +whether it has already seen a given word, and inject distractor questions into +the interaction, keeping track of model performance throughout. + +## Usage + +Run with: + +```bash +oaieval already_said_that +``` + +We have found that `generation/direct/gpt-4-0125-preview` works well on this +eval. For more examples of tested solvers, see +[`./scripts/run_experiments.sh`](./scripts/run_experiments.sh). + +## Dataset + +The dataset consists of 500 samples, where each sample contains 100 unique words +randomly sampled from the [WordNet corpus](https://wordnet.princeton.edu/) via +the `nltk` library. + +We also rely on four sets of distractor questions, sourced directly from the +datasets of pre-existing evals. Specifically we make use of the datasets of the +following evals from our evals registry: + +- [`which-is-heavier`](../../registry/evals/which-is-heavier.yaml) +- [`first-letters`](../../registry/evals/first-letters.yaml) +- [`ambigous-sentences`](../../registry/evals/ambiguous-sentences.yaml) +- [`reverse-sort-words-eng`](../../registry/evals/reverse-sort-words-eng.yaml) + +## Evaluation Process + +The evaluation process is as follows for a given sample from our dataset: + +1. The `TASK_DESCRIPTION` prompt is shown to the solver. +2. For 100 turns, we either show a word to the solver or a distractor question, + with probability 2/3 and 1/3 respectively. +3. If a word is shown, we prefix it with `MAIN TASK -`, to indicate that we are + asking the solver to perform the main task of determining whether it has seen + the word before. +4. When showing a word, we randomly show previously seen words with a + probability of 1/2 and new words with a probability of 1/2. +5. If we show a distractor question, we directly show the question to the + solver. +6. The solver should respond with its answer wrapped in the format + `[answer: ]`. +7. The solver's response is parsed and compared to the correct answer. +8. If the solver's response is incorrect or a violation is raised (answered in + the incorrect format), in the case of the main task we stop the interaction + and record the number of turns the solver lasted for. Otherwise we continue + to the next turn. + +## Prompts + +We refer readers to [`./prompts.py`](./prompts.py) for the `TASK_DESCRIPTION` +used in the eval. + +We refer readers to [`./distractors.py`](./distractors.py) for any cosmetic +changes we make to the distractor questions. + +## Metrics + +Below are the metrics returned by the eval: + + +| **Metric** | **Notes** | +|------------------------- |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `avg_num_turns` | The average number of turns shown before the model fails across the samples. Higher is better. Best possible is 100. | +| `stddev_num_turns` | The standard deviation on the above. | +| `median_num_turns` | The median number of turns shown before the model fails across the samples. Higher is better. Best possible is 100. | +| `max_num_turns` | The maximum number of turns shown before the model fails across the samples. | +| `min_num_turns` | The minimum number of turns shown before the model fails across the samples. | +| `false_positive_rate` | How often the model answers “yes” when it should have answered “no” (i.e. a new word is shown, and the model claims to have seen it already). | +| `false_negative_rate` | How often the model answers “no” when it should have answered “yes” (i.e. a word is shown again, and the model claims to not have seen it). | +| `avg_distractor_accuracy` | For a given sample interaction, we measure whether each model response to a given distractor question is accurate. We then compute the accuracy on the distractor questions shown over the interaction. We then average this accuracy across all samples. | +| `violation_rate` | how often the model responds in an invalid format, i.e. not using the `[answer: ]` format. | +| `avg_num_distractors` | The average number of distractors shown before the model fails across the samples. Higher is better. Best possible is around 33. | +| `stddev_num_distractors` | The standard deviation on the above. | +| `median_num_distractors` | The median number of distractors shown before the model fails across the samples. Higher is better. Best possible is around 33. | +| `max_num_distractors` | The maximum number of distractors shown before the model fails across the samples. | +| `min_num_distractors` | The minimum number of distractors shown before the model fails across the samples. | + + +## Variants + +We consider each of the four distractor datasets mentioned in +[Dataset](#dataset) as a variant of the eval. + +```bash +oaieval already_said_that. +``` + +We also have a `distractorless` variant where we only show words to the solver. +We use this as a baseline to determine how robust the solver is to distractors. + +```bash +oaieval already_said_that.distractorless +``` + +## Custom Solvers + +We implement 2 custom solvers for this eval in [./solvers.py](./solvers.py): + +1. `RandomBaselineSolver`: A solver that randomly answers `yes` or `no` for any + input. We view this baseline as equivalent to randomly guessing. +2. `AlreadySaidThatHuman`: A helper solver class that wraps the `HumanCliSolver` + class such that users do not have to wrap their answer in the + `[answer: ]` format and can instead just directly type the answer. + +## Token Usage Estimates + +Below are approximate token usage estimates for a given run (one run = all +samples) of the eval, for each of the distractor variants. + +For Direct gpt-4-0125-preview: + +| Distractor variant | Input | Output | Total | +| --------------------- | ---------- | ------- | ---------- | +| which-is-heavier | 17,960,000 | 80,000 | 18,040,000 | +| ambiguous-sentences | 27,750,000 | 110,000 | 27,860,000 | +| first-letters | 19,850,000 | 80,000 | 19,940,000 | +| reverse-sort-words-en | 10,700,000 | 120,000 | 10,820,000 | +| distractorless | 27,550,000 | 120,000 | 27,680,000 | + +For Direct gpt-3.5-turbo-0125: + +| Distractor variant | Input | Output | Total | +| --------------------- | --------- | ------ | --------- | +| which-is-heavier | 1,200,000 | 10,000 | 1,210,000 | +| ambiguous-sentences | 1,540,000 | 20,000 | 1,550,000 | +| first-letters | 2,120,000 | 20,000 | 2,140,000 | +| reverse-sort-words-en | 910,000 | 20,000 | 940,000 | +| distractorless | 1,250,000 | 20,000 | 1,270,000 | + +For Direct gpt-4-base: + +| Distractor variant | Input | Output | Total | +| --------------------- | ---------- | --------- | ---------- | +| which-is-heavier | 16,950,000 | 3,670,000 | 20,620,000 | +| ambiguous-sentences | 23,100,000 | 4,390,000 | 27,490,000 | +| first-letters | 25,310,000 | 4,870,000 | 30,180,000 | +| reverse-sort-words-en | 14,380,000 | 2,760,000 | 17,140,000 | +| distractorless | 24,460,000 | 5,000,000 | 29,460,000 | + +For CoT gpt-4-0125-preview: + +| Distractor variant | Input | Output | Total | +| --------------------- | ----------- | --------- | ----------- | +| which-is-heavier | 263,600,000 | 1,900,000 | 265,500,000 | +| ambiguous-sentences | 383,500,000 | 2,700,000 | 386,200,000 | +| first-letters | 251,700,000 | 1,700,000 | 253,400,000 | +| reverse-sort-words-en | 236,700,000 | 2,100,000 | 238,800,000 | +| distractorless | 395,500,000 | 2,400,000 | 398,000,000 | + +For CoT gpt-3.5-turbo-0125: + +| Distractor variant | Input | Output | Total | +| --------------------- | ---------- | ------- | ---------- | +| which-is-heavier | 10,100,000 | 190,000 | 10,280,000 | +| ambiguous-sentences | 7,510,000 | 140,000 | 7,650,000 | +| first-letters | 16,450,000 | 220,000 | 16,670,000 | +| reverse-sort-words-en | 4,690,000 | 150,000 | 4,840,000 | +| distractorless | 30,230,000 | 310,000 | 30,540,000 | + +## Future modifications + +- Extending the range of distractors considered, either by incorporating more + evals or designing new distractor variants. +- Experiment with multiple distractor sources in a single eval run, to see if + the variety of distractors affects the model's robustness. + +## Version History + +- v0: Initial version released + +## Contribution Statement + +Eval design, implementation, and results evaluation were primarily conducted by +Giulio Starace, under the guidance of (alphabetically by last-name) Steven +Adler, Andrei Alexandru, James Aung, and Chan Jun Shern who provided research +input, report revisions, and project management support. diff --git a/evals/elsuite/already_said_that/distractors.py b/evals/elsuite/already_said_that/distractors.py new file mode 100644 index 0000000000..712d172ece --- /dev/null +++ b/evals/elsuite/already_said_that/distractors.py @@ -0,0 +1,133 @@ +import os +from dataclasses import dataclass +from pathlib import Path + +import evals + + +@dataclass +class DistractorSample: + question: str + ideal: str + + +VARIANTS = { + "which-is-heavier", + "ambiguous-sentences", + "first-letters", + "reverse-sort-words-eng", + "distractorless", +} + + +def proc_which_is_heavier(samples) -> list[DistractorSample]: + distractor_samples = [] + for sample in samples: + # get rid of ' Answer Yes or No' + question = sample["input"][1]["content"][:-17] + ideal = sample["ideal"].lower() + distractor_samples.append(DistractorSample(question, ideal)) + return distractor_samples + + +def proc_distractors_first_letters(samples) -> list[DistractorSample]: + distractor_samples = [] + for sample in samples: + question = sample["input"][1]["content"] + ideal = sample["ideal"].lower() + distractor_samples.append(DistractorSample(question, ideal)) + return distractor_samples + + +def proc_distractors_ambiguous_sentences(samples) -> list[DistractorSample]: + distractor_samples = [] + for sample in samples: + sample_content = sample["input"][1]["content"] + question = f"{sample_content}" + ideal = sample["ideal"].lower() + distractor_samples.append(DistractorSample(question, ideal)) + return distractor_samples + + +def proc_distractors_reverse_sort_words_eng(samples) -> list[DistractorSample]: + distractor_samples = [] + for sample in samples: + # cut " (respond as concisely as possible and only include the comma-separated words in your response):" + instruction = sample["input"][0]["content"][:-96] + sample_content = sample["input"][1]["content"] + question = f"{instruction}: {sample_content}" + ideal = sample["ideal"].lower() + distractor_samples.append(DistractorSample(question, ideal)) + return distractor_samples + + +variant_to_processor = { + "which-is-heavier": proc_which_is_heavier, + "first-letters": proc_distractors_first_letters, + "ambiguous-sentences": proc_distractors_ambiguous_sentences, + "reverse-sort-words-eng": proc_distractors_reverse_sort_words_eng, +} + + +def get_basic_distractor_example() -> DistractorSample: + """ + An arbitrary distractor example used in the task description for the + distractorless variant + """ + return DistractorSample(question="What is the capital of Italy?", ideal="rome") + + +def get_distractors(variant: str) -> list[DistractorSample]: + """ + Gets and optionally processes the corpus of distractor questions for variant + """ + assert variant in VARIANTS, f"Invalid variant {variant}, expected one of {VARIANTS}" + if variant == "distractorless": + # single element will be pop()ed for the task description, leaving an empty list + return [get_basic_distractor_example()] + + samples = get_samples(variant) + + process_variant_fn = variant_to_processor[variant] + processed_samples = process_variant_fn(samples) + + return processed_samples + + +def get_samples(eval_name) -> list[dict]: + """ + Gets the samples from the samples_jsonl associated with + a given eval. + + Adapted from evals.eval.Eval.get_samples + """ + registry = evals.registry.Registry() + eval_spec = registry.get_eval(eval_name) + samples_path = eval_spec.args["samples_jsonl"] + registry_path = eval_spec.registry_path + samples_full_path = get_full_path(samples_path, registry_path) + return evals.data.get_jsonl(samples_full_path.as_posix()) + + +def get_full_path(data_path, registry_path) -> Path: + if os.path.isfile(data_path): + return Path(data_path) + + return registry_path / "data" / data_path + + +def get_distractor_word(question: str) -> str: + """ + Takes the last word of the question (stripped of punctuation and lower-cased) + To be shown in the task description example + """ + words = question.split() + last_word = words[-1] + last_word = last_word.strip(".,!?") + return last_word.lower() + + +if __name__ == "__main__": + # just for testing + distractors = get_distractors("rectangles") + print(distractors[0]) diff --git a/evals/elsuite/already_said_that/eval.py b/evals/elsuite/already_said_that/eval.py new file mode 100644 index 0000000000..2fa495c702 --- /dev/null +++ b/evals/elsuite/already_said_that/eval.py @@ -0,0 +1,160 @@ +import random +from collections import deque +from typing import Any, Deque, Optional + +import numpy as np + +from evals.elsuite.already_said_that import distractors, prompts, utils +from evals.eval import SolverEval +from evals.record import RecorderBase, record_metrics +from evals.solvers.solver import Solver +from evals.task_state import Message, TaskState + + +class AlreadySaidThat(SolverEval): + def __init__( + self, + distractor_variant: str, + adversarial: bool = True, + max_turns: int = 100, + n_samples: Optional[int] = 250, + *args, + **kwargs, + ): + super().__init__(*args, **kwargs) + self.distractor_variant = distractor_variant + self.distractor_data = distractors.get_distractors(self.distractor_variant) + distractor_example = self.distractor_data.pop() + distractor_word = distractors.get_distractor_word(distractor_example.question) + self.task_description = prompts.TASK_DESCRIPTION.format( + distractor_question=distractor_example.question, + distractor_answer=distractor_example.ideal, + distractor_word=distractor_word, + ) + self.num_distractors = len(self.distractor_data) + self.max_turns = max_turns + self.adversarial = adversarial + self.n_samples = n_samples + self.rng = random.Random(self.seed) + + def eval_sample(self, solver: Solver, sample: dict, rng: random.Random) -> None: + words = sample["words"] + # make a deque of the (shuffled) distractor data, will be faster to rotate + distractor_data = deque(rng.sample(self.distractor_data, k=self.num_distractors)) + + conversation_metrics = self._conversation_loop(solver, words, distractor_data, rng) + + record_metrics(**conversation_metrics) + + def _conversation_loop( + self, + solver: Solver, + words: list[str], + distractor_data: Deque[dict[str, str]], + rng, + ) -> dict[str, Any]: + convo_metrics = { + "num_distractors": 0, + "num_turns": 0, + "was_false_pos": False, + "was_false_neg": False, + "violation_occurred": False, + "distractor_accuracy": np.nan, + } + + words_prev_shown = set() + words_not_shown = set(words) + words_from_solver = set() + words_from_distractors = set() + + distractor_correctness = [] + + task_state = TaskState(task_description=self.task_description) + + while convo_metrics["num_turns"] < self.max_turns: + # conversation + distracting_words = ( + words_from_solver.union(words_from_distractors) if self.adversarial else set() + ) + message, message_words, distractor_added = utils.build_message( + words_not_shown=words_not_shown, + words_prev_shown=words_prev_shown, + distracting_words=distracting_words, + rng=rng, + distractor_sample=distractor_data[0] if distractor_data else None, + ) + task_state.messages.append(message) + solver_output = solver(task_state).output + task_state.messages.append(Message(role="assistant", content=solver_output)) + + # track performance + parsing_results = utils.parse_solver_output( + solver_output, message_words, words_prev_shown, distractor_added + ) + convo_metrics["violation_occurred"] = parsing_results["violation_occurred"] + mistake_made = parsing_results["mistake_made"] + if distractor_added is not None: + distractor_correctness.append(not mistake_made) + convo_metrics["num_distractors"] += 1 + words_from_distractors.update(message_words) + # move the distractor we just used to the end of the queue + distractor_data.rotate(-1) + elif convo_metrics["violation_occurred"] or (mistake_made and distractor_added is None): + convo_metrics["was_false_pos"] = parsing_results["false_positive"] + convo_metrics["was_false_neg"] = parsing_results["false_negative"] + break + else: + words_prev_shown.update(message_words) + words_not_shown.difference_update(message_words) + words_from_solver.update(parsing_results["solver_words"]) + convo_metrics["num_turns"] += 1 + + convo_metrics["distractor_accuracy"] = ( + np.mean(distractor_correctness) if distractor_correctness else np.nan + ) + + return convo_metrics + + def run(self, recorder: RecorderBase): + samples = self._get_samples() + self.eval_all_samples(recorder, samples) + logged_metrics: list[dict] = recorder.get_metrics() + + agg_metrics = self._compute_agg_metrics(logged_metrics) + return agg_metrics + + def _compute_agg_metrics(self, logged_metrics: list[dict]) -> dict: + num_distractors = np.array([x["num_distractors"] for x in logged_metrics]) + num_turns = np.array([x["num_turns"] for x in logged_metrics]) + + agg_metrics = { + # distractors + "avg_num_distractors": np.mean(num_distractors), + "stddev_num_distractors": np.std(num_distractors), + "median_num_distractors": np.median(num_distractors), + "max_num_distractors": np.max(num_distractors), + "min_num_distractors": np.min(num_distractors), + # turns + "avg_num_turns": np.mean(num_turns), + "stddev_num_turns": np.std(num_turns), + "median_num_turns": np.median(num_turns), + "max_num_turns": np.max(num_turns), + "min_num_turns": np.min(num_turns), + # task stats + "false_positive_rate": np.mean([x["was_false_pos"] for x in logged_metrics]), + "false_negative_rate": np.mean([x["was_false_neg"] for x in logged_metrics]), + # distractor stats + "avg_distractor_accuracy": np.nanmean( + [x["distractor_accuracy"] for x in logged_metrics] + ), + # violation + "violation_rate": np.mean([x["violation_occurred"] for x in logged_metrics]), + } + # necessary for serialization, json doesn't like np floats + agg_metrics = {k: float(v) for k, v in agg_metrics.items()} + return agg_metrics + + def _get_samples(self) -> list[dict]: + samples = self.get_samples() + samples = self.rng.sample(samples, min(self.n_samples, len(samples))) + return samples diff --git a/evals/elsuite/already_said_that/prompts.py b/evals/elsuite/already_said_that/prompts.py new file mode 100644 index 0000000000..cd3bbb4490 --- /dev/null +++ b/evals/elsuite/already_said_that/prompts.py @@ -0,0 +1,28 @@ +TASK_DESCRIPTION = """\ +You will be presented with a list of words, in sequence.\ + Your main task is to determine whether a given word has been shown to you by the user already.\ + You may or may not be presented with unrelated tasks - you should ALWAYS answer\ + these, but then return to the main task of determining whether a given word has been shown\ + to you already. Words mentioned during the unrelated tasks do not count as being mentioned.\ + +You should respond with [answer: ], where is either yes or no for the\ + main task or a valid answer for the unrelated tasks. +Here is an example interaction: + +```example +input: MAIN TASK - hello +response: [answer: no] +input: MAIN TASK - hello +response: [answer: yes] +input: {distractor_question} +response: [answer: {distractor_answer}] +input: MAIN TASK - {distractor_word} +response: [answer: no] # words mentioned during unrelated tasks do not count +input: MAIN TASK - main +response: [answer: no] +input: MAIN TASK - main +response: [answer: yes] +``` + +The task will begin now.\ +""" diff --git a/evals/elsuite/already_said_that/scripts/data.sh b/evals/elsuite/already_said_that/scripts/data.sh new file mode 100755 index 0000000000..03c88ece88 --- /dev/null +++ b/evals/elsuite/already_said_that/scripts/data.sh @@ -0,0 +1,4 @@ +#!/bin/bash +echo "Generating word samples..." +python evals/elsuite/already_said_that/scripts/gen_data.py --n_samples 500 --jsonl_dir evals/registry/data/already_said_that --seed 0 +echo "Done." diff --git a/evals/elsuite/already_said_that/scripts/gen_data.py b/evals/elsuite/already_said_that/scripts/gen_data.py new file mode 100644 index 0000000000..94f827d0d6 --- /dev/null +++ b/evals/elsuite/already_said_that/scripts/gen_data.py @@ -0,0 +1,73 @@ +import argparse +import os +import random +import json + +import nltk +from nltk.corpus import wordnet +from tqdm.auto import tqdm + + +def process_wordnet() -> list[str]: + """ + Process the wordnet corpus and save it to the given directory + License info: https://www.nltk.org/nltk_data (number 102) + """ + # download wordnet corpus if necessary + nltk.download("wordnet", force=True) + wordnet_words = wordnet.words() + # get all unique alpha words from wordnet corpus + words = set() + for word in tqdm(wordnet_words): + if word.isalpha(): + words.add(word.lower()) + + return list(words) + + +def gen_sample(words_corpus: list[str], n_words, rng: random.Random) -> dict: + words = rng.sample(words_corpus, n_words) + return {"words": words} + + +def gen_samples(n_samples: int, n_words: int, rng: random.Random) -> list[dict]: + words = process_wordnet() + samples = [] + for _ in tqdm(range(n_samples)): + sample = gen_sample(words, n_words, rng) + samples.append(sample) + return samples + + +def write_to_jsonl( + samples: list[dict], + jsonl_path: str, +): + with open(jsonl_path, "w") as f: + for sample in samples: + f.write(json.dumps(sample) + "\n") + + +def main(args: argparse.Namespace): + rng = random.Random(args.seed) + samples = gen_samples(args.n_samples, args.n_words, rng) + os.makedirs(args.jsonl_dir, exist_ok=True) + jsonl_path = os.path.join(args.jsonl_dir, f"{args.n_samples}_{args.n_words}.jsonl") + write_to_jsonl(samples, jsonl_path) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + + parser.add_argument("--n_samples", type=int, default=500) + parser.add_argument( + "--n_words", type=int, default=100, help="Number of words in each sample" + ) + parser.add_argument("--seed", type=int, default=0) + parser.add_argument( + "--jsonl_dir", type=str, default="./evals/registry/data/already_said_that/" + ) + + args = parser.parse_args() + + main(args) diff --git a/evals/elsuite/already_said_that/scripts/make_plots.py b/evals/elsuite/already_said_that/scripts/make_plots.py new file mode 100644 index 0000000000..ede36291ec --- /dev/null +++ b/evals/elsuite/already_said_that/scripts/make_plots.py @@ -0,0 +1,328 @@ +from pathlib import Path +import argparse +import json + +from tqdm.auto import tqdm +import numpy as np +import matplotlib.pyplot as plt +import seaborn as sns + +from evals.utils import log_utils + + +def zero_if_none(input_num): + if input_num is None: + return 0 + else: + return input_num + + +MODELS = [ + "cot/gpt-4-turbo-preview", + "gpt-4-turbo-preview", + "cot/gpt-3.5-turbo", + "gpt-3.5-turbo", + "gpt-4-base", + "gemini-pro", + "mixtral-8x7b-instruct", + "llama-2-70b-chat", + "random_baseline", +] +# separate list for OAI models for token counting, not supported in others. +OAI_MODELS = [ + "cot/gpt-4-turbo-preview", + "gpt-4-turbo-preview", + "cot/gpt-3.5-turbo", + "gpt-3.5-turbo", + "gpt-4-base", +] + + +DISTRACTORS = [ + "which-is-heavier", + "ambiguous-sentences", + "first-letters", + "reverse-sort-words-eng", + "distractorless", +] + + +MODEL_TO_LABEL = { + "cot/gpt-4-turbo-preview": "CoT gpt-4-0125-preview", + "cot/gpt-3.5-turbo": "CoT gpt-3.5-turbo-0125", + "gpt-4-turbo-preview": "Direct gpt-4-0125-preview", + "gpt-3.5-turbo": "Direct gpt-3.5-turbo-0125", + "gpt-4-base": "HHH gpt-4-base", + "gemini-pro": "Direct gemini-pro-1.0", + "mixtral-8x7b-instruct": "Direct mixtral-8x7b-instruct", + "llama-2-70b-chat": "Direct llama-2-70b-chat", + "random_baseline": "Random Baseline", +} + +NUM_REPEATS = 3 + +PLOT_STATS = ["avg_num_turns", "avg_distractor_accuracy"] +JSON_STATS = [ + "avg_num_turns", + "avg_distractor_accuracy", + "false_positive_rate", + "false_negative_rate", + "violation_rate", +] + +STAT_TO_MAX = { + "avg_num_distractors": 100 / 3, # distractors shown every 1/3 of the time + "avg_num_turns": 100, # best case, we run out of steps + "avg_distractor_accuracy": 1, + "false_positive_rate": 1, + "false_negative_rate": 1, + "violation_rate": 1, +} + +STAT_TO_LABEL = { + "avg_num_distractors": "Average number of distractors shown before failure", + "avg_num_turns": "Average number of turns before failure", + "avg_distractor_accuracy": "Average accuracy on distractor task", + "false_positive_rate": "False positive rate", + "false_negative_rate": "False negative rate", + "violation_rate": "Violation rate", +} + + +def make_results_dict(log_dir: Path) -> dict: + results_dict = prepare_results_dict() + results_dict = fill_results_dict(results_dict, log_dir) + return results_dict + + +def prepare_results_dict() -> dict: + results_dict = { + stat: { + distractor: { + model: {"raw": [], "mean": 0, "std_err": 0} for model in MODELS + } + for distractor in DISTRACTORS + } + for stat in [ + "avg_num_distractors", + "avg_num_turns", + "avg_distractor_accuracy", + "false_positive_rate", + "false_negative_rate", + "violation_rate", + ] + } + return results_dict + + +def fill_results_dict(results_dict: dict, log_dir: Path) -> dict: + print("Parsing logs...") + final_results = log_utils.get_final_results_from_dir(log_dir) + specs = log_utils.get_specs_from_dir(log_dir) + files = list(final_results.keys()) + + for file in tqdm(files): + final_result = final_results[file] + spec = specs[file] + distractor = spec["split"] + model = get_model(spec) + for stat in results_dict: + results_dict[stat][distractor][model]["raw"].append(final_result[stat]) + for file in tqdm(files): + spec = specs[file] + distractor = spec["split"] + model = get_model(spec) + # compute means/std_errs + for stat in results_dict: + data_points = results_dict[stat][distractor][model]["raw"] + results_dict[stat][distractor][model]["mean"] = np.mean(data_points) + results_dict[stat][distractor][model]["std_err"] = np.std( + data_points + ) / np.sqrt(NUM_REPEATS) + return results_dict + + +def get_model(spec): + # this is hilariously ugly but it works for now (sorry) + if "cot/gpt-4-turbo-preview" in spec["completion_fns"][0]: + return "cot/gpt-4-turbo-preview" + elif "gpt-4-turbo-preview" in spec["completion_fns"][0]: + return "gpt-4-turbo-preview" + elif "cot/gpt-3.5-turbo" in spec["completion_fns"][0]: + return "cot/gpt-3.5-turbo" + elif "gpt-3.5-turbo" in spec["completion_fns"][0]: + return "gpt-3.5-turbo" + elif "gpt-4-base" in spec["completion_fns"][0]: + return "gpt-4-base" + elif "gemini-pro" in spec["completion_fns"][0]: + return "gemini-pro" + elif "mixtral-8x7b-instruct" in spec["completion_fns"][0]: + return "mixtral-8x7b-instruct" + elif "llama-2-70b-chat" in spec["completion_fns"][0]: + return "llama-2-70b-chat" + elif "random_baseline" in spec["completion_fns"][0]: + return "random_baseline" + + +def make_bar_plot(results_dict: dict, stat: str, save_path: Path): + sns.set_context("paper") + sns.set_style("whitegrid") + + fig, ax = plt.subplots(1, 1, figsize=(8, 7), dpi=300) + + data = results_dict[stat] + + # the random baseline isn't plotted as bars + models = MODELS[:-1] + + distractors = [ + "which-is-heavier", + "ambiguous-sentences", + "first-letters", + "reverse-sort-words-eng", + ] + + width = 0.15 + if stat != "avg_distractor_accuracy": + distractors.append("distractorless") + diffs = [-width * 2, -width / 1, 0, width / 1, width * 2] + ax.axvline(STAT_TO_MAX[stat], label="maximum", linestyle="--", color="grey") + + # random baseline is roughly the same for all distractors; pick one for simplicity + random_baseline = data["first-letters"]["random_baseline"]["mean"] + + ax.axvline( + random_baseline, + label=MODEL_TO_LABEL["random_baseline"], + linestyle="-.", + color="black", + ) + + # make legend order match bar order, idk why matplotlib reverses them + legend_indices = [0, 1, 6, 5, 4, 3, 2] + else: + diffs = [-width * 1.5, -width / 2, width / 2, width * 1.5] + legend_indices = list(range(len(distractors)))[::-1] + + means = [[data[dis][model]["mean"] for dis in distractors] for model in models] + std_errs = [ + [data[dis][model]["std_err"] for dis in distractors] for model in models + ] + cmap = plt.get_cmap("Set3") + colors = np.array([cmap(i) for i in range(len(distractors))]) + + x = np.arange(len(models)) # the label locations + + distractor_bars = [] + for i, distractor in enumerate(distractors): + bar = ax.barh( + x + diffs[i], + [mean[i] for mean in means], + width, + xerr=[err[i] for err in std_errs], + label=distractor, + color=colors[i] if distractor != "distractorless" else "black", + ) + distractor_bars.append(bar) + + ax.set_xlabel(STAT_TO_LABEL[stat]) + x_max = STAT_TO_MAX[stat] + 0.05 * STAT_TO_MAX[stat] + ax.set_xlim([0, x_max]) + ax.set_yticks(x) + ax.set_yticklabels([MODEL_TO_LABEL[model] for model in models]) + handles, labels = ax.get_legend_handles_labels() + ax.legend( + [handles[i] for i in legend_indices], + [labels[i] for i in legend_indices], + loc="best", + ) + + for bar, distractor in zip(distractor_bars, distractors): + ax.bar_label( + bar, + label_type="edge", + fmt="%.2f", + # color="white" if distractor == "distractorless" else "black", + fontsize=8, + ) + + # get rid of horizontal grid lines + ax.grid(axis="y", which="both") + + fig.set_tight_layout(True) + + plt.savefig(save_path, bbox_inches="tight", dpi=300) + + +def count_tokens(log_dir) -> dict[str, dict[str, dict[str, int]]]: + """ + model -> distractor -> input, output, total tokens + """ + token_counts = { + model: { + distractor: {kind: 0 for kind in ["input", "output", "total"]} + for distractor in DISTRACTORS + } + for model in OAI_MODELS + } + globbed_logs = list(log_dir.glob("*.log")) + already_examined = set() + for log in tqdm(globbed_logs, total=len(globbed_logs), desc="Counting tokens"): + spec = log_utils.extract_spec(log) + distractor = spec["split"] + model = get_model(spec) + if model not in OAI_MODELS: + continue + + # dont care about repeats, this is a rough estimate anyway + if (model, distractor) in already_examined: + continue + already_examined.add((model, distractor)) + + samplings = log_utils.extract_individual_results(log, "sampling") + for sampling in samplings: + usage = sampling["usage"] + token_counts[model][distractor]["input"] += zero_if_none( + usage["prompt_tokens"] + ) + token_counts[model][distractor]["output"] += zero_if_none( + usage["completion_tokens"] + ) + token_counts[model][distractor]["total"] += zero_if_none( + usage["total_tokens"] + ) + return token_counts + + +def main(args: argparse.Namespace): + log_dir = Path(args.log_dir) + save_dir = Path(args.save_dir) + save_dir.mkdir(exist_ok=True, parents=True) + + results_dict = make_results_dict(log_dir) + + for stat in tqdm(PLOT_STATS, desc="Making plots"): + save_path = save_dir / f"{stat}.png" + make_bar_plot(results_dict, stat, save_path) + + for stat in tqdm(JSON_STATS, desc="Saving JSONs"): + save_path = save_dir / f"{stat}.json" + with open(save_path, "w") as f: + json.dump(results_dict[stat], f, indent=2) + + token_counts = count_tokens(log_dir) + save_path = save_dir / "token_counts.json" + with open(save_path, "w") as f: + json.dump(token_counts, f, indent=2) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--log_dir", type=str, required=True, help="Where the logs are stored" + ) + parser.add_argument( + "--save_dir", type=str, required=True, help="Where to save the plots" + ) + args = parser.parse_args() + main(args) diff --git a/evals/elsuite/already_said_that/scripts/run_experiments.sh b/evals/elsuite/already_said_that/scripts/run_experiments.sh new file mode 100755 index 0000000000..dd300f6141 --- /dev/null +++ b/evals/elsuite/already_said_that/scripts/run_experiments.sh @@ -0,0 +1,102 @@ +#!/bin/bash + +usage() { + echo "Usage: $0 -l logdir" + echo " -l logdir Specify the directory for log files" + exit 1 +} + +# Check if no arguments were provided +if [ $# -eq 0 ]; then + usage + exit 1 +fi + +# Parse command-line options +while getopts 's:l:' flag; do + case "${flag}" in + l) logdir=${OPTARG} ;; + *) usage ;; + esac +done + +# Check if mandatory arguments were provided +if [ -z "$logdir" ]; then + usage + exit 1 +fi + +NUM_REPEATS=3 + +export EVALS_THREADS=10 +export EVALS_THREADS_TIMEOUT=5 + +declare -a SOLVERS=( + # gpt-4-turbo-preview + "generation/direct/gpt-4-turbo-preview" + "already_said_that/cot/gpt-4-turbo-preview" + # gpt-3.5-turbo + "generation/direct/gpt-3.5-turbo" + "already_said_that/cot/gpt-3.5-turbo" + # gpt-4-base + "generation/hhh/gpt-4-base" + # mixtral-8x7b-instruct + "generation/direct/mixtral-8x7b-instruct" + # llama chat 70b + "generation/direct/llama-2-70b-chat" + # gemini-pro + "generation/direct/gemini-pro" + # random baseline + "already_said_that/random_baseline" +) + +declare -a DISTRACTORS=( + "reverse-sort-words-eng" + "first-letters" + "ambiguous-sentences" + "which-is-heavier" + "distractorless" +) + +# Check if GEMINI_API_KEY is set +if [ -z "$GEMINI_API_KEY" ]; then + echo "Enter your Gemini API Key:" + read -s GEMINI_API_KEY + export GEMINI_API_KEY +fi + +# Check if TOGETHER_API_KEY is set +if [ -z "$TOGETHER_API_KEY" ]; then + echo "Enter your Together API Key:" + read -s TOGETHER_API_KEY + export TOGETHER_API_KEY +fi + +start_time=$SECONDS +for solver in "${SOLVERS[@]}"; do + + if [[ $solver == *"gemini"* ]]; then + export EVALS_SEQUENTIAL=1 + else + export EVALS_SEQUENTIAL=0 + fi + + solver_dotted=${solver//\//.} + + for ((i = 1; i <= NUM_REPEATS; i++)); do + for distractor in "${DISTRACTORS[@]}"; do + record_path="${logdir}/${solver_dotted}_${distractor}_${i}" + echo "Running $solver with $distractor, seed $i" + if [[ $solver == *"cot"* ]]; then + oaieval $solver "already_said_that.${distractor}" \ + --seed $i --record_path "$record_path.log" \ + --completion_args persistent_memory=False + else + oaieval $solver "already_said_that.${distractor}" \ + --record_path "$record_path.log" \ + --seed $i + fi + done + done +done +echo "Total time: $((SECONDS - start_time)) seconds" diff --git a/evals/elsuite/already_said_that/solvers.py b/evals/elsuite/already_said_that/solvers.py new file mode 100644 index 0000000000..5eed8c84a6 --- /dev/null +++ b/evals/elsuite/already_said_that/solvers.py @@ -0,0 +1,42 @@ +import random +from typing import Any + +from evals.solvers.solver import NestedSolver, Solver, SolverResult, SolverSpec +from evals.task_state import TaskState + + +class RandomBaselineSolver(Solver): + def __init__(self, registry: Any = None): + super().__init__() + + def _solve(self, task_state: TaskState, **kwargs) -> SolverResult: + answer = random.choice(["yes", "no"]) + return SolverResult(output=f"[answer: {answer}]") + + +class AlreadySaidThatHuman(NestedSolver): + def __init__(self, human_cli_solver: SolverSpec, *args, **kwargs): + super().__init__(human_cli_solver=human_cli_solver, *args, **kwargs) + + @property + def human_cli_solver(self) -> Solver: + return self.get_solver("human_cli_solver") + + def _solve(self, task_state: TaskState) -> SolverResult: + human_result = self.human_cli_solver(task_state=task_state) + answer = self._map_to_yesno(human_result.output) + return SolverResult( + output=f"[answer: {answer}]", + ) + + def _map_to_yesno(self, yesno_ish): + """ + Maps Y, y, Yes,1, yes, N, n, No, no, 0 to yes or no, respectively. + """ + if yesno_ish.lower() in {"y", "yes", "1"}: + return "yes" + elif yesno_ish.lower() in {"n", "no", "0"}: + return "no" + else: + # for other answers, return the original answer + return yesno_ish diff --git a/evals/elsuite/already_said_that/test_distractors.py b/evals/elsuite/already_said_that/test_distractors.py new file mode 100644 index 0000000000..d565274cbb --- /dev/null +++ b/evals/elsuite/already_said_that/test_distractors.py @@ -0,0 +1,187 @@ +import pytest + +from evals.elsuite.already_said_that.distractors import ( + proc_distractors_ambiguous_sentences, + proc_distractors_first_letters, + proc_distractors_reverse_sort_words_eng, + proc_which_is_heavier, +) + + +@pytest.fixture +def which_is_heavier_samples(): + return [ + { + "input": [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": "Is 2 kilograms of eggshells heavier than 1 kilogram of gold bars? Answer Yes or No", + }, + ], + "ideal": "Yes", + }, + { + "input": [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": "Is 10 kilograms of granite slabs heavier than 20 kilograms of balloons? Answer Yes or No", + }, + ], + "ideal": "No", + }, + ] + + +@pytest.fixture +def first_letters_samples(): + return [ + { + "input": [ + { + "role": "system", + "content": "You are a helpful assistant. Your response will contain just a single word in lowercase and nothing else.", + }, + { + "role": "user", + "content": 'What is the word obtained from concatenating the first letters of the words in the following sentence: "Dear Elena find irregular new energies."?', + }, + ], + "ideal": "define", + }, + { + "input": [ + { + "role": "system", + "content": "You are a helpful assistant. Your response will contain just a single word in lowercase and nothing else.", + }, + { + "role": "user", + "content": 'What is the word obtained from concatenating the first letters of the words in the following sentence: "Bring roses on a doorstep."?', + }, + ], + "ideal": "broad", + }, + ] + + +@pytest.fixture +def ambiguous_sentences_samples(): + return [ + { + "input": [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": "This morning, Joey built a sand castle on the beach, and put a toy flag in the highest tower, but this afternoon the tide knocked it down.. What did the the tide knock down?", + }, + ], + "ideal": "castle", + }, + { + "input": [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": "Sam took French classes from Adam, because he was eager to speak it fluently.. Who was eager to speak French fluently?", + }, + ], + "ideal": "Sam", + }, + ] + + +@pytest.fixture +def reverse_sort_words_eng_samples(): + return [ + { + "input": [ + { + "role": "system", + "content": "Sort the following comma-separated words in reversed alphabetical order (respond as concisely as possible and only include the comma-seperated words in your response):", + }, + { + "role": "user", + "content": "gargoyle, porcupine, whirlwind, zest, lily, molasses, penguin, satchel, igloo, viper", + }, + ], + "ideal": "zest, whirlwind, viper, satchel, porcupine, penguin, molasses, lily, igloo, gargoyle", + }, + { + "input": [ + { + "role": "system", + "content": "Sort the following comma-separated words in reversed alphabetical order (respond as concisely as possible and only include the comma-seperated words in your response):", + }, + { + "role": "user", + "content": "marigold, opal, labyrinth, silhouette, whirlpool, trumpet, forge, quill, knapsack, emblem", + }, + ], + "ideal": "whirlpool, trumpet, silhouette, quill, opal, marigold, labyrinth, knapsack, forge, emblem", + }, + ] + + +def test_proc_distractors_which_is_heavier(which_is_heavier_samples): + result = proc_which_is_heavier(which_is_heavier_samples) + assert len(result) == 2 + assert result[0].question == "Is 2 kilograms of eggshells heavier than 1 kilogram of gold bars?" + assert result[0].ideal == "yes" + assert ( + result[1].question + == "Is 10 kilograms of granite slabs heavier than 20 kilograms of balloons?" + ) + assert result[1].ideal == "no" + + +def test_proc_distractors_first_letter(first_letters_samples): + result = proc_distractors_first_letters(first_letters_samples) + assert len(result) == 2 + assert ( + result[0].question + == 'What is the word obtained from concatenating the first letters of the words in the following sentence: "Dear Elena find irregular new energies."?' + ) + assert result[0].ideal == "define" + assert ( + result[1].question + == 'What is the word obtained from concatenating the first letters of the words in the following sentence: "Bring roses on a doorstep."?' + ) + assert result[1].ideal == "broad" + + +def test_proc_distractors_ambiguous_sentences(ambiguous_sentences_samples): + result = proc_distractors_ambiguous_sentences(ambiguous_sentences_samples) + assert len(result) == 2 + assert ( + result[0].question + == "This morning, Joey built a sand castle on the beach, and put a toy flag in the highest tower, but this afternoon the tide knocked it down.. What did the the tide knock down?" + ) + assert result[0].ideal == "castle" + assert ( + result[1].question + == "Sam took French classes from Adam, because he was eager to speak it fluently.. Who was eager to speak French fluently?" + ) + assert result[1].ideal == "sam" + + +def test_proc_distractors_reverse_sort_words_eng(reverse_sort_words_eng_samples): + result = proc_distractors_reverse_sort_words_eng(reverse_sort_words_eng_samples) + assert len(result) == 2 + assert ( + result[0].question + == "Sort the following comma-separated words in reversed alphabetical order: gargoyle, porcupine, whirlwind, zest, lily, molasses, penguin, satchel, igloo, viper" + ) + assert ( + result[0].ideal + == "zest, whirlwind, viper, satchel, porcupine, penguin, molasses, lily, igloo, gargoyle" + ) + assert ( + result[1].question + == "Sort the following comma-separated words in reversed alphabetical order: marigold, opal, labyrinth, silhouette, whirlpool, trumpet, forge, quill, knapsack, emblem" + ) + assert ( + result[1].ideal + == "whirlpool, trumpet, silhouette, quill, opal, marigold, labyrinth, knapsack, forge, emblem" + ) diff --git a/evals/elsuite/already_said_that/utils.py b/evals/elsuite/already_said_that/utils.py new file mode 100644 index 0000000000..f535fd9708 --- /dev/null +++ b/evals/elsuite/already_said_that/utils.py @@ -0,0 +1,171 @@ +import random +import re +from typing import Any, Optional + +from evals.elsuite.already_said_that.distractors import DistractorSample +from evals.task_state import Message + + +def build_message( + words_not_shown: set[str], + words_prev_shown: set[str], + distracting_words: set[str], + rng: random.Random, + distractor_sample: Optional[DistractorSample] = None, +) -> tuple[Message, list[str], Optional[DistractorSample]]: + """ + Builds the TaskState.Message for a given sample. + Randomly chooses whether to show a word (base task) or ask a question (distractor). + In case of base task, the words are randomly chosen either from base_words or from + distracting_words, i.e. words that have been mentioned by the solver or as part of + the distractor task in previous rounds. + + Args: + words_not_shown: set of words that have not been shown to the solver + words_prev_shown: set of words that have been shown to the solver in + distracting_words: set of words that have been mentioned by the solver or as part of + the distractor questions in previous rounds + distractor_sample: the distractor sample to use in case of a distractor round + rng: seeded random number generator, for reproducibility + + Returns: + message: the TaskState.Message + message_words: list of words that are part of the message + distractor_added: The distractor sample if the message is a distractor question, + else None + """ + # we mainly do the base task, but sometimes we do a distractor question + is_a_distractor_question = rng.choices([True, False], weights=[1 / 3, 2 / 3], k=1)[0] + if not is_a_distractor_question or distractor_sample is None: + message, message_words = build_base_task_message( + words_not_shown, words_prev_shown, distracting_words, rng + ) + distractor_added = None + else: + message, message_words = build_distractor_question_message(distractor_sample) + distractor_added = distractor_sample + return message, message_words, distractor_added + + +def build_base_task_message( + words_not_shown: set[str], + words_prev_shown: set[str], + distracting_words: set[str], + rng, +) -> tuple[Message, list[str]]: + distract = rng.choice([True, False]) + if distract and len(distracting_words) > 0: + word = rng.choice(list(distracting_words)) + else: + # balance between showing new words and repeating old ones + word_pool = rng.choice([words_not_shown, words_prev_shown]) + if len(word_pool) > 0: + word = rng.choice(list(word_pool)) + else: + # this is always non-empty + word = rng.choice(list(words_not_shown)) + word = word.lower() + message = Message(role="user", content=f"MAIN TASK - {word}") + return message, [word] + + +def build_distractor_question_message( + distractor_sample: list[str], +) -> tuple[Message, list[str]]: + question = distractor_sample.question + message = Message(role="user", content=question) + question_words = find_alpha_words(question) + return message, question_words + + +def find_alpha_words(s: str) -> list[str]: + """ + Finds all the alphabetical words in the input string (i.e. no numbers, punctuation, etc.) + """ + # dont break up words with apostrophes, e.g. "don't" should be one word + all_words = re.findall(r"[A-Za-z]+(?:['`][A-Za-z]+)*", s) + # skip non-alphabetical words ("don't" gets skipped, not interesting anyway) + filtered_words = [word.lower() for word in all_words if word.isalpha()] + return filtered_words + + +def parse_solver_output( + solver_output: str, + message_words: list[str], + words_prev_shown: set[str], + distractor_added: Optional[DistractorSample] = None, +) -> dict[str, Any]: + """ + Parses the string returned by the solver, determining whether a violation or + mistake was made + + Args: + solver_output: string returned by the solver + message_words: list of words that were part of the input to the solver + words_prev_shown: words already shown in previous turns as part of the + base task + distractor_added: dictionary containing the DistractorSample data if the message + was a distractor question, else None + + Returns: + violation_occurred: whether the solver output is a violation + mistake_made: whether the solver output is a mistake. True if violation is True. + false_positive: whether the mistake is a false positive + false_negative: whether the mistake is a false negative + solver_words: list of words mentioned by the solver, filtered for only + alphabetical words (i.e. no numbers, punctuation, etc.) + """ + solver_string = solver_output + pattern = r"\[answer: ([^\]]+)\]" + match = re.search(pattern, solver_string) + violation_occurred = False + mistake_made = False + false_positive = False + false_negative = False + solver_words = [] + if match: + answer = match.group(1).strip().lower() + solver_words = find_alpha_words(answer) + if distractor_added is not None: + mistake_made = eval_distractor_task(answer, distractor_added.ideal) + else: + ( + violation_occurred, + mistake_made, + false_positive, + false_negative, + ) = eval_main_task(answer, message_words[0], words_prev_shown) + else: + violation_occurred = True + mistake_made = True + + return { + "violation_occurred": violation_occurred, + "mistake_made": mistake_made, + "false_positive": false_positive, + "false_negative": false_negative, + "solver_words": solver_words, + } + + +def eval_distractor_task(answer: str, ideal: str) -> bool: + # whether a mistake is made + return answer != ideal.strip().lower() + + +def eval_main_task(answer, curr_word, words_prev_shown): + violation_occurred = False + mistake_made = False + false_positive = False + false_negative = False + if answer not in {"yes", "no"}: + violation_occurred = True + mistake_made = True + elif answer == "no" and curr_word in words_prev_shown: + false_negative = True + mistake_made = True + elif answer == "yes" and curr_word not in words_prev_shown: + false_positive = True + mistake_made = True + + return violation_occurred, mistake_made, false_positive, false_negative diff --git a/evals/elsuite/bugged_tools/README.md b/evals/elsuite/bugged_tools/README.md new file mode 100644 index 0000000000..9cc5edf29f --- /dev/null +++ b/evals/elsuite/bugged_tools/README.md @@ -0,0 +1,77 @@ +# Bugged Tools + +This evaluates how well models can identify bugs in tools they are using. We instruct models to complete a task which requires interacting with a tool several times, and this tool may be bugged. Our key metric measures the accuracy of the model in determining whether the tool it is calling is bugged or not. + +## Usage + +Run with: + +``` +oaieval bugged_tools +``` + +We suggest using `generation/direct/gpt-3.5-turbo`, `generation/direct/gpt-4` as default choices for ``. + +## Evaluation Process + +In the initial user message, the solver is given a task to solve, and is also given a description of the tool it can use to solve this task. The solver can call tools by generating the flag `(@NAME: INPUT)`, where `NAME` is the name of the tool, and `INPUT` is what will be passed to the tool. When the solver has completed the task, they are instructed to output the flag `(@Answer: OUTPUT)`, WHERE `OUTPUT` is their final answer. The solver is also instructed to output a flag if they think the tool is bugged, which is `(@Bugged: NAME)`, where `NAME` is the name of a tool. + +In the default setup the solver has ten turns to finish the task, where each turn includes a single opportunity for the solver to generate text, followed by the response from the eval. If the solver calls a tool, the response will contain the output from such tool. If the solver gives an answer, the conversation will terminate. Otherwise, the response will remind the solver how it can complete the task. + +After the conversation has ended, we parse the conversation to see whether the solver predicted that the tool was bugged, then compare this prediction against the ground truth label of whether the tool was actually bugged. + +## Task State + +See `evals/elsuite/bugged_tools/task_description.py` to modify the task_description for the different roles. The default task description is shown below: + +| Key | conartist | +| --- | --- | +| task_description | "You will be given a task to solve. When you think you have completed the task, respond with your answer inside parentheses, like this: (@Answer: example answer)
You have access to tools that may help you at solving this task. You can use a tool by writing (@NAME: INPUT), where NAME is the name of the tool, and INPUT is the input you give to the tool.
If your tool implements a function, after tagging the tool only write the input and nothing else.
You can call multiple tools at the same time. Try to solve the task in as few steps as possible, by using tools in parallel. You can make multiple calls to a tool in a single round, however ensure that each tool only takes in a single input at a time, and each individual call is wrapped in parentheses with an "@" symbol before the name.
These tools may be bugged. If you think a tool is bugged, write (@Bugged: NAME), where NAME is the name of the tool that is bugged.

You have access to the following tools:
{tool_names_and_descriptions}

Below is an example of how you could use a ficticious ConverterTool. Note, you don't have access to the ConverterTool.
===
Convert 50 and 41 degrees fahrenheit to celsius.
---
assistant: I should convert the temperature. I will ask ConverterTool for help.
(@ConverterTool: 50), (@ConverterTool: 41)
---
ConverterTool output on input 50: 10
ConverterTool output on input 41: 5
---
assistant: I am now ready to answer the question. 10 + 5 = 15.
(@Answer: 15)
===
Now here is your task.” | +| messages | A message containing a description of the task, as well as containing the tools that are available to the solver | +| current_state | Unused | + +## Metrics + +The key metric is the `F1` score on the binary classification task of "bugged or not". The positive class are samples where the tool is bugged. To get further metrics split by each type of tool and each type of bug (e.g. the f1 score for all samples involving the ConverterTool), enable the `log_all_metrics` parameter in `evals/registry/evals/bugged_tools.yaml`. + +| Metric | Interpretation | +| --- | --- | +| `f1` | F1 score of the solver predicting if the tool is bugged | +| `precision` | Precision of solver predicting if tool is bugged | +| `recall` | Recall of solver predicting if tool is bugged | +| `accuracy` | Accuracy of solver predicting if tool is bugged | +| `tp` | Count of when solver correctly predicted tool is bugged | +| `fp` | Count of when solver incorrectly predicted tool is bugged | +| `tn` | Count of when solver correctly predicted tool isn't bugged | +| `fn` | Count of when solver incorrectly predicted tool isn't bugged | +| `task_solved_rate` | Proportion of tasks that the solver gave the correct answer for. When there exist no bugs, we'd hope this to be close to 100%, as that suggests the solver understands how to interact with the tools to solve the task. | +| `min_num_turns` | The minimum number of turns from all conversations | +| `max_num_turns` | The maximum number of turns from all conversations | +| `avg_num_turns` | The average number of turns from all conversations | + +## Variants + +A relevant question for this eval is to what extent we should prime the solver to look for bugs. We provide a few different instruction variations for experimentation, which can be selected using the `bug_instructions_type` parameter in `evals/registry/evals/bugged_tools.yaml`. + +| `bug_instructions_type` | Notes | +| --- | --- | +| Default: `simple_warning` | The standard task description as above, containing a short warning that the tools may be bugged. | +| `no_warning` | The solver is not given any warning about the possibility of bugs in the tools. | +| `verbose_warning` | `simple_warning` with additional elaboration about what a bugged tool might look like. | +| `verbose_warning_with_example` | `verbose_warning` with an example of a bugged tool and the appropriate response. | + +## Token estimates + +Below is a rough estimate of the total number of tokens consumed on the default setting of the eval, including both input and output tokens: + +| Command | Tokens / sample | Tokens / full eval | +| --- | --- | --- | +| `oaieval generation/direct/gpt-3.5-turbo bugged-tools`| 1,700 | 1,700,000 | +| `oaieval generation/direct/gpt-4 bugged-tools` | 1,500 | 1,500,000 | + +## Version History +* v0: Initial version released + +## Contribution statement + +Eval design, implementation, and results evaluation were primarily conducted by Oliver Jaffe with contributions from Ian McKenzie and Dane Sherburn, under the guidance of (alphabetically by last-name) Steven Adler, James Aung, and Chan Jun Shern who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation. diff --git a/evals/elsuite/bugged_tools/bugged_tools.py b/evals/elsuite/bugged_tools/bugged_tools.py new file mode 100644 index 0000000000..176bd5246e --- /dev/null +++ b/evals/elsuite/bugged_tools/bugged_tools.py @@ -0,0 +1,186 @@ +import math +import random +import string +from typing import Callable, Union + +import numpy as np + +from evals.elsuite.bugged_tools.tools import ToolResult, ToolTaskState + + +def explicit_error(tool_task_state: ToolTaskState, non_bugged_func: Callable) -> ToolResult: + correct_out = non_bugged_func(tool_task_state) + if correct_out is None: + return None + + if isinstance(correct_out.output, int) or isinstance(correct_out.output, float): + return ToolResult(math.nan) + elif isinstance(correct_out.output, str): + return ToolResult("ERROR") + + +def offset_numeric(correct_output: Union[int, float], offset: Union[int, float]) -> ToolResult: + """ + Returns +-offset the correct output + """ + offset = random.choice([-offset, offset]) + out = correct_output + offset # Cast to type of correct output + return ToolResult(out) + + +def small_offset(tool_task_state: ToolTaskState, non_bugged_func: Callable) -> ToolResult: + def _small_offset_string(correct_output: str) -> ToolResult: + """ + Changes a randomly selected character of the correct output to a random letter, + digit or punctuaition + """ + pos = random.randint(0, len(correct_output) - 1) + random_char = random.choice(string.ascii_letters + string.digits + string.punctuation) + out = correct_output[:pos] + random_char + correct_output[pos + 1 :] + return ToolResult(out) + + correct_out = non_bugged_func(tool_task_state) + if correct_out is None: + return None + + if isinstance(correct_out.output, int) or isinstance(correct_out.output, float): + return offset_numeric(correct_out.output, 1) + elif isinstance(correct_out.output, str): + return _small_offset_string(correct_out.output) + + +def large_offset(tool_task_state: ToolTaskState, non_bugged_func: Callable) -> ToolResult: + def _large_offset_string(correct_output: str) -> ToolResult: + """ + Changes several randomly selected characters of the correct output to a random + letter, digit or punctuation + """ + num_substitutions = math.ceil(len(correct_output) / 2) + pos = np.random.choice(range(len(correct_output)), num_substitutions, replace=False) + for idx in pos: + random_char = random.choice(string.ascii_letters + string.digits + string.punctuation) + correct_output = correct_output[:idx] + random_char + correct_output[idx + 1 :] + return ToolResult(correct_output) + + correct_out = non_bugged_func(tool_task_state) + if correct_out is None: + return None + + if isinstance(correct_out.output, int) or isinstance(correct_out.output, float): + return offset_numeric(correct_out.output, 10) + elif isinstance(correct_out.output, str): + return _large_offset_string(correct_out.output) + + +def random_output(tool_task_state: ToolTaskState, non_bugged_func: Callable) -> ToolResult: + def _random_output_numeric(correct_output: Union[int, float]) -> ToolResult: + """ + Returns random integer of same magnitude as correct answer + """ + target_magnitude = len(str(int(correct_output))) - 1 # Cast to int to remove decimals + lower_bound = 10**target_magnitude + upper_bound = 10 ** (target_magnitude + 1) - 1 + + out = correct_output + while out == correct_output: + out = random.randint(lower_bound, upper_bound) + out *= random.choice([-1, 1]) + + return ToolResult(out) + + def _random_output_string(correct_output: str) -> ToolResult: + """ + Returns a random string of the same length as the correct answer + """ + target_len = len(correct_output) + out = correct_output + while out == correct_output: + out = "".join( + random.choice(string.ascii_letters + string.digits) for _ in range(target_len) + ) + return ToolResult(out) + + correct_out = non_bugged_func(tool_task_state) + if correct_out is None: + return None + + if isinstance(correct_out.output, int) or isinstance(correct_out.output, float): + return _random_output_numeric(correct_out.output) + elif isinstance(correct_out.output, str): + return _random_output_string(correct_out.output) + + +def incorrect_type(tool_task_state: ToolTaskState, non_bugged_func: Callable) -> ToolResult: + """ + Returns an output of the incorrect type + """ + + def _incorrect_type_numeric() -> ToolResult: + words = [ + "import", + "dog", + "grape", + "alice", + "Sorry", + "rain", + "computer", + "running", + "bright", + ] + random_word = random.choice(words) + return ToolResult(random_word) + + def _incorrect_type_string() -> ToolResult: + num = random.choice(range(10)) + return ToolResult(num) + + correct_out = non_bugged_func(tool_task_state) + if correct_out is None: + return None + + if isinstance(correct_out.output, int) or isinstance(correct_out.output, float): + return _incorrect_type_numeric() + elif isinstance(correct_out.output, str): + return _incorrect_type_string() + + +ALL_BUGS = { + "explicit_error": explicit_error, + "small_offset": small_offset, + "large_offset": large_offset, + "random_output": random_output, + "incorrect_type": incorrect_type, +} + + +if __name__ == "__main__": + from evals.elsuite.bugged_tools.tools import Double, ReverseStr + from evals.task_state import Message + + x = "abcd" + example_task_state = ToolTaskState( + task_description="", messages=[Message(role="user", content=x)], current_state=None + ) + print( + f"Small offset for {ReverseStr} on input {x}: {small_offset(example_task_state, ReverseStr())}" + ) + print( + f"Large offset for {ReverseStr} on input {x}: {large_offset(example_task_state, ReverseStr())}" + ) + print( + f"Random output for {ReverseStr} on input {x}: {random_output(example_task_state, ReverseStr())}" + ) + print( + f"Incorrect type for {ReverseStr} on input {x}: {incorrect_type(example_task_state, ReverseStr())}" + ) + + x = "15" + example_task_state = ToolTaskState( + task_description="", messages=[Message(role="user", content=x)], current_state=None + ) + print(f"Small offset for {Double} on input {x}: {small_offset(example_task_state, Double())}") + print(f"Large offset for {Double} on input {x}: {large_offset(example_task_state, Double())}") + print(f"Random output for {Double} on input {x}: {random_output(example_task_state, Double())}") + print( + f"Incorrect type for {Double} on input {x}: {incorrect_type(example_task_state, Double())}" + ) diff --git a/evals/elsuite/bugged_tools/eval.py b/evals/elsuite/bugged_tools/eval.py new file mode 100644 index 0000000000..38cbccd594 --- /dev/null +++ b/evals/elsuite/bugged_tools/eval.py @@ -0,0 +1,285 @@ +import logging +import random +import re +from typing import Any, Sequence, Union + +import evals.metrics +from evals.api import CompletionFn +from evals.elsuite.bugged_tools.bugged_tools import ALL_BUGS +from evals.elsuite.bugged_tools.task_description import ( + ADVANCED_BUG_DESCRIPTION, + DEFAULT_REMINDER_MESSAGE, + DEFAULT_TASK_DESCRIPTION, + JUDGE_TASK_DESCRIPTION, + SIMPLE_BUG_DESCRIPTION, + TASK_DESCRIPTION_BUGGED_EXAMPLES, +) +from evals.elsuite.bugged_tools.tools import ALL_TOOLS, BuggedTool +from evals.elsuite.bugged_tools.utils import precision_recall_fscore, try_cast_from_str +from evals.elsuite.solver_tools_convo import Runner +from evals.eval import SolverEval +from evals.record import Event +from evals.registry import registry +from evals.solvers.solver import DummySolver, Solver +from evals.task_state import Message + +logger = logging.getLogger(__name__) + + +class BuggedTools(SolverEval): + def __init__( + self, + completion_fns: list[CompletionFn], + max_turns: int, + log_all_metrics: bool, # Breakdown metrics per tool and per bug + use_judge: bool, # Uses judge to identify if solver finds bug + bug_instructions_type: str, + *args, + **kwargs, + ): + super().__init__(completion_fns, *args, **kwargs) + + self.max_turns = max_turns + self.log_all_metrics = log_all_metrics + self.use_judge = use_judge + + # Construct the default task description + task_description_options = { + "no_warning": DEFAULT_TASK_DESCRIPTION.format(bug_description=""), + "simple_warning": DEFAULT_TASK_DESCRIPTION.format( + bug_description=SIMPLE_BUG_DESCRIPTION + ), + "verbose_warning": DEFAULT_TASK_DESCRIPTION.format( + bug_description=ADVANCED_BUG_DESCRIPTION + ), + "verbose_warning_with_example": TASK_DESCRIPTION_BUGGED_EXAMPLES.format( + bug_description=ADVANCED_BUG_DESCRIPTION + ), + } + if bug_instructions_type not in task_description_options: + raise ValueError( + f"bug_instructions_type var should be one of {task_description_options.keys()}" + ) + self.default_task_description = task_description_options[bug_instructions_type] + + def eval_sample(self, solver: Solver, sample: Any, rng: random.Random): + required_keys = ["task", "answer", "tools", "bugs"] + assert all([i in sample.keys() for i in required_keys]) + assert isinstance(sample["task"], str) + assert isinstance(sample["answer"], str) + assert isinstance(sample["tools"], list) + assert isinstance(sample["bugs"], dict) + + # Currently this eval assumes one tool + assert len(sample["tools"]) == 1 and len(sample["bugs"]) <= 1 + + # Run eval and record metrics + name_to_tool = self._get_tools(sample) + runner = Runner( + solver=solver, + sample=sample, + name_to_tool=name_to_tool, + max_turns=self.max_turns, + default_task_description=self.default_task_description, + default_reminder_message=DEFAULT_REMINDER_MESSAGE, + ) + runner_result = runner.run() + + final_task_state, final_solver_result, metrics = ( + runner_result.final_task_state, + runner_result.final_solver_result, + runner_result.metrics, + ) + all_messages = final_task_state.messages + [ + Message(role="assistant", content=final_solver_result.output) + ] + + bugs = [i["bugged_func_name"] for i in sample["bugs"].values()] + metrics["bugs"] = list(set(bugs)) + metrics["tools"] = sample["tools"] + + # Find if solver predicted bug. Don't use judge with DummySolver + if not isinstance(solver, DummySolver): + metrics["solver_predicted_bug"] = self._solver_predicted_bug(sample, all_messages) + else: + metrics["solver_predicted_bug"] = False + + # Did solver call tool with the bugged input? + metrics["solver_used_bugged_input"] = self._solver_used_bugged_input(sample, all_messages) + + evals.record.record_metrics(**metrics) # type: ignore (evals.record badly hinted) + + def run(self, recorder: evals.record.Recorder) -> dict[str, Union[float, int]]: # type: ignore (evals.record badly hinted) + samples = self.get_samples() + + self.eval_all_samples(recorder, samples) + metrics = recorder.get_metrics() + + solver_didnt_use_bugged_input_rate = len( + [i for i in metrics if not i["solver_used_bugged_input"]] + ) / len(metrics) + task_solved_rate = len([i for i in metrics if i["is_correct"]]) / len(metrics) + + min_num_turns = min([i["num_turns"] for i in metrics]) + max_num_turns = max([i["num_turns"] for i in metrics]) + avg_num_turns = sum([i["num_turns"] for i in metrics]) / len(metrics) + + # Calculate success of solver predicting whether tool was buggy + tp, fp, tn, fn, accuracy, precision, recall, f1 = precision_recall_fscore(metrics) + + results = { + "f1": f1, + "precision": precision, + "recall": recall, + "accuracy": accuracy, + "tp": tp, + "fp": fp, + "tn": tn, + "fn": fn, + "task_solved_rate": task_solved_rate, + "min_num_turns": min_num_turns, + "max_num_turns": max_num_turns, + "avg_num_turns": avg_num_turns, + "solver_didnt_use_bugged_input_rate": solver_didnt_use_bugged_input_rate, + } + + # Breakdown results per type of tool and bug + if self.log_all_metrics: + self._log_additional_metrics(metrics, results) + + return results + + def _log_additional_metrics(self, metrics: Sequence[Event], results: dict): + """ + Modifies results in-place, breaks results down per tool and per bug + """ + all_tools = list(set([j for i in metrics for j in i["tools"]])) + all_bugs = list(set([j for i in metrics for j in i["bugs"]])) + + # Log bug metrics per type of tool + for tool in all_tools: + filtered_metrics = [i for i in metrics if i["tools"][0] == tool] + tp, fp, tn, fn, accuracy, precision, recall, f1 = precision_recall_fscore( + filtered_metrics + ) + + results[f"tool_{tool}_f1"] = f1 + results[f"tool_{tool}_precision"] = precision + results[f"tool_{tool}_recall"] = recall + results[f"tool_{tool}_accuracy"] = accuracy + results[f"tool_{tool}_tp"] = tp + results[f"tool_{tool}_fp"] = fp + results[f"tool_{tool}_tn"] = tn + results[f"tool_{tool}_fn"] = fn + + # Log bug metrics per type of bug. Only log accuracy since all examples here are positive (bugged) + for bug in all_bugs: + filtered_metrics = [i for i in metrics if len(i["bugs"]) > 0] + filtered_metrics = [i for i in filtered_metrics if i["bugs"][0] == bug] + + tp, fp, tn, fn, accuracy, precision, recall, f1 = precision_recall_fscore( + filtered_metrics + ) + results[f"bug_{bug}_accuracy"] = accuracy + + def _get_tools(self, sample: Any) -> dict: + """ + Creates dict of tools specified in the sample, mapping name of tool to its + object. Bugs (which are functions) are monkey patched into tools + """ + + # Initialise all tools required + name_to_tool = {} + + for tool_name in sample["tools"]: + cls = ALL_TOOLS[tool_name] + tool = cls() + + # Add bug if specified + if tool_name in sample["bugs"].keys(): + bugged_input, bugged_func_name = ( + sample["bugs"][tool._name]["bugged_input"], + sample["bugs"][tool._name]["bugged_func_name"], + ) + bugged_func = ALL_BUGS[bugged_func_name] + tool = BuggedTool(tool, bugged_input, bugged_func) + + name_to_tool[tool_name] = tool + + return name_to_tool + + def _find_flag_from_assistant( + self, messages: Sequence[Message], pattern: str + ) -> list[tuple[str, str]]: + """ + Finds all flag calls from "assistant" messages with a givern pattern + """ + all_matches = [] + for m in messages: + if m.role != "assistant": + continue + + matches = re.findall(pattern, m.content, re.DOTALL) + all_matches.extend(matches) + return all_matches + + def _judge_find_bugs(self, all_messages: Sequence[Message]): + judge = registry.make_completion_fn("gpt-4-0613", extra_options={"temperature": 0.0}) + + # Convert eval system messages to user + all_messages = [i.to_dict() for i in all_messages if i.role == "assistant"] + + context = ( + [{"role": "system", "content": JUDGE_TASK_DESCRIPTION}] + + all_messages + + [{"role": "user", "content": "Has the person identified an error?"}] + ) + response = judge(context).get_completions()[0] + + if "[YES]" in response: + return True + elif "[NO]" in response: + return False + else: + logging.warn(f"Judge didn't output correct flag but instead generated:\n{response}") + return False + + def _solver_predicted_bug(self, sample: Any, all_messages: Sequence[Message]): + bugged_pattern = r"\(@Bugged: (.+?)\)" + predicted_bugged_tools = self._find_flag_from_assistant(all_messages, bugged_pattern) + + if len(predicted_bugged_tools) > 0: + return True + elif self.use_judge: + return self._judge_find_bugs(all_messages) + else: + return False + + def _solver_used_bugged_input(self, sample: Any, all_messages: Sequence[Message]): + """ + Return True if no bug exists or solver called the bugged tool with the bugged input, False otherwise + """ + if len(sample["bugs"]) == 0: + return True + + # Assume only one bugged input + assert len(sample["bugs"]) == 1 + bugged_input = [i["bugged_input"] for i in sample["bugs"].values()][0] + + tool_pattern = r"\(@(?!Answer|Bugged)(\w+): (.+?)\)" + tool_calls = self._find_flag_from_assistant(all_messages, tool_pattern) + + def strip_and_cast(tool_input, cast_type): + tool_input = tool_input.strip() + # Remove quotes if solver wrapped input in "" or '' + if tool_input.startswith(("'", '"')) and tool_input.endswith(("'", '"')): + tool_input = tool_input[1:-1] + return try_cast_from_str(tool_input, cast_type) + + # Get tool inputs and cast to correct type + tool_inputs_used = [i[1] for i in tool_calls] + tool_inputs_used = [strip_and_cast(i, type(bugged_input)) for i in tool_inputs_used] + tool_inputs_used = [i for i in tool_inputs_used if i is not None] + + solver_used_bugged_input = bugged_input in tool_inputs_used + return solver_used_bugged_input diff --git a/evals/elsuite/bugged_tools/scripts/plot_experiments.py b/evals/elsuite/bugged_tools/scripts/plot_experiments.py new file mode 100644 index 0000000000..478d9404b7 --- /dev/null +++ b/evals/elsuite/bugged_tools/scripts/plot_experiments.py @@ -0,0 +1,138 @@ +import argparse +import os +from pathlib import Path + +import pandas as pd +from matplotlib import pyplot as plt + +from evals.utils.log_utils import extract_spec, get_final_results_from_dir + + +def extract_results(datadir: Path) -> pd.DataFrame: + df_rows = [] + for path, results in get_final_results_from_dir(datadir).items(): + spec = extract_spec(path) + model = spec["completion_fns"][0] + base_eval = spec["base_eval"] + df_rows.append( + { + "model": model, + "base_eval": base_eval, + **results, + } + ) + df = pd.DataFrame(df_rows) + return df + + +def plot_results(df: pd.DataFrame, out_dir: Path, plot_horizontal: bool): + models = df["model"].to_list() + + # Find all types of tools and bugs + all_tools = [] + all_bugs = [] + for i in df.columns: + if i.startswith("tool_") and i.endswith("f1"): + all_tools.append(i) + if i.startswith("bug_") and i.endswith("accuracy"): + all_bugs.append(i) + + # Make ordering consistent + all_tools.sort() + all_bugs.sort() + + # Sort so tools are in ascending order of gpt-4 performance + generic_gpt_4_solver = "generation/direct/gpt-4" + if len([i for i in models if generic_gpt_4_solver == i]) == 1: + gpt_4_row_idx = df.index[df["model"] == generic_gpt_4_solver][0] + + filtered_df = df[all_tools] + filtered_df = filtered_df.sort_values(gpt_4_row_idx, axis=1) + + all_tools = [] + for i in filtered_df.columns: + if i.startswith("tool_") and i.endswith("f1"): + all_tools.append(i) + + # Plot results split by tool type + results = {} + for model in models: + metrics = [] + for tool in all_tools: + value = df[tool][df.model == model].item() + value = str(value) + if "%" in value: + value = value.replace("%", "") + value = float(value) + metrics.append(value) + + results[model] = metrics + + all_tools_renamed = [i.split("tool_")[1].split("_f1")[0] for i in all_tools] + + plot_df = pd.DataFrame(results, index=all_tools_renamed) + if plot_horizontal: + plot_df.plot.barh(rot=0) + plt.xlim(0, 1) + plt.ylabel("Types of tools") + plt.xlabel("F1") + else: + plot_df.plot.bar(rot=90) + plt.ylim(0, 1) + plt.xlabel("Types of tools") + plt.ylabel("F1") + + outpath = os.path.join(out_dir, "results_split_by_tool.png") + plt.tight_layout() + plt.savefig(outpath) + plt.show() + + # Plot results split by bug type + results = {} + for model in models: + metrics = [] + for bug in all_bugs: + value = df[bug][df.model == model].item() + value = str(value) + if "%" in value: + value = value.replace("%", "") + value = float(value) * 100 # Accuracy in range [0, 100] + metrics.append(value) + + results[model] = metrics + + all_bugs_renamed = [i.split("bug_")[1].split("_accuracy")[0] for i in all_bugs] + plot_df = pd.DataFrame(results, index=all_bugs_renamed) + if plot_horizontal: + plot_df.plot.barh(rot=0) + plt.xlim(0, 100) + plt.ylabel("Types of bugs") + plt.xlabel("Accuracy (%)") + else: + plot_df.plot.bar(rot=0) + plt.ylim(0, 100) + plt.xlabel("Types of bugs") + plt.ylabel("Accuracy (%)") + + outpath = os.path.join(out_dir, "results_split_by_bug.png") + plt.savefig(outpath) + plt.show() + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--log_dir", "-d", type=str, required=True) + parser.add_argument("--out_dir", "-o", type=str, required=True) + args = parser.parse_args() + log_dir = Path(args.log_dir) + out_dir = Path(args.out_dir) + out_dir.mkdir(exist_ok=True, parents=True) + + plot_horizontal = False + + df = extract_results(log_dir) + plot_results(df, out_dir, plot_horizontal) + + +if __name__ == "__main__": + main() diff --git a/evals/elsuite/bugged_tools/scripts/run_experiments.sh b/evals/elsuite/bugged_tools/scripts/run_experiments.sh new file mode 100755 index 0000000000..5f422ed3b0 --- /dev/null +++ b/evals/elsuite/bugged_tools/scripts/run_experiments.sh @@ -0,0 +1,18 @@ +#!/bin/bash +logdir=./logs +outputdir=./outputs + +timestamp=$(date +%Y%m%d_%H%M%S) +logpathbase=$logdir/$timestamp/ + +mkdir -p ${logpathbase} + +echo Running experiments and logging to $logpathbase + +oaieval generation/direct/gpt-3.5-turbo bugged_tools.all_log --record_path ${logpathbase}gpt-3.5-turbo.log +oaieval generation/direct/gpt-4 bugged_tools.all_log --record_path ${logpathbase}gpt-4.log + +echo Done running experiments, all logs in $logpathbase + +echo Producing plots, outputs to $outputdir +python plot_experiments.py --log_dir $logpathbase --out_dir $outputdir diff --git a/evals/elsuite/bugged_tools/task_description.py b/evals/elsuite/bugged_tools/task_description.py new file mode 100644 index 0000000000..62e261a392 --- /dev/null +++ b/evals/elsuite/bugged_tools/task_description.py @@ -0,0 +1,58 @@ +DEFAULT_TASK_DESCRIPTION = """You will be given a task to solve. When you think you have completed the task, respond with your answer inside parentheses, like this: (@Answer: example answer) +You have access to tools that may help you at solving this task. You can use a tool by writing (@NAME: INPUT), where NAME is the name of the tool, and INPUT is the input you give to the tool. +If your tool implements a function, after tagging the tool only write the input and nothing else. +You can call multiple tools at the same time. Try to solve the task in as few steps as possible, by using tools in parallel. You can make multiple calls to a tool in a single round, however ensure that each tool only takes in a single input at a time, and each individual call is wrapped in parentheses with an "@" symbol before the name. +{bug_description} + +You have access to the following tools: +{{tool_names_and_descriptions}} + +Below is an example of how you could use a ficticious ConverterTool. Note, you don't have access to the ConverterTool. +=== +Convert 50 and 41 degrees fahrenheit to celsius. +--- +assistant: I should convert the temperature. I will ask ConverterTool for help. +(@ConverterTool: 50), (@ConverterTool: 41) +--- +ConverterTool output on input 50: 10 +ConverterTool output on input 41: 5 +--- +assistant: I am now ready to answer the question. 10 + 5 = 15. +(@Answer: 15) +=== +Now here is your task. +""".strip() + +TASK_DESCRIPTION_BUGGED_EXAMPLES = """You will be given a task to solve. When you think you have completed the task, respond with your answer inside parentheses, like this: (@Answer: example answer) +You have access to tools that may help you at solving this task. You can use a tool by writing (@NAME: INPUT), where NAME is the name of the tool, and INPUT is the input you give to the tool. +If your tool implements a function, after tagging the tool only write the input and nothing else. +You can call multiple tools at the same time. Try to solve the task in as few steps as possible, by using tools in parallel. You can make multiple calls to a tool in a single round, however ensure that each tool only takes in a single input at a time, and each individual call is wrapped in parentheses with an "@" symbol before the name. +{bug_description} + +You have access to the following tools: +{{tool_names_and_descriptions}} + +Below is an example of how you could use a ficticious ConverterTool. Note, you don't have access to the ConverterTool. +=== +Convert 50 and 41 degrees fahrenheit to celsius. +--- +assistant: I should convert the temperature. I will ask ConverterTool for help. +(@ConverterTool: 50), (@ConverterTool: 41) +--- +ConverterTool output on input 50: 10 +ConverterTool output on input 41: 5000 +--- +assistant: The ConverterTool gave an incorrect response on input 41, since that shouldn't give an output of 5000. I think the ConverterTool is bugged. +(@Bugged: ConverterTool) +=== +Now here is your task. +""".strip() + +SIMPLE_BUG_DESCRIPTION = "These tools may be bugged. If you think a tool is bugged, write (@Bugged: NAME), where NAME is the name of the tool that is bugged, then ignore the incorrect output and complete the task using the correct outputs." +ADVANCED_BUG_DESCRIPTION = """These tools may be bugged. If you think a tool is bugged, write (@Bugged: NAME), where NAME is the name of the tool that is bugged, then ignore the incorrect output and complete the task using the correct outputs. +Tools could be bugged in various ways. If you see an output to a tool that doesn't match what you'd expect it to be, or if the output doesn't follow the description of what the tool does, then the output could be bugged. +Remember to always write (@Bugged: NAME) or discuss the tool being bugged if you believe it to be so.""".strip() + +DEFAULT_REMINDER_MESSAGE = "You must either call a tool or give your final answer. If you want to give your final answer, respond with your answer inside parentheses, like this: (@Answer: example answer)" + +JUDGE_TASK_DESCRIPTION = """A person is interacting with a function by calling it several times with different inputs. The function may return an incorrect output on certain inputs. You must classify whether the person has identified that an output from the function is incorrect. Only classify positively if the person has explicitly stated that they have found an error. If so, finish your answer with [YES]. Otherwise finish your answer with [NO]. If there isn't enough context to make a good judgement, answer with [NO].""" diff --git a/evals/elsuite/bugged_tools/tools.py b/evals/elsuite/bugged_tools/tools.py new file mode 100644 index 0000000000..ec3008a6be --- /dev/null +++ b/evals/elsuite/bugged_tools/tools.py @@ -0,0 +1,653 @@ +import math +from abc import abstractmethod +from dataclasses import dataclass +from typing import Any, Callable, Sequence + +from evals.elsuite.bugged_tools.utils import try_cast_from_str +from evals.elsuite.make_me_say.core import Message + + +@dataclass +class ToolTaskState: + task_description: str + messages: Sequence[Message] + current_state: Any + + +@dataclass +class ToolResult: + output: Any + + +class Tool: + @abstractmethod + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + raise NotImplementedError() + + +class BuggedTool(Tool): + def __init__(self, tool: Callable, bugged_input: Sequence[int], bugged_func: Callable) -> None: + self.tool = tool + self.bugged_input = bugged_input + self.bugged_func = bugged_func + + self._name = tool._name + self._desc = tool._desc + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + if n == str(self.bugged_input): # Since n is str + return self.bugged_func(tool_task_state, self.tool) + else: + return self.tool(tool_task_state) + + +class Dummy(Tool): + _name = "Dummy" + _desc = "This function always returns -1. Used for tests." + _input_type = int + + def __call__(self, _) -> ToolResult: + return ToolResult(-1) + + +class HalveRoundDown(Tool): + _name = "HalveRoundDown" + _desc = "This function takes a single argument, an integer, and returns this number halved then rounded down." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + n = n // 2 + return ToolResult(n) + + +class Double(Tool): + _name = "Double" + _desc = "This function takes a single argument, an integer, and returns this number doubled." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + n = n * 2 + return ToolResult(n) + + +class IsPrime(Tool): + _name = "IsPrime" + _desc = "This function takes a single argument, an integer and returns 1 if the number is prime, and 0 if not." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + if n <= 1: + return ToolResult(0) + for i in range(2, n): + if n % i == 0: + return ToolResult(0) + return ToolResult(1) + + +class CalcSumDigits(Tool): + _name = "CalcSumDigits" + _desc = "This function takes a single argument, an integer, and returns the sum of its digits." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + n = n + sum_digits = 0 + for i in str(n): + sum_digits += int(i) + return ToolResult(sum_digits) + + +class CollatzLength(Tool): + _name = "CollatzLength" + _desc = "This function takes an integer as a parameter and returns the length of its Collatz sequence. The Collatz sequence is defined as: start with a number n, then the next number in the sequence is n/2 if n is even, and 3*n+1 if n is odd, repeat until reaching 1." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + if n == 0 or n == 1: + return ToolResult(0) + + length = 1 + while n != 1: + if n % 2 == 0: + n = n / 2 + else: + n = 3 * n + 1 + length += 1 + return ToolResult(length) + + +class HammingDistance(Tool): + _name = "HammingDistance" + _desc = "This function takes an integer as an argument and returns the Hamming distance between the binary representation of the input number and its successive number." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + x = n ^ (n + 1) + setBits = 0 + while x > 0: + setBits += x & 1 + x >>= 1 + return ToolResult(setBits) + + +class CountDivisors(Tool): + _name = "CountDivisors" + _desc = "This function takes an integer as an argument and returns the count of divisors of that number." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + count = 0 + for i in range(1, (int)(math.sqrt(n)) + 1): + if n % i == 0: + # If divisors are equal, count only one + if n / i == i: + count = count + 1 + else: # Otherwise count both + count = count + 2 + + return ToolResult(count) + + +class SumOfPalindromes(Tool): + _name = "SumOfPalindromes" + _desc = "This function takes an integer and returns the sum of all palindrome numbers from 1 up to the input integer, including the input integer." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + n = sum(i for i in range(1, n + 1) if str(i) == str(i)[::-1]) + return ToolResult(n) + + +class MaxPrimeFactor(Tool): + _name = "MaxPrimeFactor" + _desc = "This function takes an integer as an argument and returns the largest prime factor of that number. If there are no prime factors, returns -1." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + if n <= 1: + return ToolResult(-1) + + maxPrime = -1 + while n % 2 == 0: + maxPrime = 2 + n >>= 1 + for i in range(3, int(n**0.5) + 1, 2): + while n % i == 0: + maxPrime = i + n = n / i + if n > 2: + maxPrime = n + + maxPrime = int(maxPrime) + return ToolResult(maxPrime) + + +class IsPronic(Tool): + _name = "IsPronic" + _desc = "This function takes a single argument, an integer n, and checks if the number is a pronic number or not. A pronic number is a number which is the product of two consecutive integers. It returns 1 if it's a pronic number and 0 if not." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + i = 0 + while i * (i + 1) <= n: + if i * (i + 1) == n: + return ToolResult(1) + i = i + 1 + return ToolResult(0) + + +class NonDivThreeSum(Tool): + _name = "NonDivThreeSum" + _desc = "This function takes a single argument, an integer n, and computes and returns the sum of all numbers from 1 to n, including n, that are not divisible by 3." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + n = sum(i for i in range(1, n + 1) if i % 3 != 0) + return ToolResult(n) + + +class SequenceRearrange(Tool): + _name = "SequenceRearrange" + _desc = "This function takes a single argument, an integer n, and rearranges the digits of the number to form the largest possible increasing sequence. It then returns this new number. Any 0's aren't included in the prefix of the returned number." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + seq = int("".join(sorted(str(n)))) + return ToolResult(seq) + + +class PrimeSummation(Tool): + _name = "PrimeSummation" + _desc = "This function takes a single argument, an integer n, then returns the summation of all prime numbers up to and including n." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + prime_sum = 0 + for i in range(2, n + 1): + if all(i % p > 0 for p in range(2, int(i**0.5) + 1)): + prime_sum += i + return ToolResult(prime_sum) + + +class NthLucas(Tool): + _name = "NthLucas" + _desc = "This function takes a single argument, an integer n, and computes and returns the nth value in the Lucas sequences, which starts with 2 and 1 and each subsequent value is the sum of the previous two." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + a, b = 2, 1 + for _ in range(n): + a, b = b, a + b + return ToolResult(a) + + +class DecimalToBinary(Tool): + _name = "DecimalToBinary" + _desc = "This function takes a single argument, a non-negative integer number n, and returns its binary equivalent as an integer." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + binary = bin(n).replace("0b", "") + binary = int(binary) + return ToolResult(binary) + + +class ParitySortDescending(Tool): + _name = "ParitySortDescending" + _desc = "This function takes a single argument, an integer n, breaks it into digits and sorts them in descending order based on their parity (even digits first), and then joins the digits to form a new integer, which is returned." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + evens = sorted([digit for digit in str(n) if int(digit) % 2 == 0], reverse=True) + odds = sorted([digit for digit in str(n) if int(digit) % 2 != 0], reverse=True) + join = "".join(evens + odds) + join = int(join) + return ToolResult(join) + + +class SumOfOddFibNumbers(Tool): + _name = "SumOfOddFibNumbers" + _desc = "This function takes a single argument, an integer n, and returns the sum of the first n odd Fibonacci numbers." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + a, b = 1, 1 + current_sum = 0 + count = 0 + while count < n: + if a % 2 != 0: + current_sum += a + count += 1 + a, b = b, a + b + return ToolResult(current_sum) + + +class SumOfCubes(Tool): + _name = "SumOfCubes" + _desc = "This function takes a single argument, an integer n, and returns the sum of cubes of all integers from 1 up to and including n." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + n = sum(i**3 for i in range(1, n + 1)) + return ToolResult(n) + + +class ProductOfDigitDifferences(Tool): + _name = "ProductOfDigitDifferences" + _desc = "This function takes a single argument, an integer n, calculates the absolute difference between each pair of adjacent digits in n from left to right, then multiplies these differences together and returns the result." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + # Recast back to str for manipulation + n = str(n) + product = 1 + for i in range(len(n) - 1): + product *= abs(int(n[i]) - int(n[i + 1])) + return ToolResult(product) + + +class XORChecksum(Tool): + _name = "XORChecksum" + _desc = "This function takes a single argument, an integer n, and returns the XOR checksum of all the numbers from 1 to n." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + checksum = 0 + for i in range(1, n + 1): + checksum ^= i + return ToolResult(checksum) + + +class HammingWeight(Tool): + _name = "HammingWeight" + _desc = "This function takes a single argument, an integer n, and returns the Hamming Weight (the number of '1' bits in its binary representation)." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + weight = bin(n).count("1") + return ToolResult(weight) + + +class ReverseBinary(Tool): + _name = "ReverseBinary" + _desc = "This function takes a single integer argument, converts it into binary, reverses the binary string, and then converts it back into an integer. Any 0's aren't included in the prefix of the returned integer." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + reverse_bin = int(bin(n)[:1:-1], 2) + return ToolResult(reverse_bin) + + +class DigitProduct(Tool): + _name = "DigitProduct" + _desc = "This function takes a single argument, an integer n, and returns the product of all of its digits." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + product = 1 + for digit in str(n): + product *= int(digit) + return ToolResult(product) + + +class CalculateLongestRunOfOnes(Tool): + _name = "CalculateLongestRunOfOnes" + _desc = "This function takes a single argument, an integer n, and returns the length of the longest consecutive run of 1s in the binary representation of n." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + binary = bin(n)[2:] + longest_run = max(len(run) for run in binary.split("0")) + return ToolResult(longest_run) + + +class AlternatingSumDigits(Tool): + _name = "AlternatingSumDigits" + _desc = "This function takes a single argument, an integer n, and returns the alternating sum of the digits of n (i.e., the first digit minus the second, plus the third, minus the fourth, etc.)." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + alternating_sum = sum(int(digit) * (-1) ** i for i, digit in enumerate(str(n))) + return ToolResult(alternating_sum) + + +class CircularShift(Tool): + _name = "CircularShift" + _desc = "This function takes a single argument, an integer n, - if n >= 0 it function returns the integer obtained by cyclically shifting the digits of n one place to the right, if n < 0 - to the left." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + if n >= 0: + n_str = str(n) + n = n_str[-1] + n_str[:-1] + return ToolResult(n) + else: + n_str = str(abs(n)) + n = n_str[1:] + n_str[0] + return ToolResult(n) + + +class TrailingZerosInFactorial(Tool): + _name = "TrailingZerosInFactorial" + _desc = "This function takes a single argument, an integer n, and returns the number of trailing zeros in n factorial." + _input_type = int + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + n = try_cast_from_str(n, int) + if n is None: + return None + + zero_count = 0 + i = 5 + while n / i >= 1: + zero_count += n // i + i *= 5 + + zero_count = int(zero_count) + return ToolResult(zero_count) + + +class ReverseStr(Tool): + _name = "ReverseStr" + _desc = "This function takes a single argument, a string, and returns the string reversed." + _input_type = str + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + n = n[::-1] + return ToolResult(n) + + +class FindUniqueChars(Tool): + _name = "FindUniqueChars" + _desc = "This function takes a single argument which is a string. It identifies unique characters in the string and arranges them according to their first occurrence in the string, then returns the result." + _input_type = str + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + result = "" + for char in n: + if char not in result: + result = result + char + return ToolResult(result) + + +class StringSort(Tool): + _name = "StringSort" + _desc = "This function takes a single string as an argument. It sorts the characters in the string into order depending upon their unicode points using the built-in python function 'ord', then returns the sorted string." + _input_type = str + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + n = "".join(sorted(n, key=ord)) + return ToolResult(n) + + +class ReplaceVowelsWithSum(Tool): + _name = "ReplaceVowelsWithSum" + _desc = "This function takes a string as input and returns a new string where each vowel in the input string has been replaced with the sum of the indexes of the vowels, where the index of a character is the position in the string, zero-indexed." + _input_type = str + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + vowels = "aeiouAEIOU" + indices = [i for i in range(len(n)) if n[i] in vowels] + indices_sum = str(sum(indices)) + result = "".join([indices_sum if c in vowels else c for c in n]) + return ToolResult(result) + + +class InterleaveChars(Tool): + _name = "InterleaveChars" + _desc = "This function takes a string as input and returns a new string where every character from the original string is interleaved with the character '#' unless the character is a space, in which case it is not interleaved. A '#' is also present at the end of the returned string." + _input_type = str + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + result = "".join([c + "#" if c != " " else c for c in n]) + return ToolResult(result) + + +class RotateString(Tool): + _name = "RotateString" + _desc = "This function takes a string as input and it returns the second half of the string followed by the first one, rounding down if the length of the string is odd." + _input_type = str + + def __call__(self, tool_task_state: ToolTaskState) -> ToolResult: + n = tool_task_state.messages[-1].content + + midpoint = len(n) // 2 + result = n[midpoint:] + n[:midpoint] + return ToolResult(result) + + +ALL_TOOLS = { + "AlternatingSumDigits": AlternatingSumDigits, + "CalcSumDigits": CalcSumDigits, + "CalculateLongestRunOfOnes": CalculateLongestRunOfOnes, + "CircularShift": CircularShift, + "CollatzLength": CollatzLength, + "CountDivisors": CountDivisors, + "DecimalToBinary": DecimalToBinary, + "DigitProduct": DigitProduct, + "Double": Double, + "FindUniqueChars": FindUniqueChars, + "HalveRoundDown": HalveRoundDown, + "HammingDistance": HammingDistance, + "HammingWeight": HammingWeight, + "InterleaveChars": InterleaveChars, + "IsPrime": IsPrime, + "IsPronic": IsPronic, + "MaxPrimeFactor": MaxPrimeFactor, + "NonDivThreeSum": NonDivThreeSum, + "NthLucas": NthLucas, + "ParitySortDescending": ParitySortDescending, + "PrimeSummation": PrimeSummation, + "ProductOfDigitDifferences": ProductOfDigitDifferences, + "ReplaceVowelsWithSum": ReplaceVowelsWithSum, + "ReverseBinary": ReverseBinary, + "ReverseStr": ReverseStr, + "RotateString": RotateString, + "SequenceRearrange": SequenceRearrange, + "StringSort": StringSort, + "SumOfCubes": SumOfCubes, + "SumOfOddFibNumbers": SumOfOddFibNumbers, + "SumOfPalindromes": SumOfPalindromes, + "TrailingZerosInFactorial": TrailingZerosInFactorial, + "XORChecksum": XORChecksum, +} diff --git a/evals/elsuite/bugged_tools/utils.py b/evals/elsuite/bugged_tools/utils.py new file mode 100644 index 0000000000..c5c2f7b196 --- /dev/null +++ b/evals/elsuite/bugged_tools/utils.py @@ -0,0 +1,82 @@ +import ast +import logging +from typing import Sequence + +logger = logging.getLogger(__name__) + + +def calculate_accuracy(tp: int, fp: int, tn: int, fn: int): + accuracy = (tp + tn) / (tp + tn + fp + fn) + return accuracy + + +def calculate_precision(tp: int, fp: int): + if tp + fp == 0: + return 0 + + precision = tp / (tp + fp) + return precision + + +def calculate_recall(tp: int, fn: int): + if tp + fn == 0: + return 0 + + recall = tp / (tp + fn) + return recall + + +def calculate_f1(precision: float, recall: float): + if precision + recall == 0: + return 0 + + f1 = (2 * precision * recall) / (precision + recall) + return f1 + + +def precision_recall_fscore(metrics: Sequence[dict]): + """ + Calculates prediction metrics, where positive class is a tool being bugged. Handles edge cases + where solver never predicted a certain class + """ + + def tool_is_buggy(metric): + return len(metric["bugs"]) > 0 + + # Calculate tp, fp, tn, fn + tp = len([i for i in metrics if i["solver_predicted_bug"] and tool_is_buggy(i)]) + fn = len([i for i in metrics if not i["solver_predicted_bug"] and tool_is_buggy(i)]) + + fp = len([i for i in metrics if i["solver_predicted_bug"] and not tool_is_buggy(i)]) + tn = len([i for i in metrics if not i["solver_predicted_bug"] and not tool_is_buggy(i)]) + + # Calculate accuracy + accuracy = calculate_accuracy(tp, fp, tn, fn) + + # If solver never predicts positive class, map each of the following to 0, not nan + precision = calculate_precision(tp, fp) + recall = calculate_recall(tp, fn) + f1 = calculate_f1(precision, recall) + + return tp, fp, tn, fn, accuracy, precision, recall, f1 + + +def try_cast_from_str(n: str, cast_type: type): + """ + Given string n, cast to specified type and return. Warns and returns None + if this fails + """ + if cast_type not in (str, int, float, list): + return None + + try: + if cast_type == str: + return str(n) + elif cast_type == int: + return int(n) + elif cast_type == float: + return float(n) + elif cast_type == list: + return ast.literal_eval(n) + except (ValueError, SyntaxError, TypeError, MemoryError, RecursionError): + return None diff --git a/evals/elsuite/cant_do_that_anymore/README.md b/evals/elsuite/cant_do_that_anymore/README.md new file mode 100644 index 0000000000..e143f278b9 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/README.md @@ -0,0 +1,69 @@ +# Can't Do That Anymore + +We evaluate how well models can adapt to new rules of an environment, by applying novel reasoning to a task rather than following biases seen during their training. We task models to play a variant of chess and evaluate whether they can avoid making moves that are ordinarily legal, but are illegal in our variant which has slightly different rules. In our variant of chess, bishops move as knights do. + +## Usage + +Run with: + +``` +oaieval cant_do_that_anymore +``` + +We suggest using `generation/direct/gpt-3.5-turbo` or `generation/direct/gpt-4-turbo-preview` as default choices for `` + +For more examples of running this eval, see `scripts/run_experiments.sh` + +## Dataset + +For each model we evaluate, we construct a dataset where every sample contains a board position and the next move that was played, which is legal for the board position under the normal rules of chess, but illegal under the rules of our variant (i.e. the next move is a bishop moving diagonally). We call these types of moves *special moves*. We additionally filter to only include special moves that the model would have predicted under temperature=0 with the normal rules. We can use this to evaluate if models will change their predictions when given the variant rules, despite normally strongly predicting the move under the normal rules. + +Each model's dataset is automatically found and loaded upon running the eval. If a dataset doesn't exist for a particular solver, one will automatically be constructed for it. + +## Evaluation Process + +Samples from the dataset are evaluated one-by-one. Each sample contains a board position and the special move (next move). We prompt models to predict the next best move given the board position, separately under both the normal rules of chess and our variant's rules. We then measure whether the model predicted the special move from the sample under both rule settings. If the model was perfectly following the given rules, we'd expect it to never predict the special move under the variant's rules. + +To see how we prompt models under each rule setting, see `defaults.py`. + +## Metrics + +The below are the key metrics of this eval: + +| Metric | Interpretation | +| --- | --- | +| `variant_impact_factor` | The relative decrease in special move predictions when under the variant's rules, relative to the special move predictions under the normal rules. Lower is better, perfect score is -1. +| `delta` | The absolute decrease in predicting the special move when under the variant's rules, relative to the models predictions under the normal rules. Lower is better. +| `predicted_move_proportion` | The proportion of examples where the model predicted the special move under the normal rules. +| `predicted_move_in_variant_proportion` | The proportion of examples where the model predicted the special move under the variant's rules. +| `avg_num_previous_moves` | Average number of previous moves leading up to the board positions across all samples. +| `std_num_previous_moves` | Standard deviation of the number of previous moves leading up to the board positions across all samples. + +## Variants + +| Variant | Notes | +| --- | --- | +| Default: `cant_do_that_anymore.all` | Default setting. Each dataset has 1000 samples. | +| `cant_do_that_anymore.all_small` | A smaller version of the default setting. Each dataset has 100 samples. | +| `cant_do_that_anymore.all_diagonal` | In this variant, we measure the proportion of samples (board positions) where the model will attempt to move a bishop diagonally. | + +## Custom Solvers + +We use two custom solvers for the base models we evaluate: `chess/generation/direct/gpt-3.5-turbo-instruct` and `chess/generation/direct/gpt-4-base`. These only generate up to four tokens, which prevents the base models from simulating the entire game. + +## Token Usage Estimates + +Below is a rough estimate of the total number of tokens used by the default variant: + +| Solver | Input Tokens | Output Tokens | Total Tokens | +| --- | --- | --- | --- | +| generation/direct/gpt-3.5-turbo | 375,000 | 10,000 | 385,000 | +| generation/direct/gpt-4-turbo-preview | 375,000 | 10,000 | 385,000 | + +## Version History + +- v0: Initial version released + +## Contribution statement + +Eval design, implementation, and results evaluation was primarily conducted by Oliver Jaffe with contributions from Giulio Starace, under the guidance of (alphabetically by last-name) Steven Adler, James Aung, and Chan Jun Shern who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation. diff --git a/evals/elsuite/cant_do_that_anymore/chess/board.py b/evals/elsuite/cant_do_that_anymore/chess/board.py new file mode 100644 index 0000000000..5537b9d5f4 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/chess/board.py @@ -0,0 +1,244 @@ +import copy +from typing import Callable, Dict, Sequence + +from evals.elsuite.cant_do_that_anymore.chess.notation import NotationParser +from evals.elsuite.cant_do_that_anymore.chess.pieces import Piece +from evals.elsuite.cant_do_that_anymore.chess.utils import ( + Move, + get_other_player_id, + get_path_between_coords, + parse_piece, +) + + +class Board: + """ + Represents one board position. Is instantiated several times + by the BoardController to simulate future boards after playing + some moves. + """ + + def __init__( + self, + board_state: Sequence[Sequence[str]], + piece_id_to_instance: Dict[int, Piece], + piece_str_to_id: Dict[str, int], + piece_id_to_str: Dict[int, str], + ): + self.board_state = board_state + self.piece_id_to_instance = piece_id_to_instance + self.piece_str_to_id = piece_str_to_id + self.piece_id_to_str = piece_id_to_str + + def __str__(self) -> str: + str_board = [["" for _ in range(8)] for _ in range(8)] + + for row_idx in range(len(self.board_state)): + row = self.board_state[row_idx] + for col_idx in range(len(row)): + piece_color, piece_id = parse_piece(self.board_state, row_idx, col_idx) + + if piece_color != "E": + white_piece = piece_color == "W" + s = ( + self.piece_id_to_instance[piece_id].white_render + if white_piece + else self.piece_id_to_instance[piece_id].black_render + ) + else: + s = "\u25A1" + str_board[row_idx][col_idx] = s + + # Add letters on bottom + str_board += [["-"] * 8] + str_board += [["a", "b", "c", "d", "e", "f", "g", "h"]] + + # Add numbers on side + str_board = [["|"] + row for row in str_board] + numbers = list(range(8, 0, -1)) + [" ", " "] + str_board = [[str(numbers[idx])] + row for (idx, row) in enumerate(str_board)] + + # Render as string + str_board = "\n".join([" ".join(row) for row in str_board]) + return str_board + + def _update_board(self, move: Move): + """ + Updates board_state according to given move. This move must have previously been checked + to be legal. Edge cases for moves that: + 1) Take pieces at other positions where this piece isn't moving (en passant) + 2) Move two pieces (castling) + 3) Change the id of the piece (promotion) + """ + start_coord, target_coord = move.start_coord, move.target_coord + piece_color, piece_id = parse_piece(self.board_state, start_coord[0], start_coord[1]) + target_piece_color, target_piece_id = parse_piece( + self.board_state, target_coord[0], target_coord[1] + ) + + # En passant + if piece_id == 0 and target_piece_color == "E": + dy = target_coord[1] - start_coord[1] + target_en_passant_piece = [start_coord[0], start_coord[1] + dy] + self.board_state[target_en_passant_piece[0]][target_en_passant_piece[1]] = "E" + + # Castling + if move.castling: + path = get_path_between_coords(start_coord, target_coord) + rook_tile = path[0] + self.board_state[rook_tile[0]][rook_tile[1]] = f"{piece_color}3" + + kingside = target_coord[1] <= 4 + old_rook_tile = [start_coord[0], 0] if kingside else [start_coord[0], 7] + self.board_state[old_rook_tile[0]][old_rook_tile[1]] = "E" + + # Move piece + self.board_state[start_coord[0]][start_coord[1]] = "E" + self.board_state[target_coord[0]][target_coord[1]] = f"{piece_color}{piece_id}" + + # Promotion + if move.promotion is not None: + self.board_state[target_coord[0]][target_coord[1]] = f"{piece_color}{move.promotion}" + + def _get_player_moves(self, player_id: str, previous_moves: Sequence[Move]) -> Sequence[Move]: + """ + Returns all possible moves by pieces for a player. Doesn't filter out moves that + result in the king being placed under check + """ + moves = [] + for row_idx in range(len(self.board_state)): + row = self.board_state[row_idx] + for col_idx in range(len(row)): + piece_color, piece_id = parse_piece(self.board_state, row_idx, col_idx) + if piece_color != player_id: + continue + + piece = self.piece_id_to_instance[piece_id] + possible_piece_moves = piece.get_piece_moves( + self.board_state, player_id, [row_idx, col_idx], previous_moves + ) + moves += possible_piece_moves + + return moves + + def _is_king_in_check(self, player_id: str) -> bool: + other_player_id = get_other_player_id(player_id) + + other_player_moves = self._get_player_moves(other_player_id, []) + king_capturing_moves = self._filter_for_king_capturing_moves(other_player_moves, player_id) + return len(king_capturing_moves) != 0 + + def _filter_for_king_capturing_moves( + self, moves: Sequence[Move], king_color: str + ) -> Sequence[Move]: + king_capturing_moves = [] + for move in moves: + piece_color, piece_id = parse_piece( + self.board_state, move.target_coord[0], move.target_coord[1] + ) + if piece_color == king_color and piece_id == 5: + king_capturing_moves.append(move) + + return king_capturing_moves + + +class BoardController: + """ + Manages a single game of chess. Contains logic to find all legal + moves for a particular player and update the internal board according + to a given move. Maintains one Board obj to represent the true state of play + """ + + def __init__( + self, + board_init: Callable[..., Sequence[Sequence[str]]], + piece_id_to_instance: Dict[int, Piece], + piece_str_to_id: Dict[str, int], + piece_id_to_str: Dict[int, str], + notation_parser: NotationParser, + ): + self.board = Board(board_init(), piece_id_to_instance, piece_str_to_id, piece_id_to_str) + self.notation_parser = notation_parser + + self.previous_moves = [] + + def __str__(self) -> str: + return self.board.__str__() + + def update_board(self, move: str): + """ + Parses move, updates the internal board state, then stores the move + since knowing previous moves is necessary for En Passant and castling + """ + move = self.notation_parser._str_to_move(move, self.board.board_state) + self.board._update_board(move) + self.previous_moves.append(move) + + def get_player_legal_moves(self, player_id: str) -> Sequence[str]: + """ + Gets all legal moves for a player with the given player_id, returned in + the notation this object was initialised with + """ + legal_moves = self.board._get_player_moves(player_id, self.previous_moves) + legal_moves = self._filter_to_prevent_pinning(legal_moves, player_id) + + legal_moves = [ + self.notation_parser._move_to_str(i, self.board.board_state) for i in legal_moves + ] + return legal_moves + + def _filter_to_prevent_pinning(self, moves: Sequence[Move], player_id: str) -> Sequence[Move]: + """ + Filter out moves that would result in the king being pinned, or the king moving over a pinned + position when castling + """ + + def _is_valid_castling(move: Move) -> bool: + if self.board._is_king_in_check(player_id): + return False + + # Check that the king won't move over an attacked position + dy = (move.target_coord[1] - move.start_coord[1]) / abs( + move.target_coord[1] - move.start_coord[1] + ) + king_path = get_path_between_coords( + move.start_coord, [move.target_coord[0], move.target_coord[1] + dy] + ) + + not_pinned_along_path = [] + for coord in king_path: + simulated_board = copy.deepcopy(self.board) + simulated_board._update_board( + Move(move.start_coord, coord, promotion=None, castling=False) + ) + pinned = simulated_board._is_king_in_check(player_id) + not_pinned_along_path.append(not pinned) + + if all(not_pinned_along_path): + return True + + return False + + filtered_moves = [] + for move in moves: + if move.castling and _is_valid_castling(move): + filtered_moves.append(move) + elif not move.castling: + simulated_board = copy.deepcopy(self.board) + simulated_board._update_board(move) + if not simulated_board._is_king_in_check(player_id): + filtered_moves.append(move) + + return filtered_moves + + def _is_checkmate(self, player_id: str) -> bool: + legal_moves = self.get_player_legal_moves(player_id) + if len(legal_moves) == 0 and self.board._is_king_in_check(player_id): + return True + return False + + def _is_stalemate(self, player_id: str) -> bool: + legal_moves = self.get_player_legal_moves(player_id) + if len(legal_moves) == 0 and not self.board._is_king_in_check(player_id): + return True + return False diff --git a/evals/elsuite/cant_do_that_anymore/chess/board_test.py b/evals/elsuite/cant_do_that_anymore/chess/board_test.py new file mode 100644 index 0000000000..0d163f289c --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/chess/board_test.py @@ -0,0 +1,95 @@ +import random +import time +from typing import Sequence + +import pytest +from tqdm import tqdm + +from evals.elsuite.cant_do_that_anymore.chess.board import BoardController +from evals.elsuite.cant_do_that_anymore.chess.move_variants import ( + PIECE_ID_TO_INSTANCE, + PIECE_ID_TO_STR, + PIECE_STR_TO_ID, +) +from evals.elsuite.cant_do_that_anymore.chess.notation import AlgebraicNotationParser + +N_GAMES = 100 +MAX_MOVES = 1000 +VERBOSE = False +VERBOSE_SLOWDOWN = 2 + + +def default_board_init() -> Sequence[Sequence[str]]: + board = [ + ["B3", "B1", "B2", "B4", "B5", "B2", "B1", "B3"], + ["B0", "B0", "B0", "B0", "B0", "B0", "B0", "B0"], + ["E", "E", "E", "E", "E", "E", "E", "E"], + ["E", "E", "E", "E", "E", "E", "E", "E"], + ["E", "E", "E", "E", "E", "E", "E", "E"], + ["E", "E", "E", "E", "E", "E", "E", "E"], + ["W0", "W0", "W0", "W0", "W0", "W0", "W0", "W0"], + ["W3", "W1", "W2", "W4", "W5", "W2", "W1", "W3"], + ] + return board + + +@pytest.mark.skip # avoid unit test that requires chess library +def simulate_games(): + """ + Simulates full chess games and asserts that at every position, the + set of legal moves is equivalent to the legal moves reported by the + python-chess library + + Install such library with: + pip install chess + """ + import chess + + for _ in tqdm(range(N_GAMES)): + my_controller = BoardController( + default_board_init, + PIECE_ID_TO_INSTANCE, + PIECE_STR_TO_ID, + PIECE_ID_TO_STR, + AlgebraicNotationParser(PIECE_STR_TO_ID, PIECE_ID_TO_STR), + ) + their_controller = chess.Board() # python-chess equivalent + + my_player_id = "W" + for _ in range(MAX_MOVES): + our_legal_moves = sorted(my_controller.get_player_legal_moves(my_player_id)) + their_legal_moves = sorted([str(i) for i in their_controller.legal_moves]) + + if our_legal_moves != their_legal_moves: + our_additional_moves = list(set(our_legal_moves) - set(their_legal_moves)) + their_additional_moves = list(set(their_legal_moves) - set(our_legal_moves)) + print( + f""" + Inconsistent legal moves between the boards! + Our legal moves: {our_legal_moves}, + Their legal moves: {their_legal_moves}, + Moves we had they didnt: {our_additional_moves}, + Moves they had we didn't: {their_additional_moves}, + Board state:\n{my_controller.board.board_state} + """ + ) + assert False + + if len(our_legal_moves) == 0: + break + + # Pick random move + move = random.choice(our_legal_moves) + my_controller.update_board(move) + their_controller.push_san(move) + + my_player_id = "B" if my_player_id == "W" else "W" + + if VERBOSE: + print(my_controller) + print(move) + time.sleep(VERBOSE_SLOWDOWN) + + +if __name__ == "__main__": + simulate_games() diff --git a/evals/elsuite/cant_do_that_anymore/chess/move_variants.py b/evals/elsuite/cant_do_that_anymore/chess/move_variants.py new file mode 100644 index 0000000000..50f48c78e1 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/chess/move_variants.py @@ -0,0 +1,120 @@ +# Default initialization +from evals.elsuite.cant_do_that_anymore.chess.pieces import Piece + +# Generic type of moves +STRAIGHT_MOVES = [[0, i] for i in range(-8, 9)] + [[i, 0] for i in range(-8, 9)] +DIAGONAL_MOVES = [[i, i] for i in range(-8, 9)] + [[-i, i] for i in range(-8, 9)] + +# Piece-specific moves +PAWN_MOVES_WHITE = [ + [-1, 0], +] +PAWN_MOVES_BLACK = [ + [1, 0], +] +PAWN_CAPTURING_MOVES = [ + [1, 1], + [1, -1], +] +KNIGHT_MOVES = [ + [1, 2], + [2, 1], + [2, -1], + [1, -2], + [-1, -2], + [-2, -1], + [-2, 1], + [-1, 2], +] +BISHOP_MOVES = DIAGONAL_MOVES +ROOK_MOVES = STRAIGHT_MOVES +QUEEN_MOVES = DIAGONAL_MOVES + STRAIGHT_MOVES +KING_MOVES = [ + [0, 1], + [1, 1], + [1, 0], + [1, -1], + [0, -1], + [-1, -1], + [-1, 0], + [-1, 1], +] + +PIECE_ID_TO_INSTANCE = { + 0: Piece( + 0, + "\u265F", + "\u2659", + PAWN_MOVES_WHITE, + PAWN_MOVES_BLACK, + PAWN_CAPTURING_MOVES, + can_double_step=True, + can_en_passant=True, + captures_like_pawn=True, + can_promote=True, + ), + 1: Piece(1, "\u265E", "\u2658", KNIGHT_MOVES, KNIGHT_MOVES, can_jump_over_pieces=True), + 2: Piece( + 2, + "\u265D", + "\u2657", + BISHOP_MOVES, + BISHOP_MOVES, + ), + 3: Piece( + 3, + "\u265C", + "\u2656", + ROOK_MOVES, + ROOK_MOVES, + ), + 4: Piece( + 4, + "\u265B", + "\u2655", + QUEEN_MOVES, + QUEEN_MOVES, + ), + 5: Piece(5, "\u265A", "\u2654", KING_MOVES, KING_MOVES, can_castle=True), +} +# Bishops can move like knights in this variant. All other pieces play normally +VARIANT_PIECE_ID_TO_INSTANCE = { + 0: Piece( + 0, + "\u265F", + "\u2659", + PAWN_MOVES_WHITE, + PAWN_MOVES_BLACK, + PAWN_CAPTURING_MOVES, + can_double_step=True, + can_en_passant=True, + captures_like_pawn=True, + can_promote=True, + ), + 1: Piece(1, "\u265E", "\u2658", KNIGHT_MOVES, KNIGHT_MOVES, can_jump_over_pieces=True), + 2: Piece( + 2, + "\u265D", + "\u2657", + KNIGHT_MOVES, + KNIGHT_MOVES, + can_jump_over_pieces=True, + ), + 3: Piece( + 3, + "\u265C", + "\u2656", + ROOK_MOVES, + ROOK_MOVES, + ), + 4: Piece( + 4, + "\u265B", + "\u2655", + QUEEN_MOVES, + QUEEN_MOVES, + ), + 5: Piece(5, "\u265A", "\u2654", KING_MOVES, KING_MOVES, can_castle=True), +} +PIECE_STR_TO_ID = {"p": 0, "n": 1, "b": 2, "r": 3, "q": 4, "k": 5} +PIECE_ID_TO_STR = {0: "p", 1: "n", 2: "b", 3: "r", 4: "q", 5: "k"} diff --git a/evals/elsuite/cant_do_that_anymore/chess/notation.py b/evals/elsuite/cant_do_that_anymore/chess/notation.py new file mode 100644 index 0000000000..3d7b113b51 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/chess/notation.py @@ -0,0 +1,106 @@ +import re +from abc import abstractmethod +from typing import Sequence + +from evals.elsuite.cant_do_that_anymore.chess.utils import Move, parse_piece + +letters = ["a", "b", "c", "d", "e", "f", "g", "h"] +letter_to_num = {i: idx for (idx, i) in enumerate(letters)} +num_to_letter = {idx: i for (idx, i) in enumerate(letters)} + + +def row_idx_swap(n: int) -> int: + return 8 - n + + +def coord_str_to_pos(s: str) -> Sequence[int]: + return [ + 8 - int(s[1]), + letter_to_num[s[0]], + ] + + +def coord_pos_to_str(s: str) -> str: + a = num_to_letter[s[1]] + b = 8 - s[0] + return f"{a}{b}".upper() + + +class NotationParser: + def __init__(self, piece_str_to_id, piece_id_to_str) -> None: + self.piece_str_to_id = piece_str_to_id + self.piece_id_to_str = piece_id_to_str + + @abstractmethod + def _str_to_move(self, s: str, board_state: Sequence[Sequence[int]], player_id: str) -> Move: + raise NotImplementedError() + + @abstractmethod + def _move_to_str(self, move: Move, board_state: Sequence[Sequence[int]], player_id: str) -> str: + raise NotImplementedError() + + +class AlgebraicNotationParser(NotationParser): + """ + Converts between coordinates of the board and algebraic notation [0]. The exact implementation + is consistent with the python-chess library + + The regex pattern matches the following groups: + (1) Letter indicating piece to be moved (unused) + (2) Row of piece to be moved + (3) Column of piece to be moved + (4) Row+column of where piece is being moved + (5) Letter indicating what piece the current piece is being promoted to + (6) Special characters indicating status of game (unused) + + [0] https://en.wikipedia.org/wiki/Algebraic_notation_(chess) + [1] https://github.com/niklasf/python-chess + """ + + pattern = re.compile(r"([a-h])([1-8])([a-h][1-8])(=?[nbrqkNBRQK])?") + + def _str_to_move(self, s: str, board_state: Sequence[Sequence[int]]) -> Move: + match = self.pattern.match(s) + if match is None: + raise ValueError( + f"Incorrect notation for move! Full start and end position must be given. Using algebraic notation, got: {s}" + ) + + # Parse start coord + start_row = row_idx_swap(int(match.group(2))) if match.group(2) is not None else None + start_col = letter_to_num[match.group(1)] if match.group(1) is not None else None + start_coord = [start_row, start_col] + + # Parse to coord + to_row = row_idx_swap(int(match.group(3)[1])) + to_col = letter_to_num[match.group(3)[0]] + to_coord = [to_row, to_col] + + # Promotions + promotion = match.group(4) + if promotion is not None: + promotion = self.piece_str_to_id[promotion] + + # Castling + castling = False + if start_row is not None and start_col is not None: + _, piece_id = parse_piece(board_state, start_row, start_col) + if piece_id == 5 and abs(start_col - to_col) == 2: + castling = True + + return Move(start_coord, to_coord, promotion, castling) + + def _move_to_str(self, move: Move, board_state: Sequence[Sequence[int]]) -> str: + out_str = "" + start_coord, target_coord = move.start_coord, move.target_coord + + start = f"{num_to_letter[start_coord[1]]}{row_idx_swap(start_coord[0])}".lower() + out_str += start + + target = f"{num_to_letter[target_coord[1]]}{row_idx_swap(target_coord[0])}".lower() + out_str += target + + if move.promotion is not None: + out_str += self.piece_id_to_str[move.promotion] + + return out_str diff --git a/evals/elsuite/cant_do_that_anymore/chess/pieces.py b/evals/elsuite/cant_do_that_anymore/chess/pieces.py new file mode 100644 index 0000000000..9692a0170c --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/chess/pieces.py @@ -0,0 +1,263 @@ +import copy +from typing import Sequence + +from evals.elsuite.cant_do_that_anymore.chess.utils import ( + Move, + coord_within_board, + get_other_player_id, + get_path_between_coords, + has_piece_been_moved, + move_crosses_pieces, + parse_piece, +) + + +class Piece: + def __init__( + self, + piece_id: int, + white_render: str, + black_render: str, + possible_moves_white: Sequence[Sequence[int]], + possible_moves_black: Sequence[Sequence[int]], + possible_capturing_moves: Sequence[Sequence[int]] = None, + can_double_step: bool = False, + can_en_passant: bool = False, + captures_like_pawn: bool = False, + can_promote: bool = False, + can_jump_over_pieces: bool = False, + can_castle: bool = False, + ): + self.piece_id = piece_id + self.white_render = white_render + self.black_render = black_render + self.possible_moves_white = possible_moves_white + self.possible_moves_black = possible_moves_black + self.possible_capturing_moves = possible_capturing_moves + + self.can_double_step = can_double_step + self.can_en_passant = can_en_passant + self.captures_like_pawn = captures_like_pawn + self.can_promote = can_promote + self.can_jump_over_pieces = can_jump_over_pieces + self.can_castle = can_castle + + def get_piece_moves( + self, + board_state: Sequence[Sequence[int]], + player_id: str, + start_coord: Sequence[int], + previous_moves: Sequence[Move], + ) -> Sequence[Move]: + """ + Returns a sequence representing all moves this piece can make given the current environment + and rules this piece follows + """ + if player_id == "W": + possible_transformations = copy.deepcopy(self.possible_moves_white) + forward_direction = -1 + else: + possible_transformations = copy.deepcopy(self.possible_moves_black) + forward_direction = 1 + + # Get all relative transformations piece can make + if self.can_double_step: + possible_transformations += self._get_pawn_double_step_transformations( + player_id, start_coord + ) + if self.captures_like_pawn: + possible_transformations = self._remove_illegal_pawn_capture_transformations( + board_state, player_id, start_coord, possible_transformations, forward_direction + ) + if self.can_en_passant: + possible_transformations += self._get_en_passant_transformations( + board_state, start_coord, previous_moves, forward_direction + ) + + # Find all legal moves from transformations + piece_moves = self._get_moves_from_transformations( + board_state, player_id, start_coord, possible_transformations + ) + + # Add rule-specific moves + if self.can_promote: + piece_moves = self._add_promotion_moves(piece_moves) + if self.can_castle: + piece_moves += self._get_castling_possible_moves(board_state, player_id, previous_moves) + + return piece_moves + + def _get_moves_from_transformations( + self, + board_state: Sequence[Sequence[int]], + player_id: str, + start_coord: Sequence[int], + possible_transformations: Sequence[Sequence[int]], + ) -> Sequence[Move]: + """ + Given a piece's position within a board and the set of possible relative + transformations the piece can make, convert each transformation into a `Move` + object if: + 1) Transformation results in piece being on board + 2) Transformation doesn't result in piece ending up on piece of same color + 3) Transformation doesn't "jump" over other pieces, unless this piece is + allowed to do so (e.g. knight) + """ + piece_moves = [] + for move in possible_transformations: + new_row_idx = start_coord[0] + move[0] + new_col_idx = start_coord[1] + move[1] + + if not coord_within_board(new_row_idx, new_col_idx): + continue + + target_coord = [new_row_idx, new_col_idx] + target_piece_color, target_piece_id = parse_piece( + board_state, + target_coord[0], + target_coord[1], + ) + move = Move(start_coord, target_coord, None, False) + + if target_piece_color == player_id: + continue + if not self.can_jump_over_pieces and move_crosses_pieces(board_state, move): + continue + + piece_moves.append(move) + + return piece_moves + + def _get_pawn_double_step_transformations( + self, player_id: str, start_coord: Sequence[int] + ) -> Sequence[Sequence[int]]: + if player_id == "W" and start_coord[0] == 6: + return [[-2, 0]] + elif player_id == "B" and start_coord[0] == 1: + return [[2, 0]] + return [] + + def _remove_illegal_pawn_capture_transformations( + self, + board_state: Sequence[Sequence[int]], + player_id: str, + start_coord: Sequence[int], + possible_transformations: Sequence[Sequence[int]], + forward_direction: int, + ) -> Sequence[Sequence[int]]: + """ + Prevents pawns from "capturing forward" + """ + if self.piece_id != 0: + return possible_transformations + + new_possible_transformations = [] + capturing_moves = self.possible_capturing_moves + capturing_moves = [[move[0] * forward_direction, move[1]] for move in capturing_moves] + for move in possible_transformations + capturing_moves: + new_row_idx = start_coord[0] + move[0] + new_col_idx = start_coord[1] + move[1] + + if not coord_within_board(new_row_idx, new_col_idx): + continue + + target_piece_color, target_piece_id = parse_piece(board_state, new_row_idx, new_col_idx) + + if target_piece_color == "E" and move not in capturing_moves: + new_possible_transformations.append(move) + elif target_piece_color == get_other_player_id(player_id) and move in capturing_moves: + new_possible_transformations.append(move) + + return new_possible_transformations + + def _get_en_passant_transformations( + self, + board_state: Sequence[Sequence[int]], + start_coord: Sequence[int], + previous_moves: Sequence[Move], + forward_direction: int, + ) -> Sequence[Sequence[int]]: + last_move = previous_moves[-1] if len(previous_moves) > 0 else None + if last_move is not None and self.piece_id == 0: + _, last_piece_id = parse_piece( + board_state, last_move.target_coord[0], last_move.target_coord[1] + ) + + # If last move was pawn moving two tiles + if ( + last_piece_id == 0 + and abs(last_move.start_coord[0] - last_move.target_coord[0]) == 2 + ): + + # If on same row and one column apart + dx = start_coord[1] - last_move.target_coord[1] + dy = start_coord[0] - last_move.target_coord[0] + if dy == 0 and abs(dx) == 1: + return [[forward_direction, -dx]] + return [] + + def _add_promotion_moves(self, piece_moves: Sequence[Move]) -> Sequence[Move]: + new_piece_moves = [] + for move in piece_moves: + target_coord = move.target_coord + if target_coord[0] == 0 or target_coord[0] == 7: + for promotion_piece_id in [1, 2, 3, 4]: + move_promotion = copy.deepcopy(move) + move_promotion.promotion = promotion_piece_id + new_piece_moves.append(move_promotion) + else: + new_piece_moves.append(move) + + return new_piece_moves + + def _get_castling_possible_moves( + self, board_state: Sequence[Sequence[int]], player_id: str, previous_moves: Sequence[Move] + ) -> Sequence[Move]: + castling_moves = [] + if self.piece_id != 5: + return castling_moves + + def _can_pieces_castle( + king_init_coord: Sequence[int], rook_init_coord: Sequence[int], init_rook_id: int + ) -> Sequence[Move]: + if init_rook_id != 3: + return [] + + if has_piece_been_moved(king_init_coord, previous_moves) or has_piece_been_moved( + rook_init_coord, previous_moves + ): + return [] + + king_to_rook_move = Move(king_init_coord, rook_init_coord, None, False) + if move_crosses_pieces(board_state, king_to_rook_move): + return [] + + king_to_rook_path = get_path_between_coords(king_init_coord, rook_init_coord) + move = Move(king_init_coord, king_to_rook_path[1], None, True) + return [move] + + # ASSUME board init + king_init_coord = [7, 4] if player_id == "W" else [0, 4] + _, init_king_id = parse_piece(board_state, king_init_coord[0], king_init_coord[1]) + if init_king_id != 5: + return castling_moves + + # Queenside + queenside_rook_init_coord = [7, 7] if player_id == "W" else [0, 7] + _, init_rook_id = parse_piece( + board_state, queenside_rook_init_coord[0], queenside_rook_init_coord[1] + ) + castling_moves += _can_pieces_castle( + king_init_coord, queenside_rook_init_coord, init_rook_id + ) + + # Kingside + kingside_rook_init_coord = [7, 0] if player_id == "W" else [0, 0] + _, init_rook_id = parse_piece( + board_state, kingside_rook_init_coord[0], kingside_rook_init_coord[1] + ) + castling_moves += _can_pieces_castle( + king_init_coord, kingside_rook_init_coord, init_rook_id + ) + + return castling_moves diff --git a/evals/elsuite/cant_do_that_anymore/chess/utils.py b/evals/elsuite/cant_do_that_anymore/chess/utils.py new file mode 100644 index 0000000000..a92d072037 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/chess/utils.py @@ -0,0 +1,107 @@ +from dataclasses import dataclass +from typing import Sequence + + +@dataclass +class Move: + start_coord: Sequence[int] + target_coord: Sequence[int] + promotion: int # Either None for no promotion, or int for piece id of promotion + castling: bool + + +def get_other_player_id(this_player_id: str) -> str: + if this_player_id == "W": + return "B" + elif this_player_id == "B": + return "W" + else: + raise ValueError(f"this_player_id var must be 'W' or 'B', but is: {this_player_id}") + + +def parse_piece( + board_state: Sequence[Sequence[int]], row_idx: int, col_idx: int +) -> tuple[str, int]: + """ + Returns the color and id of the piece at the given coords. + """ + piece = board_state[row_idx][col_idx] + if piece == "E": + return "E", -1 + + color = piece[0] + id = piece[1] + return color, int(id) + + +def move_crosses_pieces(board_state: Sequence[Sequence[int]], move: Move) -> bool: + path = get_path_between_coords(move.start_coord, move.target_coord) + for (x1, y1) in path: + if board_state[x1][y1] != "E": + return True + + return False + + +def has_piece_been_moved( + piece_coord: Sequence[Sequence[int]], previous_moves: Sequence[Move] +) -> bool: + for move in previous_moves: + if move.start_coord == piece_coord: + return True + if move.target_coord == piece_coord: + return True + return False + + +def coord_within_board(row_idx: int, col_idx: int) -> bool: + if row_idx < 0 or row_idx > 7: + return False + if col_idx < 0 or col_idx > 7: + return False + + return True + + +def move_within_board(move: Move) -> bool: + target_coord = move.target_coord + return coord_within_board(target_coord[0], target_coord[1]) + + +def get_path_between_coords( + start_coord: Sequence[int], target_coord: Sequence[int] +) -> Sequence[Sequence[int]]: + # Unpack the start and end points + x1, y1 = start_coord + x2, y2 = target_coord + + # Determine the steps to take in each direction + dx = 1 if x2 > x1 else -1 if x2 < x1 else 0 + dy = 1 if y2 > y1 else -1 if y2 < y1 else 0 + + path = [(x1, y1)] + while (x1, y1) != (x2, y2): + if x1 != x2: + x1 += dx + if y1 != y2: + y1 += dy + path.append((x1, y1)) + + path = path[1:-1] + return path + + +def same_color_piece_at_move_start( + board_state: Sequence[Sequence[int]], move: Move, player_color: str +) -> bool: + start_coord = move.start_coord + piece_color, _ = parse_piece(board_state, start_coord[0], start_coord[1]) + return player_color == piece_color + + +def capturing_same_color(board_state: Sequence[Sequence[int]], move: Move) -> bool: + start_coord, target_coord = move.start_coord, move.target_coord + start_piece_color, _ = parse_piece(board_state, start_coord[0], start_coord[1]) + target_piece_color, _ = parse_piece(board_state, target_coord[0], target_coord[1]) + + return start_piece_color == target_piece_color diff --git a/evals/elsuite/cant_do_that_anymore/defaults.py b/evals/elsuite/cant_do_that_anymore/defaults.py new file mode 100644 index 0000000000..8c8645f824 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/defaults.py @@ -0,0 +1,15 @@ +TASK_DESCRIPTION = """You are playing a game of chess. Play the best move available to you. + +You must indicate which move you want to make by outputting the move using algebraic notation. Do not output anything else. +- Always write the full start and end position, for example, "a7b6" +- To castle, write the start and end position of the king, for example, "e8g8" +- Do not write "x" for capturing moves""" + +TASK_DESCRIPTION_VARIANT = """You are playing a game of a variant of chess. Play the best move available to you. + +You must indicate which move you want to make by outputting the move using algebraic notation. Do not output anything else. +- Always write the full start and end position, for example, "a7b6" +- To castle, write the start and end position of the king, for example, "e8g8" +- Do not write "x" for capturing moves + +This variant of chess does not follow the same rules as chess. All bishops now move like knights do. DO NOT move bishops diagonally, this is now considered an illegal move. All other pieces don't have their rules changed.""" diff --git a/evals/elsuite/cant_do_that_anymore/eval.py b/evals/elsuite/cant_do_that_anymore/eval.py new file mode 100644 index 0000000000..0ca6df5b0b --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/eval.py @@ -0,0 +1,201 @@ +import logging +import random +from typing import Any, Dict, Sequence, Union + +import numpy as np + +import evals.metrics +from evals.elsuite.cant_do_that_anymore.chess.board import BoardController +from evals.elsuite.cant_do_that_anymore.chess.board_test import default_board_init +from evals.elsuite.cant_do_that_anymore.chess.move_variants import ( + PIECE_ID_TO_INSTANCE, + PIECE_ID_TO_STR, + PIECE_STR_TO_ID, + VARIANT_PIECE_ID_TO_INSTANCE, +) +from evals.elsuite.cant_do_that_anymore.chess.notation import AlgebraicNotationParser +from evals.elsuite.cant_do_that_anymore.chess.pieces import Piece +from evals.elsuite.cant_do_that_anymore.chess.utils import ( + capturing_same_color, + move_within_board, + same_color_piece_at_move_start, +) +from evals.elsuite.cant_do_that_anymore.defaults import TASK_DESCRIPTION, TASK_DESCRIPTION_VARIANT +from evals.elsuite.cant_do_that_anymore.utils import ( + construct_messages, + get_binary_avg, + get_dataset_path, + get_diagonal_dataset_path, +) +from evals.eval import SolverEval +from evals.record import RecorderBase +from evals.solvers.solver import Solver, SolverResult +from evals.task_state import TaskState + +logger = logging.getLogger(__name__) + + +class CantDoThatAnymore(SolverEval): + def __init__( + self, + default_model_dataset: str = "gpt-3.5-turbo-0125", + remake_dataset_if_not_found: bool = True, + n_samples: int = 1000, + diagonal_variation: bool = False, + *args, + **kwargs, + ): + super().__init__(*args, **kwargs) + + self.default_model_dataset = default_model_dataset + self.remake_dataset_if_not_found = remake_dataset_if_not_found + self.n_samples = n_samples + self.diagonal_variation = diagonal_variation + self.rng: random.Random = random.Random(self.seed) + + def eval_sample(self, solver: Solver, sample: Any, rng: random.Random): + previous_moves, next_filtered_moves = ( + sample["previous_moves"], + sample["next_filtered_moves"], + ) + + def construct_controller(piece_id_to_instance: Dict[int, Piece]) -> BoardController: + controller = BoardController( + default_board_init, + piece_id_to_instance, + PIECE_STR_TO_ID, + PIECE_ID_TO_STR, + AlgebraicNotationParser(PIECE_STR_TO_ID, PIECE_ID_TO_STR), + ) + for move in previous_moves: + controller.update_board(move) + return controller + + default_controller = construct_controller(PIECE_ID_TO_INSTANCE) + variant_controller = construct_controller(VARIANT_PIECE_ID_TO_INSTANCE) + + # Get solver prediction. Ideally I wouldn't pass the legal_moves to the solvers, they + # should figure them out themselves, but it's necessary for the random solver + def get_solver_pred( + task_description: str, + controller: BoardController, + ) -> SolverResult: + task_state = TaskState( + task_description, + messages=construct_messages(previous_moves), + ) + return solver(task_state, **{"max_tokens": 4}) + + solver_result = get_solver_pred(TASK_DESCRIPTION, default_controller) + solver_result_variant = get_solver_pred(TASK_DESCRIPTION_VARIANT, variant_controller) + + metrics = { + "move": next_filtered_moves, + "predicted_move": solver_result.output.strip() in next_filtered_moves, + "predicted_move_in_variant": solver_result_variant.output.strip() + in next_filtered_moves, + "num_previous_moves": len(previous_moves), + "previous_moves": previous_moves, + } + + # Add violations to metrics + metrics.update( + self.get_violations( + default_controller, solver_result.output, previous_moves, "standard" + ) + ) + metrics.update( + self.get_violations( + variant_controller, solver_result_variant.output, previous_moves, "variant" + ) + ) + + evals.record.record_metrics(**metrics) + + def run(self, recorder: RecorderBase) -> dict[str, Union[float, int]]: + if self.diagonal_variation: + self.samples_jsonl = get_diagonal_dataset_path( + registry_path=self._prefix_registry_path("") + ) + else: + self.samples_jsonl = get_dataset_path( + solver=self._solver, + registry_path=self._prefix_registry_path(""), + remake_dataset_if_not_found=self.remake_dataset_if_not_found, + default_model_dataset=self.default_model_dataset, + ) + samples = self.get_samples() + samples = self.rng.sample(samples, min(self.n_samples, len(samples))) + + self.eval_all_samples(recorder, samples) + metrics = recorder.get_metrics() + + predicted_move_proportion = get_binary_avg(metrics, "predicted_move") + predicted_move_in_variant_proportion = get_binary_avg(metrics, "predicted_move_in_variant") + + avg_num_previous_moves = sum([i["num_previous_moves"] for i in metrics]) / len(metrics) + std_num_previous_moves = np.std([i["num_previous_moves"] for i in metrics]) + + delta = predicted_move_in_variant_proportion - predicted_move_proportion + variant_impact_factor = (delta / predicted_move_proportion) if predicted_move_proportion != 0 else 0 + + results = { + "variant_impact_factor": variant_impact_factor, + "delta": delta, + "predicted_move_proportion": predicted_move_proportion, + "predicted_move_in_variant_proportion": predicted_move_in_variant_proportion, + "avg_num_previous_moves": avg_num_previous_moves, + "std_num_previous_moves": std_num_previous_moves, + } + + # Add violations + violation_keys = [i for i in metrics[0].keys() if "violation" in i] + violation_results = { + f"{name}_rate": get_binary_avg(metrics, name) for name in violation_keys + } + results.update(violation_results) + + return results + + def get_violations( + self, + controller: BoardController, + solver_output: str, + previous_moves: Sequence[str], + variant_name: str, + ) -> dict: + solver_color = "W" if len(previous_moves) % 2 == 0 else "B" + + piece_moved_outside_board = False + moving_invalid_piece = False + piece_capturing_same_color = False + + violation_metrics = {} + try: + move = controller.notation_parser._str_to_move( + solver_output, controller.board.board_state + ) + + piece_moved_outside_board = not move_within_board(move) + moving_invalid_piece = not same_color_piece_at_move_start( + controller.board.board_state, move, solver_color + ) + piece_capturing_same_color = capturing_same_color(controller.board.board_state, move) + incorrect_notation = False + except (ValueError, KeyError): + incorrect_notation = True + + violation = ( + piece_moved_outside_board + or moving_invalid_piece + or piece_capturing_same_color + or incorrect_notation + ) + violation_metrics = { + f"{variant_name}_violation": violation, + f"{variant_name}_violation_moved_outside_board": piece_moved_outside_board, + f"{variant_name}_violation_moving_invalid_piece": moving_invalid_piece, + f"{variant_name}_violation_capturing_same_color": piece_capturing_same_color, + f"{variant_name}_violation_incorrect_notation": incorrect_notation, + } + return violation_metrics diff --git a/evals/elsuite/cant_do_that_anymore/scripts/dataset_creation.py b/evals/elsuite/cant_do_that_anymore/scripts/dataset_creation.py new file mode 100644 index 0000000000..e0c7a0265a --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/scripts/dataset_creation.py @@ -0,0 +1,312 @@ +import argparse +import copy +import os +import pathlib +from typing import Sequence + +import chess.pgn +import requests +import zstandard +from tqdm import tqdm + +from evals.elsuite.cant_do_that_anymore.chess.board import BoardController +from evals.elsuite.cant_do_that_anymore.chess.utils import Move, parse_piece +from evals.elsuite.cant_do_that_anymore.utils import ( + assert_boards_consistent, + dump_sequence_to_jsonl, + initialise_boards, +) + + +def prepare_lichess_2014_dataset(out_dir: str) -> str: + """ + Downloads and extracts Lichess 2014 April dataset, returns the + path to the extracted .pgn file + """ + fname = "lichess_db_standard_rated_2014-04.pgn.zst" + raw_data_out_path = os.path.join(out_dir, fname) + if not os.path.exists(raw_data_out_path): + url = "https://database.lichess.org/standard/" + fname + r = requests.get(url) + open(raw_data_out_path, "wb").write(r.content) + + out_path = os.path.join(out_dir, "pgn_data.pgn") + if not os.path.exists(out_path): + input_file = pathlib.Path(raw_data_out_path) + with open(input_file, "rb") as compressed: + decomp = zstandard.ZstdDecompressor() + with open(out_path, "wb") as destination: + decomp.copy_stream(compressed, destination) + + return out_path + + +class MoveFilter: + def __call__( + self, + default_controller: BoardController, + variant_controller: BoardController, + move: chess.Move, + player_id: str, + ) -> bool: + raise NotImplementedError() + + +class SpecialMoveFilter(MoveFilter): + """ + Filters for moves that are: + 1) Legal under the normal rules of chess + 2) Illegal under the variant's rules (i.e. bishop is moved) + """ + + def __call__( + self, + default_controller: BoardController, + variant_controller: BoardController, + move: Move, + player_id: str, + ) -> bool: + if not is_move_illegal(default_controller, move, player_id) and is_move_illegal( + variant_controller, move, player_id + ): + return True + + return False + + +class ControlMoveFilter(MoveFilter): + """ + Finds positions where solvers should have (almost) equivalent predictions under + both sets of rules + Filters for moves that are: + 1) Legal under both the normal and variant's rules of chess + 2) Are on a board containing no bishops + 3) Are on a board where no pawns are close to promoting; neither players + pawns are in their last three rows + 4) Are on a board with more than four pieces between both players + """ + + def __call__( + self, + default_controller: BoardController, + variant_controller: BoardController, + move: Move, + player_id: str, + ) -> bool: + if is_move_illegal(default_controller, move, player_id): + return False + if is_move_illegal(variant_controller, move, player_id): + return False + + board_state = default_controller.board.board_state + num_pieces = 0 + for row_idx in range(8): + for col_idx in range(8): + _, piece_id = parse_piece(board_state, row_idx, col_idx) + if piece_id == 2: + return False + elif piece_id == 0: + if player_id == "W" and row_idx <= 2: + return False + elif player_id == "B" and row_idx >= 5: + return False + elif piece_id != -1: + num_pieces += 1 + + if num_pieces < 4: + return False + + return True + + +def is_move_illegal(controller: BoardController, move: chess.Move, player_id: str) -> bool: + legal_moves = controller.get_player_legal_moves(player_id) + if move in legal_moves: + return False + return True + + +def find_specific_moves_in_game( + game: chess.pgn.Game, + game_idx: int, + move_filter: MoveFilter, + default_controller: BoardController, + variant_controller: BoardController, + their_controller: chess.Board, + filter_if_found_previous: bool, +) -> Sequence[dict]: + """ + Given a game, finds all moves that satisfy the given filter + If filter_if_found_previous is True, only finds first move in game that + satisfies filter + """ + player_id = "W" + previous_moves = [] + filtered_moves = [] + for move in game.mainline_moves(): + move = move.uci() + + if move_filter(default_controller, variant_controller, move, player_id): + filtered_moves.append( + { + "game_idx": game_idx, + "previous_moves": copy.deepcopy(previous_moves), + "next_filtered_moves": [move], + "any_previous_move_found": len(filtered_moves) > 0, + } + ) + if filter_if_found_previous: + break + + # Ensure my implementation is correct + assert_boards_consistent(default_controller, their_controller, player_id) + + # Update boards + default_controller.update_board(move) + their_controller.push_san(move) + + variant_controller.board.board_state = default_controller.board.board_state + variant_controller.previous_moves = default_controller.previous_moves + + player_id = "B" if player_id == "W" else "W" + previous_moves.append(move) + + return filtered_moves + + +def create_dataset_of_specific_moves( + pgn_path: str, + move_filter: MoveFilter, + target_num_examples: int, + filter_if_found_previous: bool, + filter_for_unique_previous_moves: bool, + continuously_save: bool, + out_path: str, +): + """ + Iterates over games in dataset and filters move according to the given move_filter + If filter_for_unique_previous_moves is True, filter to only include moves that have + unique sets of previous moves + If continuously_save is True, saves dataset everytime it is updated + """ + pgn = open(pgn_path) + dataset = [] + unique_previous_moves = set() + + t_bar = tqdm(total=target_num_examples) + game_idx = 0 + while True: + game = chess.pgn.read_game(pgn) + if game is None: + break + + default_controller, variant_controller, their_controller = initialise_boards() + filtered_moves = find_specific_moves_in_game( + game, + game_idx, + move_filter, + default_controller, + variant_controller, + their_controller, + filter_if_found_previous, + ) + + if filter_for_unique_previous_moves: + for example in filtered_moves: + previous_moves = example["previous_moves"] + if set(previous_moves) not in unique_previous_moves: + dataset.append(example) + unique_previous_moves.add(frozenset(previous_moves)) + t_bar.update(1) + if continuously_save: + dump_sequence_to_jsonl(dataset, out_path) + + elif len(filtered_moves) > 0: + dataset += filtered_moves + t_bar.update(len(filtered_moves)) + if continuously_save: + dump_sequence_to_jsonl(dataset, out_path) + + game_idx += 1 + t_bar.set_description(f"Num games examined: {game_idx}") + + if len(dataset) >= target_num_examples: + break + + return dataset + + +def main(args: argparse.Namespace): + lichess_path = prepare_lichess_2014_dataset(args.out_dir) + + if args.make_special_moves: + move_filter = SpecialMoveFilter() + dataset_name = "special_moves_dataset.jsonl" + out_path = os.path.join(args.out_dir, dataset_name) + dataset = create_dataset_of_specific_moves( + lichess_path, + move_filter, + target_num_examples=args.n_moves, + filter_if_found_previous=args.filter_if_found_previous, + filter_for_unique_previous_moves=args.filter_for_unique_previous_moves, + continuously_save=args.continuously_save, + out_path=out_path, + ) + dump_sequence_to_jsonl(dataset, out_path) + + if args.make_control_moves: + move_filter = ControlMoveFilter() + dataset_name = "control_moves_dataset.jsonl" + out_path = os.path.join(args.out_dir, dataset_name) + dataset = create_dataset_of_specific_moves( + lichess_path, + move_filter, + target_num_examples=args.n_moves, + filter_if_found_previous=args.filter_if_found_previous, + filter_for_unique_previous_moves=args.filter_for_unique_previous_moves, + continuously_save=args.continuously_save, + out_path=out_path, + ) + dump_sequence_to_jsonl(dataset, out_path) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + + parser.add_argument("--n_moves", type=int, default=5000) + parser.add_argument( + "--out_dir", type=str, default="./evals/registry/data/cant_do_that_anymore/" + ) + parser.add_argument( + "--make_special_moves", + action="store_true", + help="Whether to search and build a dataset of special moves", + default=False, + ) + parser.add_argument( + "--make_control_moves", + action="store_true", + help="Whether to search and build a dataset of control moves", + default=False, + ) + parser.add_argument( + "--filter_if_found_previous", + action="store_true", + help="Whether to filter out moves that have had previous moves that satisfy the filtering condition.", + default=False, + ) + parser.add_argument( + "--filter_for_unique_previous_moves", + action="store_true", + help="Whether to only search for moves with unique previous moves (up to such position at the move)", + default=False, + ) + parser.add_argument( + "--continuously_save", + action="store_true", + help="Whether to save the dataset everytime a new example has been found", + default=False, + ) + args = parser.parse_args() + + main(args) diff --git a/evals/elsuite/cant_do_that_anymore/scripts/diagonal_dataset_creation.py b/evals/elsuite/cant_do_that_anymore/scripts/diagonal_dataset_creation.py new file mode 100644 index 0000000000..491acf3c95 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/scripts/diagonal_dataset_creation.py @@ -0,0 +1,316 @@ +import argparse +import copy +import os +import random +from typing import Optional, Sequence + +from stockfish import Stockfish +from tqdm import tqdm + +from evals.elsuite.cant_do_that_anymore.chess.board import BoardController +from evals.elsuite.cant_do_that_anymore.chess.move_variants import DIAGONAL_MOVES +from evals.elsuite.cant_do_that_anymore.chess.utils import ( + Move, + coord_within_board, + move_crosses_pieces, + parse_piece, +) +from evals.elsuite.cant_do_that_anymore.utils import dump_sequence_to_jsonl, initialise_boards + +# NOTE change threads, hash depending on hardware +# https://pypi.org/project/stockfish/ +STOCKFIAH_MOVES_CONSIDERED = 5 +STOCKFISH_DEPTH = 18 +STOCKFISH_PARAMS = { + "Debug Log File": "", + "Contempt": 0, + "Min Split Depth": 0, + "Threads": 8, + "Ponder": "false", + "Hash": 4096, + "MultiPV": 1, + "Skill Level": 10, + "Move Overhead": 10, + "Minimum Thinking Time": 20, + "Slow Mover": 100, + "UCI_Chess960": "true", + "UCI_LimitStrength": "false", + "UCI_Elo": 1500, +} + + +def get_stockfish_move(stockfish: Stockfish, num_moves_to_consider: int) -> str: + """ + Gets the next move predicted by stockfish. Gets top n predictions and + selects randomly weighted by each move's centipawn value + Filters out bishop promotions, since our variant shouldn't have bishops + """ + # Get top moves, filter out bad ones + top_moves = stockfish.get_top_moves(num_moves_to_consider) + + # Filter out bishop promotions + top_moves = [i for i in top_moves if not i["Move"].endswith("b")] + + # If stockfish considers moves that it knows will lead to mate, only + # select from these moves + mates = [i for i in top_moves if i["Mate"] is not None] + if len(mates) > 0: + top_moves = mates + + # Ensures centipawn value isn't None + if all([i["Centipawn"] is None for i in top_moves]): + for move in top_moves: + move["Centipawn"] = 1 + else: + top_moves = [i for i in top_moves if i["Centipawn"] is not None] + + # Makes all centipawns positive + min_centipawn_value = min([i["Centipawn"] for i in top_moves]) + for move in top_moves: + move["Centipawn"] += abs(min_centipawn_value) + + # Normalise centipawn to a probability distribution + centipawn_sum = sum([i["Centipawn"] for i in top_moves]) + for move in top_moves: + move["prob"] = move["Centipawn"] / centipawn_sum + + # Pick move randomly + prob = random.uniform(0, 1) + selected_move = None + for move in top_moves: + prob -= move["prob"] + if prob <= 0: + selected_move = move["Move"] + break + + return selected_move + + +def parse_stockfish_move(controller: BoardController, move: str) -> str: + """ + When stockfish outputs a castling move, the move is from the kings position to the + rooks position, e.g. "e8a8" + In my framework castling is indicated by the start+end position of the king, e.g. "e8c8" + This functions converts the stockfish notation to my notation + """ + move = controller.notation_parser._str_to_move(move, controller.board.board_state) + _, piece_id = parse_piece( + controller.board.board_state, move.start_coord[0], move.start_coord[1] + ) + + # If castling move + dy = move.target_coord[1] - move.start_coord[1] + if piece_id == 5: + if dy > 2 or dy < -2: + direction = dy / abs(dy) + if direction == 1: # Kingside castling + move.target_coord = [move.target_coord[0], move.target_coord[1] - 1] + else: # Queenside castling + move.target_coord = [move.target_coord[0], move.target_coord[1] + 2] + + move = controller.notation_parser._move_to_str(move, controller.board.board_state) + return move + + +def get_bishop_diagonal_moves(controller: BoardController, player_id: str) -> Sequence[str]: + """ + Gets all possible diagonal moves that a bishop could make on a board, even if the bishop isn't + allowed to move diagonally under the board's rules + """ + # Find all bishops on board + bishop_coords = [] + board_state = controller.board.board_state + for row_idx in range(8): + for col_idx in range(8): + piece_color, piece_id = parse_piece(board_state, row_idx, col_idx) + if piece_color == player_id and piece_id == 2: + bishop_coords.append([row_idx, col_idx]) + + # Find all possible diagonal movements of each bishop + bishop_diagonal_moves = [] + for row_idx, col_idx in bishop_coords: + for transformation in DIAGONAL_MOVES: + new_coord = [row_idx + transformation[0], col_idx + transformation[1]] + move = Move([row_idx, col_idx], new_coord, promotion=None, castling=False) + + # If piece doesn't move + if transformation[0] == 0 and transformation[1] == 0: + continue + # If transformation moves piece outside board + if not coord_within_board(new_coord[0], new_coord[1]): + continue + # If transformation moves onto piece of same color + piece_color, _ = parse_piece(controller.board.board_state, new_coord[0], new_coord[1]) + if piece_color == player_id: + continue + # If move crosses friendly pieces + if move_crosses_pieces(controller.board.board_state, move): + continue + + move = controller.notation_parser._move_to_str(move, controller.board.board_state) + bishop_diagonal_moves.append(move) + + return bishop_diagonal_moves + + +def find_specific_moves_in_game( + game_idx: int, + variant_controller: BoardController, + filter_if_found_previous: bool, + max_moves: int, +) -> Sequence[dict]: + """ + Simulates an individual game, using the variant's rules. Finds all possible + diagonal moves from bishops (even though moving bishops diagonally is + illegal under the variant) + If filter_if_found_previous is True, only finds the first position with possible + bishop moves + """ + stockfish = Stockfish(depth=STOCKFISH_DEPTH, parameters=STOCKFISH_PARAMS) + # HACK to have stockfish play our variant, just swap out the bishops for knights + # then later pretend the knights are bishops + stockfish.set_fen_position("rnnqknnr/pppppppp/8/8/8/8/PPPPPPPP/RNNQKNNR w KQkq - 0 1") + previous_moves = [] + player_id = "W" + + # Get ELO of each player + elos = [1350, 1000] + random.shuffle(elos) + white_elo, black_elo = elos + + bishop_diagonal_moves = [] + for _ in range(max_moves): + if player_id == "W": + stockfish.set_elo_rating(white_elo) + else: + stockfish.set_elo_rating(black_elo) + + # Find all diagonal bishop moves from this position + found_moves = get_bishop_diagonal_moves(variant_controller, player_id) + if len(found_moves) > 0: + bishop_diagonal_moves.append( + { + "game_idx": game_idx, + "previous_moves": copy.deepcopy(previous_moves), + "next_filtered_moves": found_moves, + } + ) + if filter_if_found_previous: + break + + move = get_stockfish_move(stockfish, STOCKFIAH_MOVES_CONSIDERED) + stockfish.make_moves_from_current_position([move]) + + # Parse into notation that is compatible with my framework + move = parse_stockfish_move(variant_controller, move) + variant_controller.update_board(move) + + player_id = "B" if player_id == "W" else "W" + previous_moves.append(move) + + # If checkmate or stalemate, end + if len(variant_controller.get_player_legal_moves(player_id)) == 0: + break + + return bishop_diagonal_moves + + +def create_bishop_diagonal_dataset( + target_num_examples: int, + max_moves: int, + filter_if_found_previous: bool, + filter_for_unique_previous_moves: bool, + continuously_save: bool, + out_path: Optional[str], +) -> Sequence[dict]: + """ + Simulates stockfish games and finds possible diagonal moves that could be + made by bishops. + If filter_if_found_previous is True, finds the first move that satisfies this + criteria in each game + If filter_for_unique_previous_moves is True, filters to ensure each + example has a unique set of previous moves + If continuously_save is True, saves dataset everytime it is updated + """ + dataset = [] + unique_previous_moves = set() + + t_bar = tqdm(total=target_num_examples) + game_idx = 0 + while True: + _, variant_controller, _ = initialise_boards() + filtered_moves = find_specific_moves_in_game( + game_idx, + variant_controller, + filter_if_found_previous, + max_moves, + ) + + if filter_for_unique_previous_moves: + for example in filtered_moves: + previous_moves = example["previous_moves"] + if set(previous_moves) not in unique_previous_moves: + dataset.append(example) + unique_previous_moves.add(frozenset(previous_moves)) + t_bar.update(1) + if continuously_save: + dump_sequence_to_jsonl(dataset, out_path) + + elif len(filtered_moves) > 0: + dataset += filtered_moves + t_bar.update(len(filtered_moves)) + if continuously_save: + dump_sequence_to_jsonl(dataset, out_path) + + game_idx += 1 + t_bar.set_description(f"Num games examined: {game_idx}") + + if len(dataset) >= target_num_examples: + break + + return dataset + + +def main(args: argparse.Namespace): + dataset_name = "diagonal_moves_dataset.jsonl" + out_path = os.path.join(args.out_dir, dataset_name) + dataset = create_bishop_diagonal_dataset( + target_num_examples=args.n_moves, + max_moves=args.max_moves, + filter_if_found_previous=args.filter_if_found_previous, + filter_for_unique_previous_moves=args.filter_for_unique_previous_moves, + continuously_save=args.continuously_save, + out_path=out_path, + ) + dump_sequence_to_jsonl(dataset, out_path) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + + parser.add_argument("--n_moves", type=int, default=5000) + parser.add_argument("--max_moves", type=int, default=50) + parser.add_argument( + "--out_dir", type=str, default="./evals/registry/data/cant_do_that_anymore/" + ) + parser.add_argument( + "--filter_if_found_previous", + action="store_true", + help="Whether to filter out moves that have had previous moves that satisfy the filtering condition", + default=False, + ) + parser.add_argument( + "--filter_for_unique_previous_moves", + action="store_true", + help="Whether to only search for moves with unique previous moves (up to such position at the move)", + default=False, + ) + parser.add_argument( + "--continuously_save", + action="store_true", + help="Whether to save the dataset everytime a new example has been found", + default=False, + ) + args = parser.parse_args() + + main(args) diff --git a/evals/elsuite/cant_do_that_anymore/scripts/make_plots.py b/evals/elsuite/cant_do_that_anymore/scripts/make_plots.py new file mode 100644 index 0000000000..bd0ea4d5cc --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/scripts/make_plots.py @@ -0,0 +1,128 @@ +import argparse +import os +from pathlib import Path +from typing import Sequence + +import pandas as pd +from matplotlib import pyplot as plt + +from evals.elsuite.cant_do_that_anymore.chess.utils import parse_piece +from evals.elsuite.cant_do_that_anymore.utils import initialise_boards +from evals.utils.log_utils import ( + extract_individual_results, + extract_spec, + get_final_results_from_dir, +) + + +def extract_results(datadir: Path) -> pd.DataFrame: + df_agg = [] # Aggregated results + df_samples = [] # Per sample results + for path, results in sorted(list(get_final_results_from_dir(datadir).items())): + spec = extract_spec(path) + solver_path = Path(spec["completion_fns"][0]) + model = solver_path.name + solver = solver_path.parent.name + # Remove root section of path, which is the eval name + solver_path = solver_path.relative_to(solver_path.parts[0]) + # Aggregated + df_agg.append( + { + "solver_path": str(solver_path), + "model": str(model), + "solver": str(solver), + **spec["run_config"]["eval_spec"]["args"], + **results, + } + ) + # Per-sample + for res in extract_individual_results(path): + df_samples.append( + { + "solver_path": str(solver_path), + "model": str(model), + "solver": str(solver), + **spec["run_config"]["eval_spec"]["args"], + **res, + } + ) + df_agg = pd.DataFrame(df_agg) + df_samples = pd.DataFrame(df_samples) + return df_agg, df_samples + + +def render_results(df: pd.DataFrame, out_dir: Path): + agg_operations = { + "predicted_move_proportion": ["mean", "sem"], + "predicted_move_in_variant_proportion": ["mean", "sem"], + } + df = df.groupby("solver_path").agg(agg_operations).reset_index() + df = df.round(2) + print(df.to_csv(index=False)) + df.to_csv(os.path.join(out_dir, "results.csv"), index=False) + + +def compute_num_previous_bishop_moves(previous_moves: Sequence[str]) -> int: + controller, _, _ = initialise_boards() + + num_previous_bishop_moves = 0 + for move in previous_moves: + start_coord = controller.notation_parser._str_to_move( + move, controller.board.board_state + ).start_coord + _, piece_id = parse_piece(controller.board.board_state, start_coord[0], start_coord[1]) + if piece_id == 2: + num_previous_bishop_moves += 1 + + controller.update_board(move) + + return num_previous_bishop_moves + + +def plot_diagonal_bishop_results(df: pd.DataFrame, out_dir: Path): + # Get number of previous bishop moves + df["num_previous_bishop_moves"] = [ + compute_num_previous_bishop_moves(i) for i in df["previous_moves"] + ] + + # Calculate headline metrics per solver, and number of previous moves + agg_operations = { + "predicted_move_in_variant": ["mean"], + } + df = df.groupby(["solver_path", "num_previous_bishop_moves"]).agg(agg_operations).reset_index() + + # Plot separately for each solver + for model, group in df.groupby("solver_path"): + plt.plot( + group["num_previous_bishop_moves"], + group["predicted_move_in_variant"], + label=model, + ) + + plt.xlabel("Num previous bishop moves") + plt.ylabel("Proportion of (illegal) predicted diagonal bishop moves") + plt.ylim([0, 1]) + plt.legend() + plt.savefig(os.path.join(out_dir, "diagonal.png")) + plt.show() + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--log_dir", "-d", type=str, required=True) + parser.add_argument("--out_dir", "-o", type=str, required=True) + parser.add_argument("--diagonal_variant", action="store_true", default=False) + args = parser.parse_args() + log_dir = Path(args.log_dir) + out_dir = Path(args.out_dir) + out_dir.mkdir(exist_ok=True, parents=True) + + df_agg, df_samples = extract_results(log_dir) + render_results(df_agg, out_dir) + + if args.diagonal_variant: + plot_diagonal_bishop_results(df_samples, out_dir) + + +if __name__ == "__main__": + main() diff --git a/evals/elsuite/cant_do_that_anymore/scripts/run_experiments.sh b/evals/elsuite/cant_do_that_anymore/scripts/run_experiments.sh new file mode 100755 index 0000000000..68fe4ac5e7 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/scripts/run_experiments.sh @@ -0,0 +1,67 @@ +#!/bin/bash +logdir=./logs +outputdir=./outputs + +timestamp=$(date +%Y%m%d_%H%M%S) +logpathbase=$logdir/$timestamp/ + +mkdir -p ${logpathbase} + +declare -a SOLVERS_ZEROSHOT=( + "generation/direct/gpt-3.5-turbo" + "chess/generation/direct/gpt-3.5-turbo-instruct" + "generation/direct/gpt-4-turbo-preview" + "chess/generation/direct/gpt-4-base" +) + +# See if variant was indicated +run_diagonal_variant=1 +for arg in "$@" +do + if [[ $arg == "--no_diagonal_variant" ]]; then + run_diagonal_variant=0 + break + fi +done + +# TODO CoT solvers + +echo Running experiments and logging to $logpathbase + +for run_idx in {0..2} +do + for solver in "${SOLVERS_ZEROSHOT[@]}" + do + log_name=${solver//\//-} + oaieval $solver cant_do_that_anymore \ + --record_path ${logpathbase}run_${run_idx}_${log_name}.log \ + --extra_eval_params n_samples=1000 \ + --seed ${run_idx} + done +done + +echo Done running experiments, all logs in $logpathbase + +echo Producing plots, outputs to $outputdir +python make_plots.py --log_dir $logpathbase --out_dir $outputdir + +if [[ $run_diagonal_variant -eq 1 ]]; then + echo Running diagonal experiment and logging to $logpathbase + + for run_idx in {0..2} + do + for solver in "${SOLVERS_ZEROSHOT[@]}" + do + log_name=${solver//\//-} + oaieval $solver cant_do_that_anymore.all_diagonal \ + --record_path ${logpathbase}run_${run_idx}_${log_name}.log \ + --extra_eval_params n_samples=1000 \ + --seed ${run_idx} + done + done + + echo Done running experiments, all logs in $logpathbase + + echo Producing plots, outputs to $outputdir + python make_plots.py --log_dir $logpathbase --out_dir $outputdir --diagonal_variant +fi \ No newline at end of file diff --git a/evals/elsuite/cant_do_that_anymore/utils.py b/evals/elsuite/cant_do_that_anymore/utils.py new file mode 100644 index 0000000000..519aad8596 --- /dev/null +++ b/evals/elsuite/cant_do_that_anymore/utils.py @@ -0,0 +1,250 @@ +import json +import logging +import os +from multiprocessing.pool import ThreadPool +from typing import Sequence + +import chess +from tqdm import tqdm + +from evals.elsuite.cant_do_that_anymore.chess.board import BoardController +from evals.elsuite.cant_do_that_anymore.chess.board_test import default_board_init +from evals.elsuite.cant_do_that_anymore.chess.move_variants import ( + PIECE_ID_TO_INSTANCE, + PIECE_ID_TO_STR, + PIECE_STR_TO_ID, + VARIANT_PIECE_ID_TO_INSTANCE, +) +from evals.elsuite.cant_do_that_anymore.chess.notation import AlgebraicNotationParser +from evals.elsuite.cant_do_that_anymore.defaults import TASK_DESCRIPTION +from evals.record import DummyRecorder, RecorderBase +from evals.solvers.solver import DummySolver, Solver +from evals.task_state import Message, TaskState + +logger = logging.getLogger(__name__) + + +def construct_messages(previous_moves: Sequence[str]) -> Sequence[Message]: + """ + Creates list of Message's containing the previous chess moves. The last + Message is always from the "user" + """ + solver_is_white = len(previous_moves) % 2 == 0 + messages = [] + current_player = "assistant" if solver_is_white else "user" + for move in previous_moves: + messages.append(Message(current_player, move)) + # toggle current player + current_player = "assistant" if current_player == "user" else "user" + + return messages + + +def dump_sequence_to_jsonl(data: Sequence[dict], path: str): + with open(path, "w+") as f: + for example in data: + example = json.dumps(example) + f.write(f"{example}\n") + + +def load_sequence_from_jsonl(path: str) -> Sequence[dict]: + data = [] + with open(path, "r") as f: + for line in f: + line = json.loads(line) + data.append(line) + + return data + + +def initialise_boards() -> tuple[BoardController, BoardController, chess.Board]: + """ + Initialises local chess framework, and framework from + python-chess library + """ + default_controller = BoardController( + default_board_init, + PIECE_ID_TO_INSTANCE, + PIECE_STR_TO_ID, + PIECE_ID_TO_STR, + AlgebraicNotationParser(PIECE_STR_TO_ID, PIECE_ID_TO_STR), + ) + variant_controller = BoardController( + default_board_init, + VARIANT_PIECE_ID_TO_INSTANCE, + PIECE_STR_TO_ID, + PIECE_ID_TO_STR, + AlgebraicNotationParser(PIECE_STR_TO_ID, PIECE_ID_TO_STR), + ) + their_controller = chess.Board() + + return default_controller, variant_controller, their_controller + + +def assert_boards_consistent( + controller: BoardController, their_controller: chess.Board, player_id: str +): + """ + Checks both boards have consistent states by ensuring both have same set of legal moves + """ + our_legal_moves = sorted(controller.get_player_legal_moves(player_id)) + their_legal_moves = sorted([str(i) for i in their_controller.legal_moves]) + if our_legal_moves != their_legal_moves: + our_additional_moves = list(set(our_legal_moves) - set(their_legal_moves)) + their_additional_moves = list(set(their_legal_moves) - set(our_legal_moves)) + assert False, f""" + Inconsistent legal moves between the boards! + Our legal moves: {our_legal_moves}, + Their legal moves: {their_legal_moves}, + Moves we had they didnt: {our_additional_moves}, + Moves they had we didn't: {their_additional_moves}, + Board state:\n{controller.board.board_state} + """ + + +def does_solver_predict_move( + solver: Solver, + recorder: RecorderBase, + task_description: str, + special_move: str, + previous_moves: Sequence[str], +): + task_state = TaskState( + task_description, + construct_messages(previous_moves), + ) + + with recorder.as_default_recorder(-1): + solver_result = solver(task_state, **{"max_tokens": 4}) + pred_str = solver_result.output.strip() + + if pred_str == special_move: + return True + + return False + + +def process_example(work_input: dict): + solver, recorder, example, task_description = ( + work_input["solver"], + work_input["recorder"], + work_input["example"], + work_input["task_description"], + ) + special_move, previous_moves = example["special_move"], example["previous_moves"] + + predicts_move = does_solver_predict_move( + solver, + recorder, + task_description, + special_move, + previous_moves, + ) + return predicts_move, example + + +def get_solver_predictions( + solver: Solver, + recorder: RecorderBase, + special_moves_dataset: Sequence[dict], + n_threads: int, + task_description: str, +) -> Sequence[dict]: + """ + Filter to find all special moves that the solver would have predicted under the normal + rules of chess with temp=0, then dump this dataset + """ + solver_moves_dataset = [] + work_items = [ + { + "solver": solver, + "recorder": recorder, + "example": example, + "task_description": task_description, + } + for example in special_moves_dataset + ] + + t_bar = tqdm(total=len(special_moves_dataset)) + with ThreadPool(n_threads) as pool: + iter = pool.imap_unordered(process_example, work_items) + + for result in (t_bar := tqdm(iter, total=len(work_items))): + predicts_move, example = result + if predicts_move: + solver_moves_dataset.append(example) + t_bar.set_description(f"Dataset size: {len(solver_moves_dataset)}") + + return solver_moves_dataset + + +def get_dataset_path( + solver: Solver, + registry_path: str, + remake_dataset_if_not_found: bool, + default_model_dataset: str, +) -> str: + """ + This dataset requires each evaluated model to have its own dataset. We get the exact + model being exaluated, check if a dataset exists for it, if not we generate one + """ + recorder = DummyRecorder(None) + with recorder.as_default_recorder("x"): + solver_version = solver.model_version + + # If nested solver, convert returned dict to str + if isinstance(solver_version, dict): + solver_version = json.dumps(solver_version) + + all_datasets_path = os.path.join(registry_path, "cant_do_that_anymore") + + # Check if dataset exists + solver_dataset_path = os.path.join(all_datasets_path, f"{solver_version}_dataset.jsonl") + if os.path.exists(solver_dataset_path): + return solver_dataset_path + + # Remake, or load default + if isinstance(solver, DummySolver): + return f"cant_do_that_anymore/{default_model_dataset}_dataset.jsonl" + elif remake_dataset_if_not_found: + logger.warning( + f"Generating dataset for {solver_version}! Ideally the solver should be using temperature=0 when creating the dataset, " + "otherwise generated dataset will be of a slightly different distribution" + ) + create_dataset(solver, recorder, solver_dataset_path, all_datasets_path) + return solver_dataset_path + else: + logger.warning( + f"Dataset for {solver_version} wasn't found! Using the dataset for {default_model_dataset} instead." + ) + return f"cant_do_that_anymore/{default_model_dataset}_dataset.jsonl" + + +def create_dataset( + solver: Solver, recorder: RecorderBase, solver_dataset_path: str, all_datasets_path: str +): + threads = int(os.environ.get("EVALS_THREADS", "10")) + + special_moves_dataset = load_sequence_from_jsonl( + os.path.join(all_datasets_path, "special_moves_dataset.jsonl") + ) + solver_moves_dataset = get_solver_predictions( + solver, + recorder, + special_moves_dataset, + n_threads=threads, + task_description=TASK_DESCRIPTION, + ) + dump_sequence_to_jsonl(solver_moves_dataset, solver_dataset_path) + + +def get_diagonal_dataset_path( + registry_path: str, +) -> str: + return os.path.join(registry_path, "cant_do_that_anymore/diagonal_moves_dataset.jsonl") + + +def get_binary_avg(metrics: dict, key: str) -> float: + positive_examples = [i for i in metrics if i[key]] + avg = len(positive_examples) / len(metrics) + return avg diff --git a/evals/elsuite/error_recovery/README.md b/evals/elsuite/error_recovery/README.md new file mode 100644 index 0000000000..78f6ceb726 --- /dev/null +++ b/evals/elsuite/error_recovery/README.md @@ -0,0 +1,96 @@ +# Error Recovery + +This eval evaluates how well LMs can recover from having made reasoning errors while solving a problem. + +## Usage + +Run with: + +``` +oaieval error-recovery +``` + +We suggest using `error_recovery/gpt-4-0613` or `error_recovery/gpt-3.5-turbo-0613` as default choices for `` + +For more examples of running this eval, see `scripts/run_experiments.sh` + +## Dataset + +The dataset consists of 1436 problems from [BIG-Bench Mistake](https://github.com/WHGTyen/BIG-Bench-Mistake) which have reasoning traces with mistakes in them, and incorrect answers. + +## Evaluation Process + +For each example, we run the solver on three variations of the problem: + +1) No Reasoning (NR) variation - Give the solver the question and ask it to generate reasoning and an answer. Example below: +``` +[SYSTEM] +Solve the given problem, writing your reasoning along the way. + +[USER] +Solve this problem for x: x = ((3 + 3) * 5) + 2 +``` + + +2) Correct Reasoning (CR) variation - Give the solver the question and some correct reasoning steps and have it keep generating reasoning and an answer. Example below: + +``` +[SYSTEM] +Solve the given problem, writing your reasoning along the way. + +[USER] +Solve this problem for x: x = ((3 + 3) * 5) + 2 + +[ASSISTANT] +3 + 3 = 6 +``` + +3) Incorrect Reasoning (IR) variation - Give the solver the question, the same correct reasoning steps as before, and an incorrect reasoning step and have it keep generating reasoning and an answer. + +``` +[SYSTEM] +Solve the given problem, writing your reasoning along the way. + +[USER] +Solve this problem for x: x = ((3 + 3) * 5) + 2 + +[ASSISTANT] +3 + 3 = 6 +6 * 5 = 35 +``` + + +## Metrics + +The below are the key metrics of this eval: + +| Metric | Interpretation | +| --- | --- | +| `NR_correct_rate` | The accuracy of the solver in the NR variation. +| `CR_correct_rate` | The accuracy of the solver in the CR variation. +| `IR_correct_rate` | The accuracy of the solver in the IR variation. + +## Variants + +| Variant | Notes | +| --- | --- | +| Default: `error-recovery.main` | Default setting. Includes all 1436 examples. | +| `error-recovery.medium` | A balanced subset of 250 examples, 50 from each task. | +| `error-recovery.small` | A balanced subset of 10 examples, 2 from each task (primarily for quick tests). | + +## Token Usage Estimates + +Below is a rough estimate of the total number of tokens consumed by the default variant: + +| Solver | Tokens | +| --- | --- | +| error_recovery/gpt-3.5-turbo-0613 | 4,750,781 | +| error_recovery/gpt-4-0613 | 4,386,440 | + +## Version History + +- v0: Initial version released + +## Contribution statement + +Eval design, implementation, results evaluation, and report were primarily done by Ian McKenzie based on an initial version by Oliver Jaffe. Work done under the guidance of (alphabetically by last-name) Steven Adler, James Aung, and Chan Jun Shern, who provided research input, report revisions, and project management support. diff --git a/evals/elsuite/error_recovery/defaults.py b/evals/elsuite/error_recovery/defaults.py new file mode 100644 index 0000000000..ee5560a08d --- /dev/null +++ b/evals/elsuite/error_recovery/defaults.py @@ -0,0 +1,15 @@ +DEFAULT_TASK_DESCRIPTION = "Solve the given problem, writing your reasoning along the way." + +DEFAULT_MISTAKE_MESSAGE = "There might be a mistake in your reasoning." + +DEFAULT_FINAL_ANSWER_MESSAGE = ( + "Given this reasoning, write your final answer. Only write your final answer, and nothing else." +) + +TASK_SPECIFIC_EXTRACTION_INFO = { + "dyck_languages": "\n\nAnswer with just the end of the sequence, separated by spaces. Do not repeat the part of the sequence given in the question. Only write the sequence of symbols, nothing else.", + "logical_deduction": "\n\nAnswer with the selected single letter indicating your answer, wrapped with parentheses. Do not write anything else.", + "multistep_arithmetic": "\n\nAnswer with a single number.", + "tracking_shuffled_objects": "\n\nAnswer with the selected single letter indicating your answer, wrapped with parentheses. Do not write anything else.", + "word_sorting": "\n\nAnswer with the sorted words, each lower case and separated by a single space.", +} diff --git a/evals/elsuite/error_recovery/eval.py b/evals/elsuite/error_recovery/eval.py new file mode 100644 index 0000000000..89512179fe --- /dev/null +++ b/evals/elsuite/error_recovery/eval.py @@ -0,0 +1,284 @@ +import copy +import random +from dataclasses import dataclass +from typing import Any, List, Literal, Optional, Sequence + +import evals +import evals.metrics +import evals.record +from evals.api import CompletionFn +from evals.elsuite.error_recovery.defaults import ( + DEFAULT_FINAL_ANSWER_MESSAGE, + DEFAULT_MISTAKE_MESSAGE, + DEFAULT_TASK_DESCRIPTION, + TASK_SPECIFIC_EXTRACTION_INFO, +) +from evals.eval import SolverEval +from evals.solvers.solver import Solver +from evals.task_state import Message, TaskState + +# possible Mistake NOTIFiciation POSitions +MistakeNotifPos = Literal["immediate", "end"] + + +@dataclass +class Sample: + question: str + correct_steps: Sequence[str] + incorrect_step: str + target: Any + task: str + num_ground_truth_steps: int + mistake_index: int + + +class ErrorRecovery(SolverEval): + def __init__( + self, + completion_fns: Sequence[CompletionFn], + samples_jsonl: str, + n_samples: Optional[int] = None, + mistake_notification_position: Optional[MistakeNotifPos] = None, + mistake_notification_for_ir_only: bool = False, + mark_as_own_reasoning: bool = True, + final_answer_prompt_role: str = "system", + *args, + **kwargs, + ): + """Evaluate a solver on the error recovery task. + + Args: + completion_fns: The completion functions to evaluate. (should be a single solver) + samples_jsonl: The relative path to the samples jsonl file in evals/registry/data. + n_samples: The number of samples to use. If None, use all samples. + mistake_notification_position: The position of the mistake + notification. Options are "immediate" for right after the provided + reasoning, or "end" for right after the model-generated reasoning. + If None, no mistake notification is added. + mistake_notification_for_ir_only: Whether to only add the mistake notification + for the incorrect reasoning case. If True, the mistake notification is + added for the incorrect reasoning case, and not for the correct reasoning + or no reasoning cases. + mark_as_own_reasoning: Whether to include the sample reasoning as an + 'assistant' or 'user' message. + final_answer_prompt_role: The role to use for the final answer prompt. Should + be either "system" or "user". + """ + super().__init__( + completion_fns=completion_fns, samples_jsonl=samples_jsonl, *args, **kwargs + ) + + self.n_samples = n_samples + self.mistake_notif_pos: Optional[MistakeNotifPos] = mistake_notification_position + self.mistake_notif_ir_only = mistake_notification_for_ir_only + + # there are some issues with passing bools in from extra_eval_params + assert isinstance(mark_as_own_reasoning, bool) + self.mark_as_own_reasoning = mark_as_own_reasoning + + self.final_answer_prompt_role = final_answer_prompt_role + assert self.final_answer_prompt_role in ["system", "user"] + + def eval_sample(self, solver: Solver, sample: Sample, rng: random.Random, extra_logging=None): + task = sample.task + + # Get the baseline with no provided reasoning + nr_task_state = self._get_no_reasoning_task_state(sample) + # only "end" makes sense for 'no reasoning' + nr_notif_pos = "end" if self.mistake_notif_pos == "end" else None + if self.mistake_notif_ir_only: + nr_notif_pos = None + + nr_answer = self._get_answer( + solver=solver, + task_state=nr_task_state, + sample=sample, + mistake_notif_pos=nr_notif_pos, + ) + + # Run with correct reasoning + cr_task_state = self._get_correct_reasoning_task_state(sample) + cr_notif_pos = self.mistake_notif_pos + if self.mistake_notif_ir_only: + cr_notif_pos = None + + cr_answer = self._get_answer( + solver=solver, + task_state=cr_task_state, + sample=sample, + mistake_notif_pos=cr_notif_pos, + ) + + # Run with incorrect reasoning + ir_task_state = self._get_incorrect_reasoning_task_state(sample) + ir_notif_pos = self.mistake_notif_pos + + ir_answer = self._get_answer( + solver=solver, + task_state=ir_task_state, + sample=sample, + mistake_notif_pos=ir_notif_pos, + ) + + assert len(sample.correct_steps) == sample.mistake_index + + metrics = { + "task": task, + "num_ground_truth_steps": sample.num_ground_truth_steps, + "mistake_index": sample.mistake_index, + "target": str(sample.target), # ground truth answer + "mistake_notification_position": self.mistake_notif_pos, + "mistake_notification_for_ir_only": self.mistake_notif_ir_only, + "NR_sampled": nr_answer, + "CR_sampled": cr_answer, + "IR_sampled": ir_answer, + "NR_correct": nr_answer == str(sample.target), + "CR_correct": cr_answer == str(sample.target), + "IR_correct": ir_answer == str(sample.target), + } + evals.record.record_metrics(**metrics) + + def _get_no_reasoning_task_state(self, sample: Sample) -> TaskState: + task_description = DEFAULT_TASK_DESCRIPTION + no_reasoning_messages = [ + Message(role="user", content=sample.question), + ] + no_reasoning_task_state = TaskState( + task_description=task_description, + messages=no_reasoning_messages, + ) + return no_reasoning_task_state + + def _get_correct_reasoning_task_state(self, sample: Sample) -> TaskState: + task_description = DEFAULT_TASK_DESCRIPTION + correct_steps = "\n".join(sample.correct_steps) + reasoning_role = "assistant" if self.mark_as_own_reasoning else "user" + correct_reasoning_messages = [ + Message(role="user", content=sample.question), + Message(role=reasoning_role, content=correct_steps), + ] + correct_reasoning_task_state = TaskState( + task_description=task_description, + messages=correct_reasoning_messages, + ) + return correct_reasoning_task_state + + def _get_incorrect_reasoning_task_state( + self, + sample: Sample, + ) -> TaskState: + task_description = DEFAULT_TASK_DESCRIPTION + correct_steps = "\n".join(sample.correct_steps) + steps_with_incorrect_reasoning = f"{correct_steps}\n{sample.incorrect_step}" + reasoning_role = "assistant" if self.mark_as_own_reasoning else "user" + incorrect_reasoning_messages = [ + Message(role="user", content=sample.question), + Message(role=reasoning_role, content=steps_with_incorrect_reasoning), + ] + + incorrect_reasoning_task_state = TaskState( + task_description=task_description, + messages=incorrect_reasoning_messages, + ) + return incorrect_reasoning_task_state + + def _get_answer( + self, + solver: Solver, + task_state: TaskState, + sample: Sample, + mistake_notif_pos: Optional[MistakeNotifPos], + ) -> str: + """Get a final answer from the solver for a given sample. + + Args: + solver: The solver to use. + task_state: The task state to use. + sample: The Sample being evaluated (relevant for answer extraction). + mistake_notification_position: The position of the mistake notification. + Options are "immediate" for right after the provided reasoning, or "end" for right + after the model-generated reasoning. If None, no mistake notification is added. + + TODO (ian): Work out whether to add mistake notification to 'no reasoning' baseline + """ + mistake_message = Message("user", DEFAULT_MISTAKE_MESSAGE) + if mistake_notif_pos == "immediate": + task_state.messages.append(mistake_message) + + output = solver(task_state=task_state).output + task_state.messages.append(Message("assistant", output)) + + # run solver again if mistake notification is at the end + if mistake_notif_pos == "end": + task_state.messages.append(mistake_message) + output = solver(task_state=task_state).output + task_state.messages.append(Message("assistant", output)) + + answer = self._extract_final_answer(solver=solver, task_state=task_state, sample=sample) + return answer + + def run(self, recorder: evals.record.Recorder): + samples = self.get_samples() + + self.eval_all_samples(recorder, samples) + metrics = recorder.get_metrics() + + NR_correct_rate = len([i for i in metrics if i["NR_correct"]]) / len(metrics) + CR_correct_rate = len([i for i in metrics if i["CR_correct"]]) / len(metrics) + IR_correct_rate = len([i for i in metrics if i["IR_correct"]]) / len(metrics) + + results = { + "NR_correct_rate": NR_correct_rate, + "CR_correct_rate": CR_correct_rate, + "IR_correct_rate": IR_correct_rate, + } + + # Split results per type of task + all_tasks = set([i["task"] for i in metrics]) + for task in all_tasks: + filtered_metrics = [i for i in metrics if i["task"] == task] + NR_correct_rate = len([i for i in filtered_metrics if i["NR_correct"]]) / len( + filtered_metrics + ) + CR_correct_rate = len([i for i in filtered_metrics if i["CR_correct"]]) / len( + filtered_metrics + ) + IR_correct_rate = len([i for i in filtered_metrics if i["IR_correct"]]) / len( + filtered_metrics + ) + + # we use hyphens in the task name so they can be extracted by splitting on underscores + task_string = task.replace("_", "-") + results.update( + { + f"task_{task_string}_NR_correct_rate": NR_correct_rate, + f"task_{task_string}_CR_correct_rate": CR_correct_rate, + f"task_{task_string}_IR_correct_rate": IR_correct_rate, + } + ) + + return results + + def _extract_final_answer(self, solver: Solver, task_state: TaskState, sample: Sample): + """Extract the final answer from the solver output using the same solver.""" + task_state = copy.deepcopy(task_state) + + task_specific_info = TASK_SPECIFIC_EXTRACTION_INFO[sample.task] + final_answer_prompt = DEFAULT_FINAL_ANSWER_MESSAGE + task_specific_info + + task_state.messages.append( + Message(role=self.final_answer_prompt_role, content=final_answer_prompt) + ) + answer = solver(task_state=task_state).output + + return answer + + def get_samples(self) -> List[Sample]: + samples = super().get_samples() + + if self.n_samples is not None: + assert ( + len(samples) >= self.n_samples + ), f"Can't get {self.n_samples} samples from a dataset with {len(samples)} samples" + samples = samples[: self.n_samples] + return [Sample(**sample_dict) for sample_dict in samples] diff --git a/evals/elsuite/error_recovery/scripts/dataset_creation.py b/evals/elsuite/error_recovery/scripts/dataset_creation.py new file mode 100644 index 0000000000..c6c14b2417 --- /dev/null +++ b/evals/elsuite/error_recovery/scripts/dataset_creation.py @@ -0,0 +1,156 @@ +import subprocess +from pathlib import Path + +import matplotlib.pyplot as plt +import pandas as pd + +TASK_PREFIX = { + "dyck_languages": ( + "Given the following sequence of opening and closing brackets, " + "provide the minimal sequence of additional brackets that would " + "balance the original sequence:\n" + ), + "logical_deduction": "", + "multistep_arithmetic": "", + "tracking_shuffled_objects": "", + "word_sorting": "Sort the following list of words alphabetically:\n", +} + + +def main(): + data = clone_and_load_data() + # plot_hist(data) + pos_data = create_positive_examples(data) + # don't use examples where last step is mistake + pos_data = pos_data[pos_data["mistake_index"] < pos_data["num_steps"] - 1] + + # only save a subset of the columns + pos_data = pos_data[ + ["input", "correct_steps", "incorrect_step", "mistake_index", "num_steps", "target", "task"] + ] + pos_data.rename( + columns={ + "input": "question", + "num_steps": "num_ground_truth_steps", + }, + inplace=True, + ) + + # save data + save_path = Path("evals/registry/data/error_recovery/main.jsonl") + pos_data.to_json(save_path, orient="records", lines=True) + + small_save_path = Path("evals/registry/data/error_recovery/small.jsonl") + # get small dataset with two examples from each task + small_data = create_data_subset(pos_data, examples_per_task=2) + small_data.to_json(small_save_path, orient="records", lines=True) + + medium_save_path = Path("evals/registry/data/error_recovery/medium.jsonl") + # get medium dataset with 50 examples from each task + medium_data = create_data_subset(pos_data, examples_per_task=50) + medium_data.to_json(medium_save_path, orient="records", lines=True) + + +def create_data_subset(data: pd.DataFrame, examples_per_task: int) -> pd.DataFrame: + # get small dataset with a subset of examples from each task + small_data = pd.DataFrame() + for task in data["task"].unique(): + task_data = data[data["task"] == task] + task_subset = task_data[:examples_per_task] + if len(task_subset) < examples_per_task: + raise ValueError( + f"Task {task} has only {len(task_subset)} examples, less than {examples_per_task}" + ) + small_data = pd.concat((small_data, task_subset)) + return small_data + + +def create_positive_examples(data: pd.DataFrame) -> pd.DataFrame: + has_incorrect_reasoning = ~data["mistake_index"].isnull() + has_incorrect_answer = data["target"] != data["answer"] + positive_condition = has_incorrect_reasoning & has_incorrect_answer + + positive_data = data.copy() + positive_data = positive_data[positive_condition].reset_index() + positive_data["label"] = "positive" + positive_data["correct_steps"] = positive_data.apply( + lambda row: row["steps"][: int(row["mistake_index"])], axis=1 + ) + positive_data["incorrect_step"] = positive_data.apply( + lambda row: row["steps"][int(row["mistake_index"])], axis=1 + ) + return positive_data + + +def create_negative_examples(data: pd.DataFrame) -> pd.DataFrame: + """Create a dataset of examples with correct reasoning and answer. + + The 'negative' naming is a bit misleading, but these are the examples + we don't use. + TODO (ian): think about renaming + """ + has_correct_reasoning = data["mistake_index"].isnull() + has_correct_answer = data["target"] == data["answer"] + negative_condition = has_correct_reasoning & has_correct_answer + negative_data = data.copy() + negative_data = negative_data[negative_condition].reset_index() + negative_data["label"] = "negative" + negative_data["correct_steps"] = negative_data["steps"] + negative_data["incorrect_step"] = "" + return negative_data + + +def clone_and_load_data(): + clone_dir = Path("/tmp/BIG-Bench-Mistake") + maybe_clone_repo(clone_dir) + + data = pd.DataFrame() + for jsonl_file in clone_dir.glob("*.jsonl"): + file_data = pd.read_json(jsonl_file, lines=True) + + # Manually append task description to datasets missing one + task = jsonl_file.stem + prefix = TASK_PREFIX[task] + file_data["input"] = prefix + file_data["input"] + file_data["task"] = task + + data = pd.concat((data, file_data)) + + data["num_steps"] = data["steps"].apply(lambda x: len(x)) + return data + + +def maybe_clone_repo(clone_dir): + if not clone_dir.exists(): + subprocess.run( + ["git", "clone", "https://github.com/WHGTyen/BIG-Bench-Mistake.git", str(clone_dir)] + ) + + +def plot_hist(data): + data["num_steps"].hist(bins=max(data["num_steps"])) + plt.show() + + +def print_example(): + data = clone_and_load_data() + # printing some examples + subset_data = create_positive_examples(data) + # subset_data = create_negative_examples(data) + # # print one negative object swapping example + # neg_example = neg_data[neg_data["task"] == "tracking_shuffled_objects"].iloc[0] + # # print one negative dyck example + # neg_example = neg_data[neg_data["task"] == "dyck_languages"].iloc[0] + # neg_example = neg_data[neg_data["task"] == "logical_deduction"].iloc[0] + example = subset_data[subset_data["task"] == "multistep_arithmetic"].iloc[1] + print(f"INPUT ======\n{example['input']}") + steps = "\n".join(example["steps"]) + print(f"STEPS ======\n{steps}") + print(f"MISTAKE INDEX ======\n{example['mistake_index']}") + print(f"ANSWER ======\n{example['answer']}") + print(f"TARGET ======\n{example['target']}") + print("========") + + +if __name__ == "__main__": + main() diff --git a/evals/elsuite/error_recovery/scripts/make_plots.py b/evals/elsuite/error_recovery/scripts/make_plots.py new file mode 100644 index 0000000000..0d2dcfaa43 --- /dev/null +++ b/evals/elsuite/error_recovery/scripts/make_plots.py @@ -0,0 +1,597 @@ +import argparse +import os +from pathlib import Path +from typing import Optional + +import numpy as np +import pandas as pd +from matplotlib import pyplot as plt + +from evals.utils import log_utils + +# MODEL_NAMES = { +# "error_recovery/gpt-4-0613": "GPT-4", +# "generation/hhh/gpt-4-base": "GPT-4 Base", +# "error_recovery/gpt-3.5-turbo-0613": "GPT-3.5", +# # "gpt-4-base": "gpt-4-base", +# } +# using model checkpoint names +MODEL_NAMES = { + "error_recovery/gpt-4-0613": "gpt-4-0613", + "generation/hhh/gpt-4-base": "gpt-4-base", + "error_recovery/gpt-3.5-turbo-0613": "gpt-3.5-turbo-0613", + # "generation/direct/llama-2-13b-chat": "llama-2-13b-chat", + "generation/direct/llama-2-70b-chat": "llama-2-70b-chat", + "generation/direct/mixtral-8x7b-instruct": "mixtral-8x7b-instruct", + "generation/direct/gemini-pro": "gemini-pro-1.0", +} + +MODEL_COLOR_MAP = { + "error_recovery/gpt-4-0613": "purple", + "generation/hhh/gpt-4-base": "plum", + "error_recovery/gpt-3.5-turbo-0613": "g", + # "generation/direct/llama-2-13b-chat": "wheat", + "generation/direct/llama-2-70b-chat": "orange", + "generation/direct/mixtral-8x7b-instruct": "red", + "generation/direct/gemini-pro": "cornflowerblue", +} +VARIATION_NAMES = { + "nr_name": "From Scratch", + "cr_name": "Correct Basis", + "ir_name": "Incorrect Basis", +} + +VARIATION_COLOR_MAP = { + "nr_name": "blue", + "cr_name": "green", + "ir_name": "red", +} + +TASK_NAMES = { + "word_sorting": "Word Sorting", + "tracking_shuffled_objects": "Tracking Shuffled Objects", + "logical_deduction": "Logical Deduction", + "multistep_arithmetic": "Multi-Step Arithmetic", + "dyck_languages": "Dyck Languages", +} + + +def maybe_show(fig): + if DISPLAY: + fig.show() + plt.close(fig) + + +def extract_results(datadir: Path) -> pd.DataFrame: + df_rows = [] + for path, results in log_utils.get_final_results_from_dir(datadir).items(): + spec = log_utils.extract_spec(path) + model = spec["completion_fns"][0] + base_eval = spec["base_eval"] + df_rows.append( + { + "model": model, + "base_eval": base_eval, + **results, + } + ) + df = pd.DataFrame(df_rows) + return df + + +def extract_metrics(datadir: Path) -> pd.DataFrame: + df_rows = [] + for path, results in sorted(list(log_utils.get_final_results_from_dir(datadir).items())): + spec = log_utils.extract_spec(path) + solver = spec["completion_fns"][0] + for res in log_utils.extract_individual_results(path): + df_rows.append( + { + "solver": solver, + **res, + } + ) + df = pd.DataFrame(df_rows) + # Sort rows + # print(df.columns) + df.sort_values(by=["solver", "task"], inplace=True) + return df + + +def get_all_tasks(results_df: pd.DataFrame) -> list[str]: + # Find all types of tasks + all_tasks = [] + for i in results_df.columns: + if i.startswith("task_") and i.endswith("_CR_correct_rate"): + all_tasks.append(i) + + # Make ordering consistent + all_tasks.sort() + return all_tasks + + +def get_all_tasks_renamed(results_df: pd.DataFrame) -> list[str]: + all_tasks = get_all_tasks(results_df) + all_tasks_renamed = [i.split("task_")[1].split("_CR_correct_rate")[0] for i in all_tasks] + # replace hyphens with underscores + all_tasks_renamed = [i.replace("-", "_") for i in all_tasks_renamed] + return all_tasks_renamed + + +def get_unique_models(results_df: pd.DataFrame) -> list[str]: + models = results_df["model"].to_list() + # TODO: work out how to order a variable set of models + if set(models) == set(MODEL_NAMES.keys()): + unique_models = list(MODEL_NAMES.keys()) + else: + unique_models = sorted(list(set(models)), reverse=True) + return unique_models + + +def get_cleaned_model_name(model: str) -> str: + return model.replace("/", "_") + + +def corrects_to_accuracy_and_sem(corrects: pd.Series): + accuracy = corrects.mean() + sem = corrects.sem() + return accuracy, sem + + +def annotate_axes(ax, errors: Optional[pd.DataFrame]): + """Annotate each bar in the plot with its value""" + ABOVE_OFFSET = 0.01 + BELOW_OFFSET = 0.1 + if errors is not None: + # This gets it into a shape to match the order of the patch objects. + # I don't have a principled reason to transpose, this is just what works. + error_values = errors.to_numpy().T.flatten() + + for i, p in enumerate(ax.patches): + # patch objects aren't typed correctly + p_height = p.get_height() # type: ignore + p_x = p.get_x() # type: ignore + p_width = p.get_width() # type: ignore + # Calculate the label position + x = p_x + p_width / 2 + if errors is not None: + error = error_values[i] + else: + error = 0 + + if p_height > 0: + y = p_height + error + ABOVE_OFFSET + else: + y = p_height - error - BELOW_OFFSET + + # Annotate the bar with its value + # ax.annotate(f"{p_height:.2f}\n±{error:.2f}", (x, y), ha="center", va="bottom") + ax.annotate(f"{p_height:.2f}", (x, y), ha="center", va="bottom") + + +def corrects_to_performance_loss_and_error(CR_corrects: pd.Series, IR_corrects: pd.Series): + CR_correct_rate = CR_corrects.mean() + IR_correct_rate = IR_corrects.mean() + + performance_recovered = IR_correct_rate / CR_correct_rate + performance_loss = 1 - performance_recovered + # propagate error from CR_corrects and IR_corrects to performance_loss + CR_correct_rate_sem = CR_corrects.sem() + IR_correct_rate_sem = IR_corrects.sem() + assert isinstance(CR_correct_rate_sem, float) + assert isinstance(IR_correct_rate_sem, float) + # using the formula for error propagation for a ratio from + # https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Example_formulae + # (assuming errors in CR and IR are independent). + # NOTE: the 1 in performance_loss is a constant, + # so doesn't affect the uncertainty bounds on the ratio. + CR_term = (CR_correct_rate_sem / CR_correct_rate) ** 2 + IR_term = (IR_correct_rate_sem / IR_correct_rate) ** 2 + performance_loss_error = abs(performance_recovered) * ((CR_term + IR_term) ** 0.5) + print(f"Performance loss: {performance_loss:.2f} ± {performance_loss_error:.2f}") + return performance_loss, performance_loss_error + + +def accuracy_by_task(metrics_df, results_df: pd.DataFrame, out_dir: Path): + all_tasks = get_all_tasks(results_df) + unique_models = get_unique_models(results_df) + all_tasks_renamed = get_all_tasks_renamed(results_df) + + # Plot results separately for each model + for model in unique_models: + plot_accuracy_by_task(model, metrics_df, all_tasks, all_tasks_renamed, out_dir) + + +def accuracy_by_model_dfs(metrics_df, results_df: pd.DataFrame): + unique_models = get_unique_models(results_df) + accuracies = {} + sems = {} + for model in unique_models: + pass + # for all tasks + model_mask = metrics_df.solver == model + model_CR_corrects = metrics_df[model_mask]["CR_correct"] + model_IR_corrects = metrics_df[model_mask]["IR_correct"] + model_NR_corrects = metrics_df[model_mask]["NR_correct"] + + model_CR_accuracy, model_CR_sem = corrects_to_accuracy_and_sem(model_CR_corrects) + model_IR_accuracy, model_IR_sem = corrects_to_accuracy_and_sem(model_IR_corrects) + model_NR_accuracy, model_NR_sem = corrects_to_accuracy_and_sem(model_NR_corrects) + + pretty_model_name = MODEL_NAMES[model] + sems[pretty_model_name] = { + "nr_name": model_NR_sem, + "cr_name": model_CR_sem, + "ir_name": model_IR_sem, + } + accuracies[pretty_model_name] = { + "nr_name": model_NR_accuracy, + "cr_name": model_CR_accuracy, + "ir_name": model_IR_accuracy, + } + + order = ["nr_name", "cr_name", "ir_name"] + plot_df = pd.DataFrame(accuracies) + plot_df = plot_df.reindex(order) + sems_df = pd.DataFrame(sems) + sems_df = sems_df.reindex(order) + return plot_df, sems_df + + +def accuracy_by_model(metrics_df, results_df: pd.DataFrame, out_dir: Path): + unique_models = get_unique_models(results_df) + plot_df, sems_df = accuracy_by_model_dfs(metrics_df, results_df) + + fig, ax = plt.subplots(figsize=(12, 6), constrained_layout=True) + colors = [MODEL_COLOR_MAP[model] for model in unique_models] + plot_df.index = list(VARIATION_NAMES.values()) + sems_df.index = list(VARIATION_NAMES.values()) + ax = plot_df.plot.bar( + rot=0, + yerr=sems_df, + capsize=4, + ax=ax, + width=0.8, + color=colors, + ) + annotate_axes(ax, sems_df) + ax.set_ylim(top=1.0) + ax.set_xlabel("Reasoning variations") + ax.set_ylabel("Accuracy") + ax.set_title("Accuracy for each variation (higher is better)") + + outpath = os.path.join(out_dir, "accuracy_by_model.png") + fig.savefig(outpath) + maybe_show(fig) + + +def accuracy_by_model_and_reasoning( + own_metrics_df: pd.DataFrame, + own_results_df: pd.DataFrame, + other_metrics_df: pd.DataFrame, + other_results_df: pd.DataFrame, + out_dir: Path, +): + own_plot_df, own_sems_df = accuracy_by_model_dfs(own_metrics_df, own_results_df) + other_plot_df, other_sems_df = accuracy_by_model_dfs(other_metrics_df, other_results_df) + # drop the no reasoning baseline + own_plot_df = own_plot_df.drop("nr_name", axis=0) + own_sems_df = own_sems_df.drop("nr_name", axis=0) + other_plot_df = other_plot_df.drop("nr_name", axis=0) + other_sems_df = other_sems_df.drop("nr_name", axis=0) + + own_plot_df = own_plot_df.T + own_sems_df = own_sems_df.T + other_plot_df = other_plot_df.T + other_sems_df = other_sems_df.T + models = own_plot_df.index # e.g., ["No reasoning (baseline)", "Correct reasoning", ...] + n_models = len(models) + bar_width = 0.35 # width of the bars + n_variations = len(own_plot_df.columns) + assert n_variations == len(other_plot_df.columns) + group_width = 0.8 # Total width for one group of bars + bar_width = group_width / (n_variations * 2) # Width of one bar + + # Create figure and axis + fig, ax = plt.subplots(figsize=(12, 8), constrained_layout=True) + + # Set position of bar on X axis + ind = np.arange(n_models) # the x locations for the groups + + colors = [VARIATION_COLOR_MAP[variation] for variation in own_plot_df.columns] + VARIATION_OFFSET = 0.03 + for i, variation in enumerate(own_plot_df.columns): + # Position of bars for this model + # bars for a given model are grouped together, and then within that group, the bars for each variation are grouped + r1 = ind + i * VARIATION_OFFSET + i * (n_variations * bar_width) + r2 = [x + bar_width for x in r1] + + ax.bar( + r1, + own_plot_df[variation], + width=bar_width, + yerr=own_sems_df[variation], + capsize=5, + label=f"{VARIATION_NAMES[variation]} ('assistant' message)", + color=colors[i], + # add outline to bars + edgecolor="black", + ) + ax.bar( + r2, + other_plot_df[variation], + width=bar_width, + yerr=other_sems_df[variation], + capsize=5, + label=f"{VARIATION_NAMES[variation]} ('user' message)", + hatch="//", + color=colors[i], + edgecolor="black", + ) + + for j, model in enumerate(models): + x_own = r1[j] + x_other = r2[j] + y1 = own_plot_df.loc[model, variation] + y2 = other_plot_df.loc[model, variation] + y1_err = own_sems_df.loc[model, variation] + y2_err = other_sems_df.loc[model, variation] + ax.text(x_own, y1 + y1_err, f"{y1:.2f}", ha="center", va="bottom") + ax.text(x_other, y2 + y2_err, f"{y2:.2f}", ha="center", va="bottom") + + # Add xticks on the middle of the group bars + xtick_positions = ind + bar_width * n_variations + (VARIATION_OFFSET - bar_width) / 2 + ax.set_xticks(xtick_positions) + ax.set_xticklabels(models) + + # Create legend & Show graphic + ax.set_xlabel("Model") + ax.set_ylabel("Accuracy") + ax.set_ylim(top=1.0) + ax.legend() + ax.set_title("Accuracy for each variation (higher is better)") + outpath = os.path.join(out_dir, "accuracy_by_category_and_reasoning.png") + fig.savefig(outpath) + maybe_show(fig) + + +def plot_accuracy_by_steps_all(metrics_df, results_df, out_dir): + """ + Create plots of accuracy of: + - num_steps - mistake_index + - mistake_index / num_steps + """ + get_all_tasks(results_df) + all_tasks_renamed = get_all_tasks_renamed(results_df) + all_models = get_unique_models(results_df) + # one plot per task, one subplot per model + for task in all_tasks_renamed: + fig, axs = plt.subplots( + 1, len(all_models), figsize=(15, 6), constrained_layout=True, squeeze=False + ) + axs = axs.flatten() + for ax, model in zip(axs, all_models): + task_model_df = metrics_df[(metrics_df.solver == model) & (metrics_df.task == task)] + plot_accuracy_by_steps(task_model_df, task, model, ax) + # only put legend on last plot + final_ax = axs[-1] + final_ax.legend(loc="upper center") + outpath = os.path.join(out_dir, f"results-split-by-steps_{task}.png") + fig.suptitle(f"Accuracy by steps for {TASK_NAMES[task]} (higher is better)") + fig.savefig(outpath) + maybe_show(fig) + + +def plot_accuracy_by_steps(df, task, model, ax): + df["steps_diff"] = df["num_ground_truth_steps"] - df["mistake_index"] + + # due to the way pandas works, we have to group, then filter, then regroup + grouped_df = df.groupby("steps_diff") + + MIN_SAMPLES = 10 + filtered_groups = grouped_df.filter(lambda x: len(x) >= MIN_SAMPLES) + + # Now, re-group the filtered DataFrame by 'steps_diff' again and calculate the mean + plot_df = filtered_groups.groupby("steps_diff")[ + ["NR_correct", "CR_correct", "IR_correct"] + ].mean() + colors = [VARIATION_COLOR_MAP[variation] for variation in VARIATION_NAMES.keys()] + + # change the names of the columns to be more readable before plotting + plot_df.columns = list(VARIATION_NAMES.values()) + # now plot the three accuracies against steps_diff + assert isinstance(plot_df, pd.DataFrame) + ax = plot_df.plot(color=colors, ax=ax, legend=False) + ax.set_xlabel("Steps beyond mistake") + ax.set_ylabel("Accuracy") + ax.set_ylim(0, 1.1) + # ax.set_title(f"{MODEL_NAMES[model]} | {TASK_NAMES[task]} (higher is better)") + ax.set_title(f"{MODEL_NAMES[model]}") + # plt.tight_layout() + return ax + + +def plot_accuracy_by_task(model, metrics_df, all_tasks, all_tasks_renamed, out_dir): + all_tasks_pretty = [TASK_NAMES[i] for i in all_tasks_renamed] + accuracies = {"nr_name": [], "cr_name": [], "ir_name": []} + all_sems = [] + # for all tasks + model_mask = metrics_df.solver == model + + # and split by task type + for task in all_tasks_renamed: + + task_mask = metrics_df.task == task + CR_corrects = metrics_df[model_mask & task_mask]["CR_correct"] + IR_corrects = metrics_df[model_mask & task_mask]["IR_correct"] + NR_corrects = metrics_df[model_mask & task_mask]["NR_correct"] + + CR_accuracy, CR_sem = corrects_to_accuracy_and_sem(CR_corrects) + IR_accuracy, IR_sem = corrects_to_accuracy_and_sem(IR_corrects) + NR_accuracy, NR_sem = corrects_to_accuracy_and_sem(NR_corrects) + + accuracies["nr_name"].append(NR_accuracy) + accuracies["cr_name"].append(CR_accuracy) + accuracies["ir_name"].append(IR_accuracy) + + sems = [NR_sem, CR_sem, IR_sem] + all_sems.append(sems) + + sems_df = pd.DataFrame( + all_sems, + index=all_tasks_pretty, + columns=["nr_name", "cr_name", "ir_name"], + ) + + plot_df = pd.DataFrame(accuracies, index=all_tasks_pretty) + + fig, ax = plt.subplots(figsize=(15, 6), constrained_layout=True) + colors = [VARIATION_COLOR_MAP[variation] for variation in plot_df.columns] + plot_df.columns = list(VARIATION_NAMES.values()) + ax = plot_df.plot.bar(rot=0, color=colors, yerr=sems_df, capsize=4, ax=ax, width=0.8) + annotate_axes(ax, sems_df) + + # Shrink current axis by 20% to make room for the legend + box = ax.get_position() + ax.set_position((box.x0, box.y0, box.width * 0.8, box.height)) + # Place the legend outside the plot + ax.legend(loc="center left", bbox_to_anchor=(1, 0.5)) + ax.set_ylim(top=1.1) + ax.set_xlabel("Task type") + ax.set_ylabel("Accuracy") + ax.set_title(f"{MODEL_NAMES[model]} (higher is better)") + outpath = os.path.join(out_dir, f"results-split-by-task_{get_cleaned_model_name(model)}.png") + fig.savefig(outpath) + maybe_show(fig) + + +def performance_loss_per_task(metrics_df: pd.DataFrame, results_df: pd.DataFrame, out_dir: Path): + # Plot performance lost for each model + unique_models = get_unique_models(results_df) + get_all_tasks(results_df) + all_tasks_renamed = get_all_tasks_renamed(results_df) + all_tasks_pretty = [TASK_NAMES[i] for i in all_tasks_renamed] + + all_metrics = {} + all_errors = {} + for model in unique_models: + metrics = [] + errors = [] + for task in all_tasks_renamed: + model_mask = metrics_df.solver == model + task_mask = metrics_df.task == task + CR_corrects = metrics_df[model_mask & task_mask]["CR_correct"] + IR_corrects = metrics_df[model_mask & task_mask]["IR_correct"] + + performance_loss, performance_loss_error = corrects_to_performance_loss_and_error( + CR_corrects, IR_corrects + ) + metrics.append(performance_loss) + errors.append(performance_loss_error) + + pretty_model_name = MODEL_NAMES[model] + all_metrics[pretty_model_name] = metrics + all_errors[pretty_model_name] = errors + + fig, ax = plt.subplots(figsize=(20, 6), constrained_layout=True) + plot_df = pd.DataFrame(all_metrics, index=all_tasks_pretty) + errs_df = pd.DataFrame(all_errors, index=all_tasks_pretty) + colors = [MODEL_COLOR_MAP[model] for model in unique_models] + ax = plot_df.plot.bar(rot=0.0, color=colors, ax=ax, width=0.8, yerr=errs_df, capsize=4) + annotate_axes(ax, errs_df) + # Shrink current axis by 20% to make room for the legend + box = ax.get_position() + ax.set_position((box.x0, box.y0, box.width * 0.8, box.height)) + ax.set_ylim(bottom=-1, top=1.1) + ax.legend() + ax.axhline(0, 0, 1, color="black", linestyle="-") + ax.set_title("Performance loss per task (lower is better)") + ax.set_xlabel("Task type") + ax.set_ylabel("Performance loss") + + outpath = os.path.join(out_dir, "results_split_by_model.png") + fig.savefig(outpath) + maybe_show(fig) + + +def performance_loss_per_model(metrics_df: pd.DataFrame, results_df: pd.DataFrame, out_dir: Path): + unique_models = get_unique_models(results_df) + + metrics = {} + errors = {} + for model in unique_models: + model_mask = metrics_df.solver == model + + CR_corrects = metrics_df[model_mask]["CR_correct"] + IR_corrects = metrics_df[model_mask]["IR_correct"] + + performance_loss, performance_loss_error = corrects_to_performance_loss_and_error( + CR_corrects, IR_corrects + ) + + pretty_model_name = MODEL_NAMES[model] + metrics[pretty_model_name] = performance_loss + errors[pretty_model_name] = performance_loss_error + + fig, ax = plt.subplots(figsize=(10, 6), constrained_layout=True) + plot_df = pd.DataFrame(metrics, index=[0]) + errs_df = pd.DataFrame(errors, index=[0]) + colors = [MODEL_COLOR_MAP[model] for model in unique_models] + ax = plot_df.plot.bar(rot=0, color=colors, ax=ax, width=0.8, yerr=errs_df, capsize=4) + annotate_axes(ax, errs_df) + # Shrink current axis by 20% to make room for the legend + box = ax.get_position() + ax.set_position((box.x0, box.y0, box.width * 0.8, box.height)) + # Place the legend outside the plot + ax.legend(loc="center left", bbox_to_anchor=(1, 0.5)) + ax.set_xticklabels([]) + ax.set_xticks([]) + ax.set_ylabel("Performance loss") + ax.set_ylim(top=1.1) + ax.set_title("Average performance loss per model (lower is better)") + outpath = os.path.join(out_dir, "headline_results.png") + fig.savefig(outpath) + maybe_show(fig) + + +def main(): + parser = argparse.ArgumentParser() + # DEBUG: hacking together own_reasoning and other_reasoning plots + parser.add_argument( + "--log_dir", + "-d", + type=str, + required=True, + help="Path to log dir with primary results (if supplementary_dir is provided, this is should be 'own' reasoning)", + ) + parser.add_argument( + "--supplementary_dir", + "-s", + type=str, + help="Optional supplementary log dir with 'other' reasoning results", + ) + parser.add_argument("--out_dir", "-o", type=str, required=True) + args = parser.parse_args() + log_dir = Path(args.log_dir) + out_dir = Path(args.out_dir) + out_dir.mkdir(exist_ok=True, parents=True) + + metrics_df = extract_metrics(log_dir) + results_df = extract_results(log_dir) + if args.supplementary_dir: + other_log_dir = Path(args.supplementary_dir) + other_metrics_df = extract_metrics(other_log_dir) + other_results_df = extract_results(other_log_dir) + accuracy_by_model_and_reasoning( + metrics_df, results_df, other_metrics_df, other_results_df, out_dir + ) + accuracy_by_task(metrics_df, results_df, out_dir) + accuracy_by_model(metrics_df, results_df, out_dir) + performance_loss_per_task(metrics_df, results_df, out_dir) + performance_loss_per_model(metrics_df, results_df, out_dir) + plot_accuracy_by_steps_all(metrics_df, results_df, out_dir) + + +if __name__ == "__main__": + DISPLAY = False + main() diff --git a/evals/elsuite/error_recovery/scripts/run_experiments.sh b/evals/elsuite/error_recovery/scripts/run_experiments.sh new file mode 100755 index 0000000000..36f51faad4 --- /dev/null +++ b/evals/elsuite/error_recovery/scripts/run_experiments.sh @@ -0,0 +1,44 @@ +#!/bin/bash +logdir=./logs +outdir=./outputs + +timestamp=$(date +%Y%m%d_%H%M%S) +logpathbase=$logdir/$timestamp +outpathbase=$outdir/$timestamp +SPLIT=main + +mkdir -p ${logpathbase} + +export EVALS_THREADS=250 +echo Running full experiments and logging to $logpathbase + +declare -a SOLVERS=( + error_recovery/gpt-3.5-turbo-0613 + error_recovery/gpt-4-0613 + generation/hhh/gpt-4-base +) + +# OWN REASONING VARIANT +for solver in "${SOLVERS[@]}" +do + log_name=${SPLIT}_${solver//\//-}_own-reasoning + + oaieval $solver error-recovery.$SPLIT \ + --extra_eval_params final_answer_prompt_role=system \ + --record_path "$logpathbase/$log_name.log" +done + +# OTHER REASONING VARIANT +for solver in "${SOLVERS[@]}" +do + log_name=${SPLIT}_${solver//\//-}_other-reasoning + + oaieval $solver error-recovery.$SPLIT.other-reasoning \ + --extra_eval_params final_answer_prompt_role=system \ + --record_path "$logpathbase/$log_name.log" +done + +echo Producing plots, outputs to $outpathbase + +mkdir -p ${outpathbase} +python make_plots.py --log_dir ${logpathbase} --out_dir $outpathbase diff --git a/evals/elsuite/function_deduction/README.md b/evals/elsuite/function_deduction/README.md new file mode 100644 index 0000000000..924b4e47fb --- /dev/null +++ b/evals/elsuite/function_deduction/README.md @@ -0,0 +1,91 @@ +# Function Deduction + +This eval evaluates how well a model can refine a hypothesis according to new evidence and how well it chooses to gather new information. + +In Function Deduction: + +- There is a secret mathematical function that maps an integer to another integer. +- The evaluated model interacts with the function by picking inputs to run through the function and observing black-box outputs. +- The model’s goal is to correctly predict outputs for a specified set of inputs, which is only possible by working out the underlying logic of the function. + +![fd](https://github.com/openai/policy-research-evals/assets/129281094/6c41be74-8237-4bb3-b0fc-13454c20389c) + +## Usage + +Run with: + +``` +oaieval function_deduction +``` + +We suggest using `function_deduction/cot/gpt-4-32k` or `function_deduction/cot/gpt-3.5-turbo-16k` as default choices for `` + +See `evals/registry/solvers/function_deduction.yaml` for a full list of recommended Solvers. + +For more examples of running this eval, see `scripts/run_experiments.sh` + +## Dataset + +The dataset are 100 int -> int functions of varying difficulty. Outputs of the functions for inputs 0 through 100 (the allowable range) are pre-cached and in the dataset. During the eval run, the eval simply looks up the output for any input the solver is asking about. + +## Evaluation Process + +Samples from the dataset are evaluated one-by-one. + +A model has up to 20 rounds to guess what the function would return for the three test input integers. + +In each round the model can either: + +- Ask what the function returns for a given integer (other than the test inputs) *or* +- Make a guess for the values that would be returned by the function for the test inputs. + +If the model makes a guess and the guess is correct for all 3 test inputs, the model has solved this sample; otherwise, if it guesses wrong, the model is told whether it guessed any subset of the test inputs correctly. + +After 20 rounds without a correct guess, the evaluation for that sample ends and the next sample starts. Rounds where the model answered in an incorrect format are still counted towards the 20 round total. + +## Task State + +| Key | Value | +| --- | --- | +| `task_description` | I have an important task for you. Instructions below.

I have a deterministic function `def calc(x: int) -> int:` that accepts an integer in range [0, 100] as an argument and returns an integer.

I want you to correctly determine the result of this function for each of the following inputs: {inputs}

At each turn you must select one of the following 2 options:
Option (A): Gather useful information. Ask me what would be returned for a given input to inform your hypothesis of what the function is. \
This must be an integer in range [0, 100], other than {inputs}.
Option (B): State your guess for the values returned by the function for inputs {inputs}. Your guess must include all three numbers, in this order. \
If your guess is correct, you win. If not, the task will continue.

You have {n_rounds} turns to determine the correct answer, else you lose.

Your goal is to determine the correct answer in as few turns as possible, so choose your actions carefully!
Your final score will be:
* If you make the correct guess: {n_rounds} - number_of_rounds_played
* If you fail to make the correct guess: -{n_rounds}
You should try to score as high as possible.

You should respond either with a single integer in range 0-100 if you decide to take option A or with three integers if you make the final guess (i.e. use option B).
Don't say anything more except for either an integer or 3 integers. | +| `current_state` | CurrentState object that tracks various data from the current dialog. | + +## Metrics + +The below are the key metrics of this eval: + +| Metric | Interpretation | +| --- | --- | +| `adjusted_avg_score` | Combination metric of the below 2 metrics. The average number of rounds for solved samples, or 40 for not-solved samples. (lower is better) | +| `solved_ratio` | The percentage of solved samples (higher is better) | +| `avg_success_rounds` | The average number of rounds for solved samples (lower is better) | + +## Variants + +| Variant | Notes | +| --- | --- | +| Default: `function_deduction.easy` | Default setting as described above. 1 trial per sample | +| `function_deduction.easy.long` | 10 trials per sample | +| `function_deduction.easy.dev5` | Dev set with only 5 samples | +| `function_deduction.hard` | A hard variant where the model is only told ‘this guess is incorrect’ if its wrong, instead of being told which inputs it got right/wrong. | +| `function_deduction.hard.dev5` | Dev set with only 5 samples | + +## Token Usage Estimates + +Below is a rough estimate of the total number of tokens consumed by the default variant: + +| Solver | Tokens | +| --- | --- | +| function_deduction/gpt-4-base | 3 840 000 | +| gpt-4-32k | 880 000 | +| gpt-3.5-turbo-16k | 1 560 000 | +| function_deduction/cot/gpt-4-32k | 12 400 000 | +| function_deduction/cot/gpt-3.5-turbo-16k | 13 230 000 | + +## Version History + +- v0: Initial version released + +## Contribution statement + +Eval design, implementation, and results evaluation were primarily conducted by Jan Betley with contributions from Andrei Alexandru. Report by James Aung. Work done under the guidance of (alphabetically by last-name) Steven Adler, and Chan Jun Shern, who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation. diff --git a/evals/elsuite/function_deduction/baselines.py b/evals/elsuite/function_deduction/baselines.py new file mode 100644 index 0000000000..3a81624e03 --- /dev/null +++ b/evals/elsuite/function_deduction/baselines.py @@ -0,0 +1,133 @@ +import logging +import math +from collections import Counter +from pathlib import Path + +import numpy as np +from scipy.stats import entropy + +from evals.data import get_jsonl +from evals.elsuite.function_deduction.eval import CurrentState, Sample +from evals.registry import Registry +from evals.solvers.solver import Solver, SolverResult +from evals.task_state import TaskState + + +class AverageBaseline(Solver): + """ + For given test inputs (x, y, z): + * Ask about values of (x-1, x+1, y-1, y+1, z-1, z+1) + * Make three guesses: round/floor/ceil of average values for neighboring numbers + If didn't succeed in 9 rounds (6x ask 3x guess) - surrender. + + Note: This algorithm fails on the edge cases where, for any of the inputs i: + - i-1 or i+1 is out of range + - i-1 or i+1 are part of the test inputs + In this scenario, the algorithm will fail at the _get_guess stage and skip the guessing. + """ + + def __init__(self, registry=None): + pass + + def _solve(self, task_state: TaskState): + cs: CurrentState = task_state.current_state + + assert len(cs.test_inputs) == 3, "AverageBaseline assumes 3 test inputs" + + if cs.round_ix < 6: + response = self._get_ask(cs.test_inputs, cs.round_ix) + elif 6 <= cs.round_ix < 9: + response = self._get_guess(cs.test_inputs, cs.known_values, cs.round_ix - 6) + else: + response = "I've run out of ideas sorry :(" + return SolverResult(response) + + def _get_guess(self, test_inputs, known_values: dict[int, int], guess_round_ix) -> str: + known_values = { + x: y for x, y in known_values.items() if x - 1 in test_inputs or x + 1 in test_inputs + } + + pairs = [[], [], []] + for i, test_input in enumerate(test_inputs): + try: + lower = known_values[test_input - 1] + higher = known_values[test_input + 1] + except KeyError: + return "Unfortunately I don't have enough data to make a guess, will pass." + pairs[i] = [lower, higher] + + funcs = [round, math.floor, math.ceil] + func = funcs[guess_round_ix] + vals = [func((pair[0] + pair[1]) / 2) for pair in pairs] + return " ".join([str(x) for x in vals]) + + def _get_ask(self, test_inputs, round_ix) -> str: + queries = [] + for x in test_inputs: + queries.append(x - 1) + queries.append(x + 1) + + ask = queries[round_ix] + if ask in test_inputs or ask < 0 or ask > 100: + logging.warning( + f"Invalid query on inputs {test_inputs}: {ask}. AverageBaseline algorithm will fail." + ) + return str(ask) + + +class FullKnowledge(Solver): + """Assuming solver knows all the samples, how well would it perform? + + Two modes - "random", where it selects random integer when asking, + and "best" where it selects the best integer. + + The "best" mode should be close to unbeatable (except for lucky guesses). + """ + + def __init__(self, mode: str, samples_jsonl: str, registry: Registry): + assert mode in ("random", "best"), "mode must be either random or best" + self.mode = mode + self._all_samples = self._get_samples(samples_jsonl, registry._registry_paths[0]) + self._rng = np.random.default_rng() + + def _solve(self, task_state: TaskState): + cs: CurrentState = task_state.current_state + + matching_samples = self._get_matching_samples(cs.known_values) + if len(matching_samples) > 1: + if self.mode == "random": + response = self._get_ask_random(cs.known_values) + else: + response = self._get_ask_best(matching_samples) + else: + sample_values = matching_samples[0].values + result = [sample_values[test_input] for test_input in cs.test_inputs] + response = " ".join([str(x) for x in result]) + return SolverResult(str(response)) + + def _get_matching_samples(self, known_values): + def matches(sample: Sample) -> bool: + for key, val in known_values.items(): + if sample.values[key] != val: + return False + return True + + return [sample for sample in self._all_samples if matches(sample)] + + def _get_ask_best(self, samples): + def get_entropy(x: int) -> float: + values = [sample.values[x] for sample in samples] + counter = Counter(values) + return entropy([val for val in counter.values()]) + + return max(range(0, 101), key=get_entropy) + + def _get_ask_random(self, known_values): + while True: + x = self._rng.integers(0, 100) + if x not in known_values: + return x + + def _get_samples(self, samples_jsonl: str, registry_path: Path): + path = registry_path / "data" / samples_jsonl + return [Sample(**x) for x in get_jsonl(path.as_posix())] diff --git a/evals/elsuite/function_deduction/eval.py b/evals/elsuite/function_deduction/eval.py new file mode 100644 index 0000000000..6542852153 --- /dev/null +++ b/evals/elsuite/function_deduction/eval.py @@ -0,0 +1,302 @@ +import logging +import random +import re +from dataclasses import dataclass, field +from typing import List, Literal, Optional, Tuple, Union + +import numpy as np +import scipy + +import evals +from evals.api import CompletionFn +from evals.elsuite.function_deduction import prompts +from evals.eval import SolverEval +from evals.solvers.solver import Solver +from evals.task_state import Message, TaskState + +logger = logging.getLogger(__name__) + + +@dataclass(frozen=True) +class Sample: + sample_ix: int + code: str + complexity: int + range: List[int] + values: List[int] + + +@dataclass +class CurrentState: + """This class tracks all the information from the dialog with the model. + + Some things are tracked to make writing solvers easier. + Other are tracked for metrics. + """ + + n_rounds: int + mode: str + test_inputs: tuple[int, int, int] + success: bool = False + known_values: dict[int, int] = field(default_factory=dict) + negative_known_values: dict[int, int] = field(default_factory=dict) + ask_rounds: int = 0 + guess_rounds: int = 0 + incorrect_format_rounds: int = 0 + parsed_responses: list[tuple[int]] = field(default_factory=list) + + @property + def round_ix(self): + return self.ask_rounds + self.guess_rounds + self.incorrect_format_rounds + + def ask_update(self, input_: int, value: Optional[int]) -> None: + self.ask_rounds += 1 + self.parsed_responses.append((input_,)) + if value is not None: + self.known_values[input_] = value + + def guess_update( + self, guessed_ints: tuple[int, int, int], expected_ints: tuple[int, int, int] + ) -> None: + self.guess_rounds += 1 + self.parsed_responses.append(guessed_ints) + if guessed_ints == expected_ints: + self.success = True + + if self.mode == "easy": + for test, guess, correct in zip(self.test_inputs, guessed_ints, expected_ints): + if guess == correct: + self.known_values[test] = guess + else: + self.negative_known_values[test] = guess + + +class FunctionDeductionEval(SolverEval): + def __init__( + self, + completion_fns: list[CompletionFn], + mode: Literal["easy", "hard"], + n_rounds: int, + n_samples: Optional[int] = None, + n_repeat: int = 3, + failed_sample_rounds: Optional[int] = None, + seed: Optional[int] = None, + samples_jsonl: str = "function_deduction/data.jsonl", + *args, + **kwargs, + ): + super().__init__(completion_fns, seed=seed, samples_jsonl=samples_jsonl, *args, **kwargs) + + self.mode = mode + self.n_rounds = n_rounds + self.n_samples = n_samples + self.n_repeat = n_repeat + + # This is used for the main metric - "how many rounds for a sample that was not solved?" + self.failed_sample_rounds = ( + failed_sample_rounds if failed_sample_rounds is not None else n_rounds * 2 + ) + + def eval_sample(self, solver: Solver, sample: Sample, rng: random.Random): + test_inputs = rng.sample(range(101), 3) + values = sample.values + expected = tuple(sample.values[test_input] for test_input in test_inputs) + + cs = CurrentState(self.n_rounds, self.mode, test_inputs) + task_state = TaskState( + prompts.task_description.format(inputs=test_inputs, n_rounds=self.n_rounds), + current_state=cs, + ) + + for round_ix in range(self.n_rounds): + raw_response = solver(task_state).output + try: + ints = self._parse_raw_response(raw_response) + except ValueError: + cs.incorrect_format_rounds += 1 + answer = prompts.incorrect_format + else: + if len(ints) == 1: + ask = ints[0] + result = values[ask] if ask not in test_inputs else None + cs.ask_update(ask, result) + if result is None: + answer = prompts.test_input_not_allowed.format(inputs=test_inputs) + else: + answer = prompts.new_value.format(in_=ask, out=result) + else: + cs.guess_update(ints, expected) + if cs.success: + break + else: + answer = self._bad_guess_answer(test_inputs, ints, expected) + + task_state.messages += [ + Message("assistant", raw_response), + Message("system", answer), + ] + + evals.record.record_metrics( + sample_ix=sample.sample_ix, + success=cs.success, + num_rounds=cs.round_ix if cs.success else None, + ask_rounds=cs.ask_rounds, + guess_rounds=cs.guess_rounds, + incorrect_format_rounds=cs.incorrect_format_rounds, + repeated_rounds=len(cs.parsed_responses) - len(set(cs.parsed_responses)), + code="lambda x: " + sample.code, + complexity=sample.complexity, + ) + + def run(self, recorder: evals.record.Recorder): + samples = self.get_samples() + + # Add copies according to self.n_repeat + # NOTE: we have copies next to each other -> more convenient when reading in logviz + copied_samples = [sample for sample in samples for _ in range(self.n_repeat)] + logger.info( + f"{len(samples)} unique samples, {self.n_repeat} attempts for each sample, {len(copied_samples)} total samples" + ) + self.eval_all_samples(recorder, copied_samples) + metrics = recorder.get_metrics() + + adjusted_rounds = [x["num_rounds"] or self.failed_sample_rounds for x in metrics] + main_metric = sum(adjusted_rounds) / len(metrics) + result = { + "adjusted_avg_score": main_metric, + "sem_adjusted_avg_score": self._calculate_sem(adjusted_rounds), + } + + result.update(self._get_success_metrics(metrics)) + result.update(self._get_sample_std(metrics)) + for name in ("ask_rounds", "guess_rounds", "incorrect_format_rounds"): + result[f"avg_{name}"] = sum(x[name] for x in metrics) / len(metrics) + result[f"sem_avg_{name}"] = self._calculate_sem([x[name] for x in metrics]) + result.update(self._get_complexity_tests(metrics)) + result.update(self._get_per_complexity_metrics(metrics)) + + return result + + def _calculate_sem(self, values: list) -> float: + return np.std(values) / np.sqrt(len(values)) + + def _get_success_metrics(self, metrics): + success = [x for x in metrics if x["success"]] + return { + "solved_ratio": round(len(success) / len(metrics), 2), + "sem_solved_ratio": self._calculate_sem([x["success"] for x in metrics]), + "solved": len(success), + "samples": len(metrics), + "avg_success_rounds": round(sum(x["num_rounds"] for x in success) / len(success), 2) + if success + else None, + "sem_avg_success_rounds": self._calculate_sem([x["num_rounds"] for x in success]) + if success + else None, + } + + def _get_sample_std(self, metrics): + adjusted = [] + no_failed = [] + solved_ratio_if_any_solved = [] + sample_ixs = set(metric["sample_ix"] for metric in metrics) + for sample_ix in sample_ixs: + sample_metrics = [metric for metric in metrics if metric["sample_ix"] == sample_ix] + sample_adjusted = [ + metric["num_rounds"] or self.failed_sample_rounds for metric in sample_metrics + ] + sample_no_failed = [ + metric["num_rounds"] for metric in sample_metrics if metric["success"] + ] + solved_ratio = sum(1 for metric in sample_metrics if metric["success"]) / len( + sample_metrics + ) + + if len(sample_adjusted) > 1: + adjusted.append(np.std(sample_adjusted)) + if len(sample_no_failed) > 1: + no_failed.append(np.std(sample_no_failed)) + if solved_ratio: + solved_ratio_if_any_solved.append(solved_ratio) + + return { + "avg_sample_rounds_std_adjusted": sum(adjusted) / len(adjusted) if adjusted else None, + "avg_sample_rounds_std_no_failed": sum(no_failed) / len(no_failed) + if no_failed + else None, + # This is just solved_ratio but excluding samples that had no succesful attempt. + # So 1 is full stability (i.e. if sample was solved once, it will be solved always), + # and (1/self.n_repeat) is "no sample was solved more than once" + "solved_ratio_if_any_solved": sum(solved_ratio_if_any_solved) + / len(solved_ratio_if_any_solved) + if solved_ratio_if_any_solved + else None, + } + + def _get_complexity_tests(self, metrics): + solved = [x["complexity"] for x in metrics if x["success"]] + not_solved = [x["complexity"] for x in metrics if not x["success"]] + result = { + "solved_avg_complexity": sum(solved) / len(solved) if solved else None, + "not_solved_avg_complexity": sum(not_solved) / len(not_solved) if not_solved else None, + } + + # This tests if solved have lower complexity than non-solved + if solved and not_solved: + _, p_value = scipy.stats.mannwhitneyu(solved, not_solved, alternative="less") + else: + p_value = None + result["solved_or_not_mann_whitney_u_p_value"] = p_value + + # TODO: add more complexity-related metrics, such as correlation or linear regression coefficient. + # Leaving this for the future because we might want to change how the complexity is calculated, + # or generally improve the concept somehow. + + return result + + def _get_per_complexity_metrics(self, all_metrics): + complexity_values = sorted(x["complexity"] for x in all_metrics) + result = {} + for complexity in complexity_values: + metrics = [x for x in all_metrics if x["complexity"] == complexity] + result[f"complexity_{complexity}"] = self._get_success_metrics(metrics) + return result + + def _parse_raw_response(self, response: str) -> Union[Tuple[int], Tuple[int, int, int]]: + # Remove all non-numbers first. This way we accept also e.g. "1, 2, 3", "[1, 2, 3]", '"1", "2", "3"' etc. + response = re.sub(r"[^0-9\s-]", "", response) + + vals = tuple(int(x) for x in response.split()) + if len(vals) not in (1, 3): + raise ValueError("Expected 1 or 3 integers") + if len(vals) == 1 and not 0 <= vals[0] <= 100: + raise ValueError("Single int should be between 0 and 100") + return vals + + def _bad_guess_answer(self, test_inputs, guessed, expected) -> str: + correct = [test_inputs[i] for i in range(0, 3) if guessed[i] == expected[i]] + incorrect = [x for x in test_inputs if x not in correct] + assert incorrect, "This is not a bad answer" + + if self.mode == "hard": + return "This is not the correct answer. At least one of the values is wrong." + elif self.mode == "easy": + if len(correct) == 0: + return "All numbers are wrong." + elif len(correct) == 1: + return f"Your guess is correct for {correct[0]} and incorrect for {incorrect[0]} and {incorrect[1]}" + elif len(correct) == 2: + return f"Your guess is correct for {correct[0]} and {correct[1]} and incorrect for {incorrect[0]}" + else: + raise ValueError(f"Unknown mode {self.mode}") + + def get_samples(self) -> List[Sample]: + samples = super().get_samples() + + if self.n_samples is not None: + assert ( + len(samples) >= self.n_samples + ), f"Can't get {self.n_samples} samples from a dataset with {len(samples)} samples" + np.random.default_rng(seed=self.seed).shuffle(samples) + samples = samples[: self.n_samples] + return [Sample(**sample_dict) for sample_dict in samples] diff --git a/evals/elsuite/function_deduction/prompts.py b/evals/elsuite/function_deduction/prompts.py new file mode 100644 index 0000000000..948e14079f --- /dev/null +++ b/evals/elsuite/function_deduction/prompts.py @@ -0,0 +1,43 @@ +task_description = """\ +I have an important task for you. Instructions below. + +I have a deterministic function `def calc(x: int) -> int:` that accepts an integer in range [0, 100] as an argument and returns an integer. + +I want you to correctly determine the result of this function for each of the following inputs: {inputs} + +At each turn you must select one of the following 2 options: +Option (A): Gather useful information. Ask me what would be returned for a given input to inform your hypothesis of what the function is. \ +This must be an integer in range [0, 100], other than {inputs}. +Option (B): State your guess for the values returned by the function for inputs {inputs}. Your guess must include all three numbers, in this order. \ +If your guess is correct, you win. If not, the task will continue. + +You have {n_rounds} turns to determine the correct answer, else you lose. + +Your goal is to determine the correct answer in as few turns as possible, so choose your actions carefully! +Your final score will be: +* If you make the correct guess: {n_rounds} - number_of_rounds_played +* If you fail to make the correct guess: -{n_rounds} +You should try to score as high as possible. + +You should respond either with a single integer in range 0-100 if you decide to take option A or with three integers if you make the final guess (i.e. use option B). +Don't say anything more except for either an integer or 3 integers. +""" +# TODO: Include in the task description whether we're in the easy or hard mode? +# (i.e. whether we'll be told about correctly guessed numbers (if any) or not). + +incorrect_format = "You responded in an incorrect format. Respond with either a single integer in range 0-100 or 3 integers. Just one number or three numbers, nothing more." +new_value = "calc({in_}) = {out}" +test_input_not_allowed = "Numbers {inputs} are not allowed" + +cot_template_first_round = """This is the first round. Think out loud about a general strategy of solving tasks like this.""" +cot_template_later_rounds = """\ +This is round {round_ix} out of {num_rounds}. + +Summary of the information you have already gathered: +{summary} + +Think out loud about the following questions: +* Do you have any hypothesis on what this function might be doing? +* If yes, should you try to test it (how?), or just use it to calculate the answer? +* If not, what additional information should you gather to be able to formulate a hypothesis? +""" diff --git a/evals/elsuite/function_deduction/scripts/dataset/create_dataset.py b/evals/elsuite/function_deduction/scripts/dataset/create_dataset.py new file mode 100644 index 0000000000..931e1cc27a --- /dev/null +++ b/evals/elsuite/function_deduction/scripts/dataset/create_dataset.py @@ -0,0 +1,62 @@ +import argparse +import dis +import json +import math + +DEFAULT_RANGE = [0, 100] # inclusive + + +def get_func_from_code(code): + return lambda x: eval(code, {"math": math, "x": x}) + + +def get_complexity(code: str) -> int: + # NOTE: this is quite ugly, but should be good enough for dataset-creating code + code = "global func_name\ndef func_name(x): return " + code + exec(code) + return len(list(dis.get_instructions(func_name))) + + +def create_dataset(out_file, in_file): + samples = [] + + for line in in_file: + line = line.strip() + + if not line or line.startswith("#"): + continue + + func = get_func_from_code(line) + values = list(int(func(x)) for x in range(DEFAULT_RANGE[0], DEFAULT_RANGE[1] + 1)) + samples.append( + { + "code": line, + "complexity": get_complexity(line), + "range": DEFAULT_RANGE, + "values": values, + } + ) + + # Ensure we don't have duplicates - they might be different functions, but if they return the same + # value for every input in the DEFAULT_RANGE then they are in fact the same sample. + for sample_ix, sample in enumerate(samples): + for other_sample in samples[sample_ix + 1 :]: + if sample["values"] == other_sample["values"]: + raise ValueError( + f"Samples {sample['code']} and {other_sample['code']} are indistinguishable" + ) + + samples.sort(key=lambda x: x["complexity"]) + + for i, sample in enumerate(samples): + sample = dict(sample_ix=i, **sample) + json.dump(sample, out_file) + out_file.write("\n") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out", type=argparse.FileType("w"), required=True) + parser.add_argument("--in", dest="in_", type=argparse.FileType("r"), default="raw_code.txt") + args = parser.parse_args() + create_dataset(out_file=args.out, in_file=args.in_) diff --git a/evals/elsuite/function_deduction/scripts/dataset/raw_code.txt b/evals/elsuite/function_deduction/scripts/dataset/raw_code.txt new file mode 100644 index 0000000000..ff03a0c76e --- /dev/null +++ b/evals/elsuite/function_deduction/scripts/dataset/raw_code.txt @@ -0,0 +1,141 @@ +# Lines starting with '#' or empty are ignored. +# Every other line is code for a single sample. +# This file is parsed by create_datset.py script +# (-> is not accessed when the eval is running). + +# Single operation +x +x * 2 +x * 27 +-x +x * -2 +x * -19 +math.floor(x * 1.5) +math.floor(x * 8.5) +math.floor(x / 2) +math.floor(x / 10) +math.ceil(x / 2) +round(x / 10) +math.ceil(x / 10) +x + 1 +x + 17 +x - 1 +x - 29 +7 - x +x ** 2 +x ** 3 + +# Small set of values +7 +7 if x % 2 else 17 +x % 3 +x % 7 +x % 10 +int(x % 3 == 1) +int(x % 3 == 2) +int(x % 3 == 1) * 7 +int(x % 3 == 2) * 18 +int(x < 32) +int(x % 8 < 4) + +# Threshold +min(x, 30) +max(x, 30) +min(x * 2, 70) +max(x * 2, 70) +x * 2 if x < 50 else x +x + 7 if x < 50 else x - 7 +x + 50 if x < 50 else 100 - x +x * 2 if x > 40 else x * 3 +3 if 30 < x < 70 else 4 +min(1000000, 2 ** x) + +# Multiple operations +math.floor(x + math.sqrt(x)) +math.floor(math.sqrt(x)) +math.floor(math.sqrt(x)) - 1 +math.floor(math.sqrt(x)) * 2 +math.floor(math.sqrt(x) * 2) +math.floor(round(x ** (1/3), 8)) +x / 2 if not x % 2 else x * 3 +x / 2 if not x % 2 else x * 3 + 1 +x ** 2 if x % 2 else x ** 3 +x / 3 if not x % 3 else x +x / 3 if not x % 3 else x * 2 +(x + 1) / 3 if x % 3 == 2 else x +x ** 2 - 10 +x ** 3 - x ** 2 +x ** 2 * 2 +x * (x - 1) +x * (x - 1) * (x - 2) +x * (x + 1) / 2 +5 - (x % 5) +10 - (x % 10) +16 - (x % 16) +x - x % 6 +x - x % 15 +x - x % 10 +x + x % 10 +x + x % 4 +x + x // 10 +x + x // 8 +x // 10 + x % 2 +(x + 5) * 3 +(x + 2) * 7 +(2 * x) ** 2 + + +# Math, sin, cos etc +round(math.sin(x)) +round(math.sin(x * 0.5 * math.pi)) +round(math.sin(x * 0.25 * math.pi) * 10) +round(math.sin(x * 0.1 * math.pi) * 10) +round(math.cos(x)) +round(math.cos(x * 0.5 * math.pi)) +round(math.cos(x * 0.25 * math.pi) * 10) +round(math.cos(x * 0.1 * math.pi) * 10) + +# Is prime number? +int(x > 1 and all(x % i for i in range(2, x))) +x if x > 1 and all(x % i for i in range(2, x)) else x + 1 + +# Is perfect square? +int(int(x**0.5)**2 == x) + +# Divisors - number / sum +sum(1 for i in range(1, x + 1) if not x % i) +sum(i for i in range(1, x + 1) if not x % i) + +# Reverse digits +int(str(x)[::-1]) +abs(x - int(str(x)[::-1])) +x + int(str(x)[::-1]) + +# Sum of digits +sum(int(d) for d in str(x)) +x + sum(int(d) for d in str(x)) +int(sum(int(d) for d in str(x)) % 10) + +# Count odd/even digits +sum(1 for d in str(x) if int(d) % 2) +sum(1 for d in str(x) if not int(d) % 2) + +# Multiple digits +0 if x < 10 else (x % 10) * (x // 10) + +# Higher vs lower digit +0 if x < 10 else max(int(d) for d in str(x)) - min(int(d) for d in str(x)) + +# Other +bin(x).count("1") +x | 1 +int(str(x) == str(x)[::-1]) +x * int(str(x)[-1]) + +# More ideas: convert to binary +# int(bin(x)[2:]) +# int(bin(~x)[3:]) +# int(bin(x * 2)[2:]) + +# More ideas: highest divisor lower than x? +# 0 if x == 0 else max(1 for i in range(1, x) if not x % i) diff --git a/evals/elsuite/function_deduction/scripts/make_plots.py b/evals/elsuite/function_deduction/scripts/make_plots.py new file mode 100644 index 0000000000..4c8f5f5e78 --- /dev/null +++ b/evals/elsuite/function_deduction/scripts/make_plots.py @@ -0,0 +1,256 @@ +import argparse +from pathlib import Path + +import matplotlib.pyplot as plt +import pandas as pd +import seaborn as sns + +from evals.utils import log_utils + +palette = { + "Average Baseline": "blue", + "Full Knowledge Best": "blue", + "Full Knowledge Random": "blue", + + "Human": "steelblue", + + "gpt-4-32k": "purple", + "gpt-4-32k w CoT": "purple", + + "gpt-4-base w Few-shot": "orange", + "gpt-4-base w CoT and Few-shot": "orange", + + "gpt-3.5-turbo-16k": "green", + "gpt-3.5-turbo-16k w CoT": "green", + + "gemini-pro": "peru", + "gemini-pro w CoT": "peru", + + "llama-2-13b-chat": "brown", + "llama-2-13b-chat w CoT": "brown", + + "llama-2-70b-chat": "maroon", + "llama-2-70b-chat w CoT": "maroon", + + "mixtral-8x7b-instruct": "grey", + "mixtral-8x7b-instruct w CoT": "grey", +} + +solver_to_name = { + "function_deduction/full_knowledge_best": "Full Knowledge Best", + "function_deduction/full_knowledge_random": "Full Knowledge Random", + "function_deduction/average_baseline": "Average Baseline", + + "human_cli": "Human", + + "gpt-4-32k": "gpt-4-32k", + "function_deduction/cot/gpt-4-32k": "gpt-4-32k w CoT", + + "function_deduction/gpt-4-base": "gpt-4-base w Few-shot", + "function_deduction/cot/gpt-4-base": "gpt-4-base w CoT and Few-shot", + + "gpt-3.5-turbo-16k": "gpt-3.5-turbo-16k", + "function_deduction/cot/gpt-3.5-turbo-16k": "gpt-3.5-turbo-16k w CoT", + + "generation/direct/gemini-pro": "gemini-pro", + "function_deduction/cot/gemini-pro": "gemini-pro w CoT", + + "generation/direct/llama-2-13b-chat": "llama-2-13b-chat", + "function_deduction/cot/llama-2-13b-chat": "llama-2-13b-chat w CoT", + + "generation/direct/llama-2-70b-chat": "llama-2-70b-chat", + "function_deduction/cot/llama-2-70b-chat": "llama-2-70b-chat w CoT", + + "generation/direct/mixtral-8x7b-instruct": "mixtral-8x7b-instruct", + "function_deduction/cot/mixtral-8x7b-instruct": "mixtral-8x7b-instruct w CoT", +} + +rename_columns = { + "adjusted_avg_rounds": "adjusted_avg_score", + "sem_adjusted_avg_rounds": "sem_adjusted_avg_score", +} + + +def extract_final_reports( + datadir: Path, rename_solvers: dict, rename_columns: dict +) -> pd.DataFrame: + df_rows = [] + for path, results in sorted(list(log_utils.get_final_results_from_dir(datadir).items())): + spec = log_utils.extract_spec(path) + solver_path = spec["completion_fns"][0] + print("adding report for", solver_path) + df_rows.append( + { + "solver": rename_solvers.get(solver_path, solver_path), + **{rename_columns.get(k, k): v for k, v in results.items()}, + } + ) + df = pd.DataFrame(df_rows) + return df + + +def make_plot( + df, + x_column: str, + y_column: str, + x_err_column: str, + title: str, + xlabel: str, + ylabel: str, + out_path: Path, +): + # Avg rounds until success (failure counts as 40) + plt.figure(figsize=(10, 6)) + ax = sns.barplot( + x=x_column, + y=y_column, + data=df, + xerr=df[x_err_column] * 1.96, + palette=palette, + ) + + plt.xlabel(xlabel) + plt.ylabel(ylabel) + plt.title(title) + plt.grid(axis="x") + plt.tight_layout() + + # Expanding the x-axis limit + x_lim = ax.get_xlim() + ax.set_xlim([x_lim[0], x_lim[1] * 1.05]) # Increase the upper limit by 5% + + # Annotating each bar with its value + for p in ax.patches: + width = p.get_width() + ax.text( + width + x_lim[1] * 0.02, # x position of text + p.get_y() + p.get_height() / 2, # y position of text + "{:.1f}".format(width), # text to be shown + va="center", + ) # vertical alignment + + plt.savefig(out_path) + return + + +def make_ask_guess_incorrect_plot(df, out_path: Path): + # Ask/Guess/Incorrect + + ask_guess_incorrect_data = { + "solver": df["solver"], + "Ask": df["avg_ask_rounds"], + "SEM Average Ask Rounds": df["sem_avg_ask_rounds"], + "Guess": df["avg_guess_rounds"], + "SEM Average Guess Rounds": df["sem_avg_guess_rounds"], + "Incorrect Format": df["avg_incorrect_format_rounds"], + "SEM Average Incorrect Format Rounds": df["sem_avg_incorrect_format_rounds"], + } + + agi_palette = { + "Ask": "blue", + "Guess": "pink", + "Incorrect Format": "red", + } + + ask_guess_incorrect_df = pd.DataFrame(ask_guess_incorrect_data) + + # Melting the DataFrame to make it suitable for seaborn's factorplot + melted_df = pd.melt( + ask_guess_incorrect_df, + id_vars="solver", + value_vars=["Ask", "Guess", "Incorrect Format"], + var_name="Round Type", + value_name="Average Rounds", + ) + + # Generating the plot for Average Ask/Guess/Incorrect Format Rounds + plt.figure(figsize=(14, 14)) + ax = sns.barplot( + x="Average Rounds", y="solver", hue="Round Type", data=melted_df, palette=agi_palette + ) + + plt.xlabel("Average Number of Rounds") + plt.ylabel("Solver") + plt.title("Distribution of Type of Responses by Model") + plt.grid(axis="x") + plt.legend(title="Response Type") + plt.tight_layout() + + # Expanding the x-axis limit + x_lim = ax.get_xlim() + ax.set_xlim([x_lim[0], x_lim[1] * 1.05]) # Increase the upper limit by 5% + + # Annotating each bar with its value + for p in ax.patches: + width = p.get_width() + ax.text( + width + 0.1, # x position of text + p.get_y() + p.get_height() / 2, # y position of text + "{:.1f}".format(width), # text to be shown + va="center", + ) # vertical alignment + + plt.savefig(out_path) + return + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--log-dir", "-d", type=str, required=True) + parser.add_argument("--out-dir", "-o", type=str, default="./outputs") + args = parser.parse_args() + log_dir = Path(args.log_dir) + out_dir = Path(args.out_dir) + + df = extract_final_reports(log_dir, solver_to_name, rename_columns) + + # Drop all columns named "complexity*" + df = df[df.columns.drop(list(df.filter(regex="complexity")))] + + # Creating a new DataFrame with the desired order + ordered_df = df.set_index("solver").loc[list(solver_to_name.values())].reset_index() + print(ordered_df) + + make_plot( + df=ordered_df, + x_column="adjusted_avg_score", + y_column="solver", + x_err_column="sem_adjusted_avg_score", + title="Adjusted Average Score (Lower is Better)", + xlabel="Adjusted Average Score", + ylabel="Solver", + out_path=out_dir / "avg_adjusted_score.png", + ) + + ordered_df["solved_ratio"] = 100 * ordered_df["solved_ratio"] + ordered_df["sem_solved_ratio"] = 100 * ordered_df["sem_solved_ratio"] + make_plot( + df=ordered_df, + x_column="solved_ratio", + y_column="solver", + x_err_column="sem_solved_ratio", + title="Solved Samples Ratio (Higher is Better)", + xlabel="Solved Ratio (%)", + ylabel="Solver", + out_path=out_dir / "solved_ratio.png", + ) + + make_plot( + df=ordered_df, + x_column="avg_success_rounds", + y_column="solver", + x_err_column="sem_avg_success_rounds", + title="Average Number of Rounds for Solved Samples (Lower is Better)", + xlabel="No. of Rounds", + ylabel="Solver", + out_path=out_dir / "avg_success_rounds.png", + ) + + make_ask_guess_incorrect_plot( + df=ordered_df, + out_path=out_dir / "ask_guess_incorrect.png", + ) + + +if __name__ == "__main__": + main() diff --git a/evals/elsuite/function_deduction/scripts/run_experiments.sh b/evals/elsuite/function_deduction/scripts/run_experiments.sh new file mode 100755 index 0000000000..4e67f5c7be --- /dev/null +++ b/evals/elsuite/function_deduction/scripts/run_experiments.sh @@ -0,0 +1,27 @@ + +logdir=./logs +timestamp=$(date +%Y%m%d_%H%M%S) +logpathbase="$logdir/$timestamp" + +echo Running experiments and logging to $logpathbase + +# Baselines +oaieval function_deduction/average_baseline function_deduction.easy --record_path "$logpathbase/average_baseline.log" +oaieval function_deduction/full_knowledge_best function_deduction.easy --record_path "$logpathbase/full_knowledge_best.log" +oaieval function_deduction/full_knowledge_random function_deduction.easy --record_path "$logpathbase/full_knowledge_random.log" --extra_eval_params n_repeat=100 + +declare -a SOLVERS=( + gpt-3.5-turbo-16k + gpt-4-32k + function_deduction/cot/gpt-3.5-turbo-16k + function_deduction/cot/gpt-4-32k + function_deduction/gpt-4-base + function_deduction/cot/gpt-4-base +) + +# Models +for solver in "${SOLVERS[@]}" +do + log_name=${solver//\//-} + oaieval $solver function_deduction.easy --record_path "$logpathbase/$log_name.log" +done diff --git a/evals/elsuite/function_deduction/solvers.py b/evals/elsuite/function_deduction/solvers.py new file mode 100644 index 0000000000..4830afe34a --- /dev/null +++ b/evals/elsuite/function_deduction/solvers.py @@ -0,0 +1,173 @@ +from typing import Any + +from evals.elsuite.function_deduction import prompts +from evals.elsuite.function_deduction.eval import CurrentState +from evals.solvers.nested.cot_solver import CoTSolver +from evals.solvers.nested.hhh_solver import HHHSolver +from evals.solvers.solver import SolverResult, SolverSpec +from evals.task_state import Message, TaskState + + +class CustomCoT(CoTSolver): + def __init__( + self, + cot_solver: SolverSpec, + extract_solver: SolverSpec, + persistent_memory: bool = True, + registry: Any = None, + ): + super().__init__( + cot_solver=cot_solver, + extract_solver=extract_solver, + persistent_memory=persistent_memory, + ) + + def cot_template(self, task_state: TaskState) -> str: + round_ix = task_state.current_state.round_ix + if round_ix == 0: + return prompts.cot_template_first_round + else: + summary = self._get_summary(task_state.current_state) + return prompts.cot_template_later_rounds.format( + round_ix=round_ix + 1, # displayed round number starts from 1 + num_rounds=task_state.current_state.n_rounds, + summary=summary, + ) + + def _get_summary(self, current_state: CurrentState) -> str: + rows = [] + for key, val in sorted(current_state.known_values.items()): + rows.append(f"calc({key}) = {val}") + + negative_rows = [] + for key, val in sorted(current_state.negative_known_values.items()): + negative_rows.append(f"calc({key}) != {val}") + + parts = [] + if rows: + parts.append("\n".join(rows)) + if negative_rows: + msg = "Information from your incorrect guesses:\n" + parts.append(msg + "\n".join(negative_rows)) + + if not parts: + return "You don't know anything yet." + else: + return "\n\n".join(parts) + + +class BaseModelSolver(HHHSolver): + def _solve(self, task_state: TaskState): + task_state = TaskState( + task_state.task_description, + self._few_shot_messages() + task_state.messages, + task_state.current_state, + ) + result = super()._solve(task_state) + result = result.output.splitlines()[0] + return SolverResult(result) + + def _few_shot_messages(self) -> list[Message]: + role = "system" + messages = [ + (role, "I have a hidden function. What is your first action?"), + ("assistant", "40"), + (role, "calc(40) = 160"), + ("assistant", "52"), + (role, "calc(52) = 204"), + ("assistant", "68 144 272"), + (role, "Correct guess!"), + (role, "I now have a new function. Forget about the previous one, we start again."), + ] + return [Message(*row) for row in messages] + + +class BaseModelCoTSolver(CustomCoT): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + @property + def cot_solver(self): + # This is a hack to enable us to postprocess the output during the CoT generation step: + # Here, we always keep just first line of the CoT generation, otherwise the base model + # goes on to copy the extraction prompt and confuses itself. + # TODO: Once we have solvers with proper built-in support for output postprocessors, + # update this to use that instead. + def cot_solver(task_state): + result = self.get_solver("cot_solver")(task_state).output + result = result.splitlines()[0] + return SolverResult(result) + + return cot_solver + + def _solve(self, task_state: TaskState): + task_state = TaskState( + task_state.task_description, + self._few_shot_messages(task_state.current_state) + task_state.messages, + task_state.current_state, + ) + + result = super()._solve(task_state) + result = result.output.splitlines()[0] + + # Fix the interaction history so that we can have persistent_memory = True + self.interaction_cache.last_interaction.messages[-1] = Message("assistant", result) + + return SolverResult(result) + + def _few_shot_messages(self, current_state) -> list[Message]: + # This is a bit hackish, but this way we can use self.cot_template (defined on CustomCoT), + # -> we'll have exactly the same system prompts in few shot part as we'd have in the eval + cs = CurrentState(current_state.n_rounds, current_state.mode, current_state.test_inputs) + task_state = TaskState("", [], cs) + + role = "system" + + messages = [ + (role, "I have a hidden function. What is your first action?"), + ] + messages.append((role, self.cot_template(task_state))) + messages.append( + ( + "assistant", + ( + "I don't know anything about the function yet. I should start with gathering some information, " + "e.g. by asking about the return value for 40." + ), + ) + ) + messages.append((role, self.extract_template(task_state))) + messages.append(("assistant", "40")) + messages.append((role, "calc(40) = 160")) + cs.ask_update(40, 160) + + messages.append((role, self.cot_template(task_state))) + messages.append( + ( + "assistant", + "Perhaps this function does multiplication by 4? That's not certain yet, so let's try some other number, e.g. 52.", + ) + ) + messages.append((role, self.extract_template(task_state))) + messages.append(("assistant", "52")) + messages.append((role, "calc(52) = 204")) + cs.ask_update(52, 204) + + messages.append((role, self.cot_template(task_state))) + messages.append( + ( + "assistant", + ( + "Now we have two results where the ouput is the input times 4. It seems that the function multiplies by 4. " + "I will make the guess now. 17 * 4 = 68, 36 * 4 = 144 and 68 * 4 = 272, so my guess will be 68 144 272." + ), + ) + ) + messages.append((role, self.extract_template(task_state))) + messages.append(("assistant", "68 144 272")) + messages.append((role, "Correct guess!")) + messages.append( + (role, "I now have a new function. Forget about the previous one, we start again.") + ) + + return [Message(*row) for row in messages] diff --git a/evals/elsuite/function_deduction/solvers_test.py b/evals/elsuite/function_deduction/solvers_test.py new file mode 100644 index 0000000000..8fadec107f --- /dev/null +++ b/evals/elsuite/function_deduction/solvers_test.py @@ -0,0 +1,149 @@ +from evals.elsuite.function_deduction.eval import CurrentState +from evals.elsuite.function_deduction.prompts import ( + cot_template_first_round, + cot_template_later_rounds, +) +from evals.elsuite.function_deduction.solvers import BaseModelCoTSolver, CustomCoT +from evals.solvers.solver import SolverSpec +from evals.task_state import Message, TaskState + +dummy_solver_spec = SolverSpec( + { + "class": "evals.solvers.solver:DummySolver", + "args": {}, + } +) + +GUESS_INPUT = 7 +ANSWER = 0 +N_ROUNDS = 10 +ROUNDS_SIMULATED = 2 +MODE = "easy" +TEST_INPUTS = (10, 20, 30) + + +def simulate_dummy_game(solver): + # Init state + task_description = "" # Not used + msgs = [] + cs = CurrentState( + n_rounds=N_ROUNDS, + mode=MODE, + test_inputs=TEST_INPUTS, + ) + + # ROUND 1 + solver_result = solver( + TaskState( + task_description=task_description, + messages=msgs, + current_state=cs, + ) + ) + + msgs.append(Message("assistant", solver_result.output)) + msgs.append(Message("system", f"The answer to your query is {ANSWER}")) + cs.ask_update(GUESS_INPUT, ANSWER) # Collect data for input=7 + + # ROUND 2 + solver_result = solver( + TaskState( + task_description=task_description, + messages=msgs, + current_state=cs, + ) + ) + return solver + + +def test_custom_cot(): + solver = CustomCoT(dummy_solver_spec, dummy_solver_spec) + simulate_dummy_game(solver) + + # Check that the customized CoT generation prompts appear as expected + # (and that the persistent memory in fact persists) + solver_private_memory = solver.interaction_cache.last_interaction.messages + assert solver_private_memory[0].content == cot_template_first_round + assert solver_private_memory[2].content == solver._extract_template + assert solver_private_memory[5].content == cot_template_later_rounds.format( + round_ix=ROUNDS_SIMULATED, + num_rounds=N_ROUNDS, + summary=f"calc({GUESS_INPUT}) = {ANSWER}", + ) + assert solver_private_memory[7].content == solver._extract_template + + +def test_base_model_cot_solver(): + solver = BaseModelCoTSolver(dummy_solver_spec, dummy_solver_spec) + simulate_dummy_game(solver) + + # Check that the memory contains the few-shot prompts + # followed by the customized CoT generation prompts + solver_private_memory = solver.interaction_cache.last_interaction.messages + + expected_few_shot_msgs = [ + Message(role="system", content="I have a hidden function. What is your first action?"), + Message( + role="system", + content="This is the first round. Think out loud about a general strategy of solving tasks like this.", + ), + Message( + role="assistant", + content="I don't know anything about the function yet. I should start with gathering some information, e.g. by asking about the return value for 40.", + ), + Message( + role="system", + content="Given the above reasoning, the answer in the format requested by the question is:", + ), + Message(role="assistant", content="40"), + Message(role="system", content="calc(40) = 160"), + Message( + role="system", + content="This is round 2 out of 10.\n\nSummary of the information you have already gathered:\ncalc(40) = 160\n\nThink out loud about the following questions:\n* Do you have any hypothesis on what this function might be doing?\n* If yes, should you try to test it (how?), or just use it to calculate the answer?\n* If not, what additional information should you gather to be able to formulate a hypothesis?\n", + ), + Message( + role="assistant", + content="Perhaps this function does multiplication by 4? That's not certain yet, so let's try some other number, e.g. 52.", + ), + Message( + role="system", + content="Given the above reasoning, the answer in the format requested by the question is:", + ), + Message(role="assistant", content="52"), + Message(role="system", content="calc(52) = 204"), + Message( + role="system", + content="This is round 3 out of 10.\n\nSummary of the information you have already gathered:\ncalc(40) = 160\ncalc(52) = 204\n\nThink out loud about the following questions:\n* Do you have any hypothesis on what this function might be doing?\n* If yes, should you try to test it (how?), or just use it to calculate the answer?\n* If not, what additional information should you gather to be able to formulate a hypothesis?\n", + ), + Message( + role="assistant", + content="Now we have two results where the ouput is the input times 4. It seems that the function multiplies by 4. I will make the guess now. 17 * 4 = 68, 36 * 4 = 144 and 68 * 4 = 272, so my guess will be 68 144 272.", + ), + Message( + role="system", + content="Given the above reasoning, the answer in the format requested by the question is:", + ), + Message(role="assistant", content="68 144 272"), + Message(role="system", content="Correct guess!"), + Message( + role="system", + content="I now have a new function. Forget about the previous one, we start again.", + ), + ] + assert solver_private_memory[: len(expected_few_shot_msgs)] == expected_few_shot_msgs + assert ( + solver_private_memory[len(expected_few_shot_msgs) + 0].content == cot_template_first_round + ) + assert ( + solver_private_memory[len(expected_few_shot_msgs) + 2].content == solver._extract_template + ) + assert solver_private_memory[ + len(expected_few_shot_msgs) + 5 + ].content == cot_template_later_rounds.format( + round_ix=ROUNDS_SIMULATED, + num_rounds=N_ROUNDS, + summary=f"calc({GUESS_INPUT}) = {ANSWER}", + ) + assert ( + solver_private_memory[len(expected_few_shot_msgs) + 7].content == solver._extract_template + ) diff --git a/evals/elsuite/identifying_variables/.gitattributes b/evals/elsuite/identifying_variables/.gitattributes new file mode 100644 index 0000000000..e256da66cb --- /dev/null +++ b/evals/elsuite/identifying_variables/.gitattributes @@ -0,0 +1 @@ +images/*.png filter=lfs diff=lfs merge=lfs -text diff --git a/evals/elsuite/identifying_variables/README.md b/evals/elsuite/identifying_variables/README.md new file mode 100644 index 0000000000..59912f0b27 --- /dev/null +++ b/evals/elsuite/identifying_variables/README.md @@ -0,0 +1,177 @@ +# Identifying Variables + +This eval tests how well models can determine what should be treated as the +independent, dependent, and control variables for an experiment that tests a +particular hypothesis, given some observational context. + +## Usage + +Run with: + +```bash +oaieval identifying_variables +``` + +We have found that `generation/cot/gpt-4-1106-preview` works well on this eval. For more examples of tested solvers, see [`./scripts/run_experiments.sh`](./scripts/run_experiments.sh). + +## Evaluation Process + +The evaluation process is as follows for a given sample from our dataset: + +1. The `TASK_DESCRIPTION` prompt is shown to the solver. +2. The sample is passed through a _renderer_ that processes the samples and + renders an observation of the interactions of variables, which is placed in + the `SAMPLE_MESSAGE` prompt template. +3. The solver answers in the form: `[@ANSWER valid_hyp: ; independent: ; dependent: ; control: ]`. The answer is parsed and evaluated by the eval. If the answer cannot be parsed, we mark this as a violation and the sample is treated as incorrect. + +## Prompts + +We refer readers to the [`./prompts.py`](./prompts.py) file for the +`TASK_DESCRIPTION` and `SAMPLE_MESSAGE` prompts used in the eval. + +## Metrics + + +| **Metric** | **Notes** | +|---|---| +| `ctrl_nDCG` | A modified version of the [normalized discounted cumulative gains (nDCG)](https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG) metric, which rewards listing the correct control variables first and penalizes naming irrelevant variables. | +| `ctrl_recall` | Number of variables correctly marked as control variables / total number of variables to control according to the gold label | +| `ctrl_recall` | Number of variables incorrectly marked as control variables / total number of variables not to control according to the gold label | +| `hyp_valid_acc` | Target hypothesis plausibility validation accuracy (correct/incorrect) | +| `ind_acc` | Independent variable determination accuracy (correct/incorrect) | +| `dep_acc` | Dependent variable determination accuracy (correct/incorrect) | +| `violation_rate` | Number of samples with violations (model failed to answer in correct format) / total number of samples | + + +## Variants + +We support variations on the eval along two dimensions, `renderer` and `dataset`: + +```bash +oaieval identifying_variables.. +``` + +The eval defaults to `identifying_variables.language-corrset.balanced-ctrl`. + +### Dataset + +We provide 4 dataset variants: + +| `dataset` | Notes | +| --- | --- | +| `balanced-ctrl` | 500 samples balanced across number of control variables (from 0 to 8). | +| `balanced-ctrl-large` | As `balanced-ctrl`, but with 5,000 samples. | +| `balanced-hypotheses` | 500 samples balanced across target hypotheses being implausible/plausible. | +| `balanced-hypotheses-large` | As `balanced-hypotheses`, but with 5,000 samples. | + +### Renderers + +We have 6 different renderers, implemented in [`./renderers/`](./renderers/). + +The default renderer is `language-corrset`. Here is an example render from this type: +``` +The following is a description of the observations made about a set of variables. + +In general, there were cases where some variables changed in tandem with each other, while others did not. +For example, changes in x_5075 were observed to reflect changes in x_3314 and viceversa. +Changes in x_9549 were not observed to reflect any changes in previously mentioned variables. +Changes in x_1808 were not observed to reflect any changes in previously mentioned variables. +Likewise, changes in x_9726 were observed to reflect changes in x_1808 and viceversa. +``` + +### Show Tree + +We provide an additional variant of the eval where the decision tree implementing +the reasoning for scoring a perfect score is shown to the model. This variant +can be run by passing the `show_tree=True` flag to eval, e.g. + +```bash +oaieval identifying_variables --extra_eval_params show_tree=True +``` + +## Custom Solvers + +We implement two custom programmatic solvers to serve as baselines. + +1. `identifying_variables/random`: a solver that randomly selects whether the + hypothesis is plausible with probability 0.5, and if so randomly samples the + independent, dependent and control variables. We view this baseline as + equivalent to randomly guessing. +2. `identifying_variables/noctrl`: this is a solver that always outputs an empty + list for the variables to control, essentially eliminating any chance of + false positives. This can provide stronger performance than the random + baseline, since it avoids any penalization for returning incorrect variables, + and can even achieve a perfect score on samples that indeed do not have any + variables to control + +We refer to [`./solvers.py`](./solvers.py) for the implementation of these +solvers. + +## Token Usage Estimates + +We estimated per-run token usage on the default dataset size (500 samples) +for the least and most token-intensive configurations for each model type +(respectively, direct models on `identifying_variables.corrset` with +`show_tree=False`; and CoT models on `identifying_variables.language-tabular` +with `show_tree=True`). + + +| | **input tokens/run** | **output tokens/run** | **total tokens/run** | +|---|---|---|---| +| **GPT-4-base HHH (corrset, no tree)** | 1,200,000 | 250,000 | 1,450,000 | +| **GPT-4-base CoT HHH (language-tabular, with tree)** | 1,500,000 | 240,000 | 1,740,000 | +| **GPT-3.5-turbo Direct (corrset, no tree)** | 430,000 | 88,000 | 518,000 | +| **GPT-3.5-turbo CoT (language-tabular, with tree)** | 780,000 | 14,000 | 794,000 | +| **GPT-4-1106-preview Direct (corrset, no tree)** | 430,000 | 53,000 | 483,000 | +| **GPT-4-1106-preview CoT (language-tabular, with tree)** | 860,000 | 14,000 | 874,000 | + + +These estimates were taken using the `balanced-hypotheses` dataset but should +roughly apply to the `-balanced-ctrl` datasets. For `-large` datasets (5000 +samples), multiply the above numbers by 10. + +## Future modifications + +- Revisit the definition of the headline `ctrl_nDCG` metric +- Devise additional auxiliary metrics to paint a more complete picture +- What if we show the decision trees described in natural language rather than + pseudocode? +- How can we extend this eval to multi-variable dependencies? + +## Version History + +- v0: Initial version released + +## Contribution Statement + +Eval design, implementation, and results evaluation and writeup were primarily +conducted by Giulio Starace. James Aung was of enormous assistance in report +writing, and additionally provided general feedback and project management +throughout the eval. Oliver Jaffe and Jan Betley were responsible for code +reviews throughout the implementation process, along with fine-grained feedback +on the project in general. Additional guidance was provided by (alphabetically +by last-name) Steven Adler and Chan Jun Shern, who helped with brainstorming, +gave research input and report revisions. + +## Appendix + +### Perfect output decision trees + +The following decision trees are used to determine the perfect output (aka "gold +label") for a given sample. + +--- + + + +**Figure A1**: Decision tree for determining whether a given variable should be +controlled. + +--- + + + +**Figure A2**: Decision tree for determining a hypothesis is valid and if so +what the independent and dependent variables are. + +--- diff --git a/evals/elsuite/identifying_variables/constants.py b/evals/elsuite/identifying_variables/constants.py new file mode 100644 index 0000000000..60729828c7 --- /dev/null +++ b/evals/elsuite/identifying_variables/constants.py @@ -0,0 +1,19 @@ +# variables that have at least this amount of sparsity are considered to be unobserved +SPARSITY_FOR_UNOBS = 0.8 +# num of variables in a given sample +MIN_VARS = 2 +MAX_VARS = 10 +# num of hypotheses in a given sample +MIN_HYPS = 1 +MAX_HYPS = 3 +# sparse var rate: percentage of variables to sparsify +MIN_SPARSE_VAR_RATE = 0 +MAX_SPARSE_VAR_RATE = 1 +# sparsity: percentage of NaNs in a sparsified variable +MIN_SPARSITY = 0.2 +MAX_SPARSITY = 1 + +# specific to tabular renderers ------------ + +# num of observations +NUM_OBS = 20 diff --git a/evals/elsuite/identifying_variables/eval.py b/evals/elsuite/identifying_variables/eval.py new file mode 100644 index 0000000000..31b3b743e0 --- /dev/null +++ b/evals/elsuite/identifying_variables/eval.py @@ -0,0 +1,292 @@ +""" +Implementation logic for Identifying Variables eval +""" +import logging +import random +from dataclasses import asdict +from typing import Dict, List, Optional, Tuple + +import networkx as nx +import numpy as np + +from evals.elsuite.identifying_variables import constants, graph_utils, prompts +from evals.elsuite.identifying_variables.metrics import ( + compute_fallout, + compute_nDCG, + compute_recall, +) +from evals.elsuite.identifying_variables.renderers import RENDERER_MAP +from evals.elsuite.identifying_variables.scripts.gen_data import gen_samples +from evals.elsuite.identifying_variables.structs import Answer, Sample +from evals.elsuite.identifying_variables.utils import json_to_sample, parse_solver_preds +from evals.eval import SolverEval +from evals.record import RecorderBase, record_metrics +from evals.solvers.solver import Solver, SolverResult +from evals.task_state import Message, TaskState + +logging.getLogger("httpx").setLevel(logging.WARNING) + + +class IdentifyingVariables(SolverEval): + def __init__( + self, + renderer: str, + n_samples: Optional[int] = None, + show_tree: bool = False, + group_metrics: bool = False, + debug: bool = False, + *args, + **kwargs, + ): + super().__init__(*args, **kwargs) + self.rng: random.Random = random.Random(self.seed) + self.np_rng: np.random.Generator = np.random.default_rng(self.seed) + self.renderer = RENDERER_MAP[renderer](rng=self.rng, np_rng=self.np_rng) + self.renderer_variant = renderer + self.n_samples = n_samples + self.show_tree = show_tree + self.task_description = self._build_task_description() + self.group_metrics = group_metrics + self.debug = debug + + def _build_task_description(self) -> str: + decision_tree_section = "" + if self.show_tree: + decision_tree_section = prompts.DECISION_TREE_SECTION + return prompts.TASK_DESCRIPTION.format( + optional_decision_tree_section=decision_tree_section, + ).strip() + + def eval_sample(self, solver: Solver, sample: Sample, rng: random.Random) -> None: + message: Message = self._build_message(sample) + + task_state = TaskState( + task_description=self.task_description, + messages=[message], + # to be used by the Random baseline solver only + current_state={"variables": [var for var in sample.causal_graph.nodes]}, + ) + + solver_result: SolverResult = solver(task_state) + + try: + preds = parse_solver_preds(solver_result) + except ValueError: # in case of invalid solver output + preds = None + gold, num_not_ctrl = sample.gold_label, sample.num_not_ctrl + + metrics: Dict[str, float] = self._evaluate_sample(preds, gold, num_not_ctrl) + + record_metrics( + **metrics, + # hack: logviz doesn't support custom log fields, so logging as metric + causal_graph=nx.to_dict_of_lists(sample.causal_graph), + gold_answer=asdict(gold), + n_hyps=sample.hypotheses.number_of_edges(), + valid_hyp=gold.valid_hypothesis, + num_not_ctrl=num_not_ctrl, + ) + + def run(self, recorder: RecorderBase) -> Dict[str, float]: + samples: List[Dict] = self._get_samples() + self.rng.shuffle(samples) + self.eval_all_samples(recorder, samples) + metrics: List[Dict] = recorder.get_metrics() + + return self._compute_agg_metrics(metrics) + + def _compute_agg_metrics(self, metrics: List[Dict]) -> Dict[str, float]: + """ + Computes aggregate metrics across all samples + """ + main_metrics = { + "hyp_valid_acc": np.mean([x["hyp_valid_correct"] for x in metrics]), + "violation_count": np.sum([x["violation"] for x in metrics]), + "violation_rate": np.mean([x["violation"] for x in metrics]), + # Some samples may be NaN for cases where the target hypothesis is invalid + "ctrl_nDCG": np.nanmean([x["ctrl_nDCG"] for x in metrics]), + "ctrl_recall": np.nanmean([x["ctrl_recall"] for x in metrics]), + "ctrl_fallout": np.nanmean([x["ctrl_fallout"] for x in metrics]), + "ind_acc": np.nanmean([x["ind_correct"] for x in metrics]), + "dep_acc": np.nanmean([x["dep_correct"] for x in metrics]), + "n_valid_hyp": np.sum([x["valid_hyp"] for x in metrics]), + } + if self.group_metrics: + grouped_metrics = self._compute_grouped_metrics(metrics) + else: + grouped_metrics = {} + + total_metrics = {**main_metrics, **grouped_metrics} + total_metrics = {k: float(v) for k, v in total_metrics.items()} + return total_metrics + + def _compute_grouped_metrics(self, metrics: List[Dict]) -> Dict[str, float]: + """ + Computes metrics aggregated across samples grouped by + - number of variables + - number of roots in random forest + - number of control variables + - number of hypotheses + - max correlation depth + """ + metric_to_agg_func = { + "hyp_valid_acc": np.mean, + "violation_count": np.sum, + "violation_rate": np.mean, + "ctrl_nDCG": np.nanmean, + "ctrl_recall": np.nanmean, + "ctrl_fallout": np.nanmean, + "ind_acc": np.nanmean, + "dep_acc": np.nanmean, + } + raw_metric_names = [ + "hyp_valid_correct", + "violation", + "violation", + "ctrl_nDCG", + "ctrl_recall", + "ctrl_fallout", + "ind_correct", + "dep_correct", + ] + group_to_bins = { + "n_vars": np.arange(constants.MIN_VARS, constants.MAX_VARS + 1), + "n_roots": np.arange(1, constants.MAX_VARS + 1), + "n_ctrl_vars": np.arange(0, (constants.MAX_VARS - 2) + 1), + "n_hyps": np.arange(constants.MIN_HYPS, constants.MAX_HYPS + 1), + "max_corr_depth": np.arange(1, constants.MAX_VARS), + } + grouped_metrics = { + f"{metric}-{group}-{g_bin}": [] + for metric in metric_to_agg_func.keys() + for group in group_to_bins.keys() + for g_bin in group_to_bins[group] + } + for log_entry in metrics: + causal_graph = nx.from_dict_of_lists(log_entry["causal_graph"], create_using=nx.DiGraph) + ctrl_vars = log_entry["gold_answer"]["ctrl_vars"] + dep_var = log_entry["gold_answer"]["dep_var"] + group_to_bin = { + "n_vars": causal_graph.number_of_nodes(), + "n_roots": len(graph_utils.find_graph_roots(causal_graph)), + "n_ctrl_vars": len(ctrl_vars) if ctrl_vars is not None else None, + "n_hyps": log_entry["n_hyps"], + "max_corr_depth": graph_utils.find_farthest_node(causal_graph, dep_var)[1] + if dep_var is not None + else None, + } + for group, g_bin in group_to_bin.items(): + if g_bin is not None: + for metric, raw_metric in zip(metric_to_agg_func.keys(), raw_metric_names): + grouped_metrics[f"{metric}-{group}-{g_bin}"].append(log_entry[raw_metric]) + + # aggregate + grouped_metrics = { + k: metric_to_agg_func[k.split("-")[0]](v) + # signal empty groups with np.nan + if len(v) > 0 else np.nan + for k, v in grouped_metrics.items() + } + return grouped_metrics + + def _evaluate_sample(self, preds: Optional[Answer], gold: Answer, num_not_ctrl: int) -> Dict: + """ + If the gold hypothesis is invalid, then all other metrics are skipped, and we + only evaluate whether the solver correctly identified the hypothesis as invalid. + + Mistakes are propagated: If the solver incorrectly identifies a hypothesis as + invalid, then its missing answers for the remaining tasks are counted as wrong. + + In case of violations, the worst possible metrics are recorded, accounting for + the gold hypothesis validity caveat above (e.g. if the gold hypothesis is + invalid, then the worst case ctrl_nDCG is NaN since we'd skip this anyway, + whereas if the gold hypothesis were valid, then the worst case ctrl_nDCG would + be 0.0) + """ + hyp_valid_correct = preds.valid_hypothesis == gold.valid_hypothesis if preds else False + + if gold.valid_hypothesis: + ind_correct = preds.ind_var == gold.ind_var if preds else False + dep_correct = preds.dep_var == gold.dep_var if preds else False + ctrl_nDCG = ( + self._ctrl_vars_nDCG(preds.ctrl_vars, gold.ctrl_vars, num_not_ctrl) + if preds and preds.ctrl_vars is not None + else 0.0 + ) + ctrl_recall = ( + self._ctrl_vars_recall(preds.ctrl_vars, gold.ctrl_vars) + if preds and preds.ctrl_vars is not None + else 0.0 + ) + # not in final report, since experiments had already been run + ctrl_fallout = ( + self._ctrl_vars_fallout(preds.ctrl_vars, gold.ctrl_vars, num_not_ctrl) + if preds and preds.ctrl_vars is not None + else 1.0 + ) + + else: + ctrl_nDCG = np.nan + ctrl_recall = np.nan + ctrl_fallout = np.nan + ind_correct = np.nan + dep_correct = np.nan + + return { + "ctrl_nDCG": ctrl_nDCG, + "ctrl_recall": ctrl_recall, + "ctrl_fallout": ctrl_fallout, + "ind_correct": ind_correct, + "dep_correct": dep_correct, + "hyp_valid_correct": hyp_valid_correct, + "violation": preds is None, + } + + def _ctrl_vars_fallout(self, preds: List[str], gold: List[str], num_not_ctrl: int) -> float: + return compute_fallout(set(preds), set(gold), num_not_ctrl) + + def _ctrl_vars_recall(self, preds: List[str], gold: List[str]) -> float: + return compute_recall(set(preds), set(gold)) + + def _ctrl_vars_nDCG(self, preds: List[str], gold: List[str], num_not_ctrl: int) -> float: + best = [1.0] * len(gold) + ranking = [1.0 if var in gold else -1.0 for var in preds] + worst_case_ctrl = [-1.0] * num_not_ctrl + return compute_nDCG(ranking, best, worst_case_ctrl) + + def _build_message(self, sample: Sample) -> Message: + observations: str = self.renderer.render_obs(sample) + hypotheses: List[str] = self._render_hypotheses(sample.hypotheses) + target_hypothesis: str = self._render_hypothesis(sample.target_hypothesis) + + message_content = prompts.SAMPLE_MESSAGE.format( + observations=observations, + hypotheses=hypotheses, + target_hypothesis=target_hypothesis, + ).strip() + message = Message("user", content=message_content) + + return message + + def _render_hypotheses(self, hypotheses: nx.DiGraph) -> List[str]: + hyp_list = [(n, adj) for n in hypotheses for adj in hypotheses[n]] + return [self._render_hypothesis(h) for h in hyp_list] + + def _render_hypothesis(self, hypothesis: Tuple[str, str]) -> str: + hyp_template = self.rng.choice(prompts.hypothesis_templates) + rendered_hyp = hyp_template.format(ind=hypothesis[0], dep=hypothesis[1]) + return rendered_hyp + + def _get_samples(self) -> List[Sample]: + if self.debug: + return gen_samples(n_samples=1000, signal_noise_ratio=None, np_rng=self.np_rng) + + dict_samples = self.get_samples() + if self.n_samples is not None: + assert ( + len(dict_samples) >= self.n_samples + ), f"Can't get {self.n_samples} samples from a dataset with {len(dict_samples)} samples" + np.random.default_rng(seed=self.seed).shuffle(dict_samples) + dict_samples = dict_samples[: self.n_samples] + samples = [json_to_sample(dict_sample) for dict_sample in dict_samples] + return samples diff --git a/evals/elsuite/identifying_variables/graph_utils.py b/evals/elsuite/identifying_variables/graph_utils.py new file mode 100644 index 0000000000..815ab968cc --- /dev/null +++ b/evals/elsuite/identifying_variables/graph_utils.py @@ -0,0 +1,254 @@ +"""Utils for network graph related operations.""" +from typing import Any, List, Optional, Set, Tuple, Union + +import networkx as nx +import numpy as np + + +def val_and_count_roots( + nodes: List[str], + np_rng: np.random.Generator, + total_edges: Optional[int] = None, + min_roots: Optional[int] = None, +) -> int: + """ + Validates the parameters for the construction of a random forest via + `gen_random_forest` and determines the min number of roots to use. + + A random forest following the constraints of `gen_random_forest` with + N nodes will have + - R <= N roots + - E <= N - R edges + If min_roots is not specified, then E <= N - 1, since R >= 1. + """ + n_nodes = len(nodes) + if min_roots is not None: + assert min_roots <= n_nodes, "Total roots must be less than or equal to the number of nodes" + if total_edges is not None: + assert ( + 0 <= total_edges <= n_nodes - min_roots + ), "Total edges must be between 0 and the number of nodes minus the number of roots" + else: + if total_edges is None: + min_roots = np_rng.integers(1, n_nodes + 1) + else: + assert ( + 0 <= total_edges <= n_nodes - 1 + ), "Total edges must be between 0 and the number of nodes minus 1" + # if total edges is specified, then we have an upper bound on R, R <= N - E + max_roots = n_nodes - total_edges + min_roots = np_rng.integers(1, max_roots + 1) + + return min_roots + + +def gen_random_forest_tree_size( + nodes: List[str], + tree_size: int, + np_rng: Optional[np.random.Generator] = None, +) -> nx.DiGraph: + """ + Builds a random forest, i.e. a Directed Acyclic Graph (DAG) + with potentially more than one root. + + We enforce the following constraints for our purposes: + 1. No self connections + 2. No bi-directional connections + 3. No children with multiple parents + 4. At least one root node (no parents) + 5. No cycles + + We additionally allow the user to specify the size that at least one + of the trees in the forest should be. + + Args: + nodes: A list of node names to build the graph from + tree_size: The number of nodes that at least one of the trees in the forest + should have + np_rng: A numpy random number generator + """ + num_nodes = len(nodes) + assert tree_size <= num_nodes, "Tree size must be less than or equal to the number of nodes" + + max_number_roots = num_nodes - tree_size + 1 + min_number_roots = 1 # 1 root is always reserved to the tree of size tree_size + + np_rng = np_rng or np.random.default_rng() + + num_roots = np_rng.integers(min_number_roots, max_number_roots + 1) + roots = set(np_rng.choice(nodes, num_roots, replace=False).tolist()) + + size_controlled_root = np_rng.choice(list(roots)) + size_controlled_tree_nodes = {size_controlled_root} + + shuffled_nodes = np_rng.permutation(nodes) + + graph_children = set() + + graph = nx.DiGraph() + graph.add_nodes_from(shuffled_nodes) + + while len(size_controlled_tree_nodes) < tree_size: + possible_children = [ + n for n in nodes if n not in size_controlled_tree_nodes and n not in roots + ] + child = np_rng.choice(possible_children) + possible_parents = list(size_controlled_tree_nodes) + parent = np_rng.choice(possible_parents) + graph.add_edge(parent, child) + size_controlled_tree_nodes.add(child) + graph_children.add(child) + + remaining_nodes = set(nodes) - size_controlled_tree_nodes + + for node in remaining_nodes: + possible_children = [ + n + for n in remaining_nodes + # avoid self connections + if n != node and + # avoid cycles and bi-directional conns -> ancestors can't be children + n not in nx.ancestors(graph, node) and + # avoid children with multiple parents + n not in graph_children and + # roots can't be children + n not in roots + ] + num_edges = np_rng.integers(0, len(possible_children) + 1) + children = np_rng.choice(possible_children, num_edges, replace=False).tolist() + + for child in children: + graph.add_edge(node, child) + graph_children.update(children) + + return graph + + +def gen_random_forest( + nodes: List[str], + total_edges: Optional[int] = None, + min_roots: Optional[int] = None, + np_rng: Optional[np.random.Generator] = None, +) -> nx.DiGraph: + """ + Builds a random forest, i.e. a Directed Acyclic Graph (DAG) + with potentially more than one root. + + We enforce the following constraints for our purposes: + 1. No self connections + 2. No bi-directional connections + 3. No children with multiple parents + 4. At least one root node (no parents) + 5. No cycles + + Args: + nodes: A list of node names to build the graph from + total_edges: The total number of edges in the graph. If None, will be random. + min_roots: The minimum number of roots in the graph. If None, will be random. + """ + np_rng = np_rng or np.random.default_rng() + graph = nx.DiGraph() + graph.add_nodes_from(nodes) + + min_roots = val_and_count_roots(nodes, np_rng, total_edges, min_roots) + + # the minimal set of roots, there may be more as we create the graph + roots = set(np_rng.choice(nodes, min_roots, replace=False).tolist()) + + graph_children = set() + edge_count = 0 + + shuffled_nodes = np_rng.permutation(nodes) + + for node in shuffled_nodes: + possible_children = [ + n + for n in nodes + # avoid self connections + if n != node and + # avoid cycles and bi-directional conns -> ancestors can't be children + n not in nx.ancestors(graph, node) and + # avoid children with multiple parents + n not in graph_children and + # roots can't be children + n not in roots + ] + + if len(possible_children) == 0: + continue + + if total_edges is not None: + remaining_edges = total_edges - edge_count + if remaining_edges <= 0: + break + num_edges = np_rng.integers(0, min(remaining_edges, len(possible_children)) + 1) + else: + num_edges = np_rng.integers(0, len(possible_children) + 1) + + children = np_rng.choice(possible_children, num_edges, replace=False).tolist() + + for child in children: + graph.add_edge(node, child) + graph_children.update(children) + edge_count += num_edges + + if total_edges is not None and edge_count < total_edges: + # If we didn't reach the total number of edges, try again + return gen_random_forest(nodes, total_edges, min_roots, np_rng) + + return graph + + +def find_farthest_node(graph: nx.DiGraph, source: str) -> Tuple[str, int]: + """ + Performs Breadth-First Search (BFS) to find the farthest node from the source node + and the distance to that node. Distance is defined as the number of edges between + the source node and the farthest node. + """ + graph = graph.to_undirected() + + # Compute shortest path lengths from source to all other nodes + path_lengths = nx.single_source_shortest_path_length(graph, source) + + # Find the farthest node + farthest_node = max(path_lengths, key=path_lengths.get) + max_distance = path_lengths[farthest_node] + + return farthest_node, max_distance + + +def find_graph_roots(graph: nx.DiGraph) -> Set[str]: + """ + Finds the root nodes of a graph + """ + return set([n for n, d in graph.in_degree() if d == 0]) + + +def find_graph_trees(graph: nx.DiGraph) -> List[Set[str]]: + """ + Finds the trees of a graph + """ + return [{root, *nx.descendants(graph, root)} for root in find_graph_roots(graph)] + + +def find_connected_nodes_pair( + graph: nx.DiGraph, np_rng: np.random.Generator +) -> Union[Tuple[Any, Any], None]: + """ + Finds a pair of connected nodes in a graph + If no such pair exists, returns None + """ + connected_pair = tuple(np_rng.choice(list(graph.edges))) if graph.edges else None + return connected_pair + + +def find_unconnected_nodes_pair(graph: nx.DiGraph) -> Union[Tuple[Any, Any], None]: + """ + Finds a pair of unconnected nodes in a graph + If no such pair exists, returns None + """ + components = list(nx.connected_components(graph.to_undirected())) + + if len(components) > 1: + return next(iter(components[0])), next(iter(components[1])) + return None diff --git a/evals/elsuite/identifying_variables/images/control_var_tree.png b/evals/elsuite/identifying_variables/images/control_var_tree.png new file mode 100755 index 0000000000..59de243e29 --- /dev/null +++ b/evals/elsuite/identifying_variables/images/control_var_tree.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:60bbedac103bae669c4cec1037faaa18b87df63ab5d2c61734f2c60211240fd6 +size 273556 diff --git a/evals/elsuite/identifying_variables/images/valid_hyp_tree.png b/evals/elsuite/identifying_variables/images/valid_hyp_tree.png new file mode 100644 index 0000000000..d005e47b47 --- /dev/null +++ b/evals/elsuite/identifying_variables/images/valid_hyp_tree.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:758a23f6b4bd7676852af320b28f8b6af61c404d22835eda99f2b8dc89a0277b +size 69394 diff --git a/evals/elsuite/identifying_variables/latent_funcs.py b/evals/elsuite/identifying_variables/latent_funcs.py new file mode 100644 index 0000000000..6f66a1c44e --- /dev/null +++ b/evals/elsuite/identifying_variables/latent_funcs.py @@ -0,0 +1,43 @@ +"""Latent functions for the project.""" +import numpy as np + + +def linear(x: np.ndarray, grad: float, bias: float) -> np.ndarray: + return grad * x + bias + + +def quadratic(x: np.ndarray, grad: float, bias: float) -> np.ndarray: + return grad * x**2 + bias + + +def random_uniform(num_samples, min_v, max_v, rng: np.random.Generator) -> np.ndarray: + return rng.uniform(min_v, max_v, num_samples) + + +def random_ints(num_samples, min_v, max_v, rng: np.random.Generator) -> np.ndarray: + return rng.integers(min_v, max_v, num_samples) + + +LATENT_FUNC_MAP = { + "linear": linear, + "quadratic": quadratic, +} +LATENT_FUNC_KWARG_MAP = { + "linear": { + "grad": {"min_v": -10, "max_v": 10}, + "bias": {"min_v": -100, "max_v": 100}, + }, + "quadratic": { + "grad": {"min_v": -10, "max_v": 10}, + "bias": {"min_v": -100, "max_v": 100}, + }, +} + +DISTRIBUTIONS = { + # "random_uniform": random_uniform, + "random_ints": random_ints, +} +DISTRIBUTIONS_KWARG_MAP = { + "random_uniform": {"min_v": -1, "max_v": 1}, + "random_ints": {"min_v": -100, "max_v": 100}, +} diff --git a/evals/elsuite/identifying_variables/metrics.py b/evals/elsuite/identifying_variables/metrics.py new file mode 100644 index 0000000000..501ec3b1a9 --- /dev/null +++ b/evals/elsuite/identifying_variables/metrics.py @@ -0,0 +1,105 @@ +from typing import Dict, List, Set + +import numpy as np + +from evals.elsuite.identifying_variables.utils import parse_solver_preds +from evals.solvers.solver import SolverResult + + +def compute_DCG(ranking: List[float], ceil_negs: bool = False) -> float: + """ + Computes the DCG of a ranking + """ + dcg = 0 + for i, rel in enumerate(ranking, start=1): + if ceil_negs: + rel = max(rel, 0) + dcg += rel / np.log2(i + 1) # (i+1) to avoid log_2(1) which = 0 + return dcg + + +def compute_nDCG(ranking: List[float], best: List[float], worst: List[float]) -> float: + """ + Computes nDCG, allowing for negative scores, based on the nDCG variant + from Gienapp et al. (2020) (https://dl.acm.org/doi/10.1145/3340531.3412123) + """ + idcg = compute_DCG(best) + min_dcg = compute_DCG(worst) + dcg = compute_DCG(ranking) + return (dcg - min_dcg) / (idcg - min_dcg) + + +def compute_metric_posthoc( + metric: str, metric_entries: List[Dict], sampling_entries: List[Dict] +) -> float: + """ + Computes a metric that was not logged by the eval, post-hoc, i.e. + after the eval has run, by reading the log file. + """ + metric_to_func = { + "ctrl_recall": compute_ctrl_recall_posthoc, + } + if metric not in metric_to_func.keys(): + raise ValueError(f"Metric {metric} not supported") + return metric_to_func[metric](metric_entries, sampling_entries) + + +def compute_ctrl_recall_posthoc(metric_entries: List[Dict], sampling_entries: List[Dict]) -> float: + """ + Computes the average recall for identified control variables + + i.e. the no. of correctly identified control variables / no. gold control variables + Averaged across the samples. + + - We skip any samples where the gold hypothesis is invalid + - And we skip any samples where there are no control variables in the gold label, + since recall is undefined in this case + """ + recalls = [] + for metric_entry, sampling_entry in zip(metric_entries, sampling_entries): + try: + preds = parse_solver_preds(SolverResult(output=sampling_entry["sampled"][0])) + except ValueError: # in case of invalid solver output (violation) + preds = None + + if metric_entry["gold_answer"]["valid_hypothesis"]: + if preds and preds.ctrl_vars is not None: + recall = compute_recall( + set(preds.ctrl_vars), set(metric_entry["gold_answer"]["ctrl_vars"]) + ) + else: + # worst case scenario in case of violation or incorrect hyp validation + recall = 0 + else: + recall = np.nan + recalls.append(recall) + return np.nanmean(recalls).astype(float) + + +def compute_fallout(retrieved: Set[str], gold_relevants: Set[str], num_irrelevant: int) -> float: + """ + Computes fallout for a sample + Number of retrieved irrelevant items / number of irrelevant items + + This value is undefined when there are no irrelevant items in the gold label + """ + if num_irrelevant == 0: + # undefined + return np.nan + retrieved_irrel_count = len([r for r in retrieved if r not in gold_relevants]) + return retrieved_irrel_count / num_irrelevant + + +def compute_recall(retrieved: Set[str], gold_relevants: Set[str]): + """ + Computes recall for a sample + Number of retrieved relevant items / number of relevant items + + This value is undefined when there are no relevant items in the gold label + """ + num_relevant = len(gold_relevants) + if num_relevant == 0: + # undefined + return np.nan + retrieved_rel_count = len([r for r in retrieved if r in gold_relevants]) + return retrieved_rel_count / num_relevant diff --git a/evals/elsuite/identifying_variables/prompts.py b/evals/elsuite/identifying_variables/prompts.py new file mode 100644 index 0000000000..4c2eecefda --- /dev/null +++ b/evals/elsuite/identifying_variables/prompts.py @@ -0,0 +1,103 @@ +TASK_DESCRIPTION = """\ +Your task is to determine what the correct independent, dependent\ + and control variables should be when designing an experiment for validating a specific\ + hypothesis, given some observations. + +The observations are organized into N different variables describing some environment.\ + The rules of this environment are unknown. The order in which the variables are\ + presented is arbitrary, and so are their names. In some cases, some if not all\ + observations for a variable may be missing, which simply means that those observations\ + have not been collected for that variable. + +You will also be presented with a list of hypotheses regarding the effect on some\ + variable by some other variable(s). From this list, a "target hypothesis" is chosen for\ + you to focus your choice of variables on. + +You should determine the independent, dependent and control variables based on the above. + +If there is more than one control variable, please separate them with commas, and order\ + them from most important to least important. If there are no control variables, output\ + "None" for the control variable field. Note that you are budget-constrained, so\ + you should only list control variables which would be quite risky not to control, given\ + what is known. + +If the target hypothesis is unlikely to be worth investigating, it is considered\ + invalid, and you should determine this as part of your answer. + +Your answer should be in the following format, including the square brackets: + +[@ANSWER valid_hyp: ; independent: ; dependent: ; control: ] + +For example, \ + [@ANSWER valid_hyp: true; independent: x_1; dependent: x_2; control: x_3, x_4] + +In case of invalid hypothesis, you may ignore the other fields\ + e.g. [@ANSWER valid_hyp: false; independent: NA; dependent: NA; control: NA] or \ + [@ANSWER valid_hyp: false] + +Only perform reasoning if requested. + +{optional_decision_tree_section} +""" + + +DECISION_TREE_SECTION = """\ +To determine the validity and thereby the independent and dependent variables of a\ + hypothesis stating that x_1 is a function of x_2, you can refer to the following\ + pseudocode + +```pseudocode +if correlated(x_1, x_2) is not None and not correlated(x_1, x_2): + "invalid hypothesis" +else: + "independent: x_2; dependent: x_1" +``` + +where `correlated` returns `True` if its arguments are correlated `False` if not,\ + and `None` if it is unknown. + +To determine whether a given variable x_n should be a control variable in an experiment\ + testing a (valid) hypothesis stating that x_1 is a function of x_2, you can refer to the\ + following pseudocode: + +```pseudocode +if x_n in {x_1, x_2}: + "do not control for x_n" +else: + if correlated(x_1, x_n) or correlated(x_2, x_n): + "control for x_n" + else: + if correlated(x_1, x_n) is not None: + "do not control for x_n" + else: + if hypothesized(ind=x_n, dep=x_1, allow_indirect=True): + "control for x_n" + else: + "do not control for x_n" +``` + +where `hypothesized` returns whether `ind` is hypothesized to be a cause of `dep`,\ + even indirectly through chains of hypotheses. +""" + + +SAMPLE_MESSAGE = """\ +Observations: + +{observations} + +Hypotheses: + +{hypotheses} + +Target Hypothesis: + +{target_hypothesis} +""" + + +hypothesis_templates = [ + "{dep} is a function of {ind}", + "{ind} affects {dep} through some function", + "{dep} is affected by {ind} through some function", +] diff --git a/evals/elsuite/identifying_variables/renderers/__init__.py b/evals/elsuite/identifying_variables/renderers/__init__.py new file mode 100644 index 0000000000..c155624761 --- /dev/null +++ b/evals/elsuite/identifying_variables/renderers/__init__.py @@ -0,0 +1,11 @@ +from . import tabular +from . import corrset + +RENDERER_MAP = { + "markdown": tabular.MarkdownTableRenderer, + "csv": tabular.CSVTableRenderer, + "json": tabular.JSONTableRenderer, + "language-tabular": tabular.LanguageTableRenderer, + "language-corrset": corrset.LanguageCorrSetRenderer, + "corrset": corrset.PureCorrSetRenderer, +} diff --git a/evals/elsuite/identifying_variables/renderers/base.py b/evals/elsuite/identifying_variables/renderers/base.py new file mode 100644 index 0000000000..90c1d27ae5 --- /dev/null +++ b/evals/elsuite/identifying_variables/renderers/base.py @@ -0,0 +1,16 @@ +import abc +import random + +import numpy as np + +from evals.elsuite.identifying_variables.structs import Sample + + +class RendererBase(abc.ABC): + def __init__(self, rng: random.Random, np_rng: np.random.Generator) -> None: + self.rng = rng + self.np_rng = np_rng + + @abc.abstractmethod + def render_obs(self, sample: Sample) -> str: + raise NotImplementedError diff --git a/evals/elsuite/identifying_variables/renderers/corrset.py b/evals/elsuite/identifying_variables/renderers/corrset.py new file mode 100644 index 0000000000..39563527a6 --- /dev/null +++ b/evals/elsuite/identifying_variables/renderers/corrset.py @@ -0,0 +1,346 @@ +from typing import List, Set, Tuple + +from evals.elsuite.identifying_variables.structs import Sample +from evals.elsuite.identifying_variables.renderers.base import RendererBase +import evals.elsuite.identifying_variables.graph_utils as graph_utils +import evals.elsuite.identifying_variables.renderers.templates as templates +from evals.elsuite.identifying_variables.constants import SPARSITY_FOR_UNOBS + + +class CorrSetRenderer(RendererBase): + """ + Describes the correlation structure of variables + """ + + def determine_sample_type(self, sample: Sample) -> Tuple[str, List[Set[str]]]: + """ + Determines the type of sample we have, returning the correlation sets in + the process. Accounts for unobserved variables by removing them from + the correlation sets. + + Returns: + str: The type of causal graph we have, ignoring unobserved variables. + Either + - "many_correl_sets": there are at least two correlation sets, at least + one of which has at least two variables. + - "single_correl_set": there is only one correlation set. + - "only_ind": there are at least two correlation sets, all of which + have exactly one variable. + List[Set[str]]: The list of correlation sets. A correlation set is the + set of observed variables in a tree from the causal graph + """ + causal_graph = sample.causal_graph + graph_trees = graph_utils.find_graph_trees(causal_graph) + correl_sets = [] + unobserved_vars = set( + var + for var in sample.variable_metadata + if sample.variable_metadata[var]["extra"]["sparsity_rate"] + > SPARSITY_FOR_UNOBS + ) + for tree in graph_trees: + correl_set = set(tree) + for var in tree: + if var in unobserved_vars: + # correlations to unobserved variables are, well, unobserved + correl_set.remove(var) + correl_sets.append(correl_set) + # need to check for empty sets, since we removed unobserved variables + correl_sets = [correl_set for correl_set in correl_sets if len(correl_set) > 0] + if len(correl_sets) == 1: + return "single_correl_set", correl_sets + else: + for correl_set in correl_sets: + if len(correl_set) > 1: + # at least one set with more than one observed var + return "many_correl_sets", correl_sets + # all sets have only one node + return "only_ind", correl_sets + + def _get_hypd_unobserved_vars(self, sample: Sample) -> List[str]: + vars_to_mention = [] + hypotheses = sample.hypotheses + + hypothesized_vars = set( + var + for var in hypotheses + if hypotheses.in_degree(var) > 0 or hypotheses.out_degree(var) > 0 + ) + vars_to_mention = [ + var + for var in hypothesized_vars + if sample.variable_metadata[var]["extra"]["sparsity_rate"] + > SPARSITY_FOR_UNOBS + ] + return vars_to_mention + + +class PureCorrSetRenderer(CorrSetRenderer): + def render_obs(self, sample: Sample) -> str: + _, observed_sets = self.determine_sample_type(sample) + + render_string = ( + "The following correlation sets were observed. Variables in the" + " same correlation set are correlated with each other, but not with variables in" + " other correlation sets." + ) + render_string += "\n\n" + self._render_observed_sets(observed_sets) + render_string += "\n\n" + self._render_unobserved_vars(sample) + + return render_string + + def _render_observed_sets(self, observed_sets: List[Set[str]]) -> str: + """ + Renders the observed sets. + """ + render_string = "" + for idx, correl_set in enumerate(observed_sets, start=1): + render_string += f"\nCorrelation set {idx}: {{{', '.join(correl_set)}}}." + return render_string.strip() + + def _render_unobserved_vars(self, sample: Sample) -> str: + """ + Renders the unobserved variables. + """ + unobserved_variables = self._get_hypd_unobserved_vars(sample) + if len(unobserved_variables) == 0: + render_string = "There were no unobserved variables." + else: + render_string = f"Unobserved variables: [{', '.join(unobserved_variables)}]." + return render_string.strip() + + +class LanguageCorrSetRenderer(CorrSetRenderer): + """ + Describes the correlation structure of variables in natural language. + """ + + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + self.type_to_renderer = { + "many_correl_sets": self.render_many_sets, + "single_correl_set": self.render_single_set, + "only_ind": self.render_only_ind, + } + + def render_obs(self, sample: Sample) -> str: + """ + Describes the interactions between variables in the sample. + + The description looks like + ``` + {opening statement} + + {description of the interactions} + + {optional mention of unobserved variables that were hypothesized about} + ``` + + The description of the interactions depends on the type of causal graph. + """ + sample_type, observed_sets = self.determine_sample_type(sample) + + opening_statement = templates.OPENING_STATEMENT + main_observation = self.type_to_renderer[sample_type](observed_sets) + unobserved_variables = self.mention_unobserved_vars(sample) + return "\n\n".join([opening_statement, main_observation, unobserved_variables]) + + def render_many_sets(self, correl_sets: List[Set[str]]): + """ + Renders a causal graph where we have at least two correlation + sets, one of which has at least two variables. + The description looks like: + ``` + In general, there were cases where some variables changed in tandem with each + other, while others did not. + {example of two variables that changed in tandem} + {interleaved mentions of remaining variables, specifying which other already + mentioned variables they changed in tandem with, if any} + ``` + """ + # Sort the sets by size, largest first + correl_sets = sorted(correl_sets, key=lambda x: len(x), reverse=True) + variables = [var for correl_set in correl_sets for var in correl_set] + + correl_set_idx_to_already_mentioned_vars = [set() for _ in correl_sets] + var_to_correl_set_idx = { + var: idx for idx, correl_set in enumerate(correl_sets) for var in correl_set + } + return_string = templates.MANY_CORREL_SETS_MAIN + + # hard-code mention first two variables, from first (largest) set + current_set_idx = 0 + return_string += "\n" + templates.CORREL_VARS_EXAMPLE.format( + optional_transition="For example, ", + # the first set is guaranteed to have at least two variables + var_1=variables[0], + var_2=variables[1], + ) + correl_set_idx_to_already_mentioned_vars[0].update([variables[0], variables[1]]) + + # go through remaining variables, randomly + variables = variables[2:] + self.rng.shuffle(variables) + + for var in variables: + correl_set_idx = var_to_correl_set_idx[var] + if correl_set_idx == current_set_idx: + transition_word = self.rng.choice(["Similarly", "Likewise"]) + transition_phrase = f"{transition_word}, " + else: + transition_phrase = "" + current_set_idx = correl_set_idx + + mentioned_vars_from_set = correl_set_idx_to_already_mentioned_vars[ + correl_set_idx + ] + if len(mentioned_vars_from_set) == 0: # first time mentioning this set + mention_string = templates.IND_VARS_EXAMPLE.format( + optional_transition=transition_phrase, + var_1=var, + var_2="previously mentioned variables", + ) + else: # variables from this set have been mentioned + mention_string = templates.CORREL_VARS_EXAMPLE.format( + optional_transition=transition_phrase, + var_1=var, + var_2=templates.list_to_nl_list(list(mentioned_vars_from_set)), + ) + return_string += "\n" + mention_string.capitalize() + # we have now mentioned this variable + correl_set_idx_to_already_mentioned_vars[correl_set_idx].add(var) + + return return_string + + def render_single_set(self, correl_sets: List[Set[str]]) -> str: + """ + Renders a causal graph where we have only one correlation set. + By definition, this set has at least two variables. + The description looks like: + ``` + In general, all of the variables seemed to change in tandem with each other. + For example, changes in {var_1} were observed to reflect changes in {var_2} and + viceversa. + {optional example of other pair} + {optional concluding statement that this holds for all pairs} + ``` + """ + correl_set = correl_sets[0] + # we won't use more than 3 variables in the examples. + exemplar_vars = list(correl_set)[:3] + remaining_vars = correl_set - set(exemplar_vars) + # always have at least 2 vars + example_1 = templates.CORREL_VARS_EXAMPLE.format( + optional_transition="", + var_1=exemplar_vars[0], + var_2=exemplar_vars[1], + ) + example_2 = "" + concluding_statement = "" + if len(exemplar_vars) == 3: + example_2 = templates.CORREL_VARS_EXAMPLE.format( + optional_transition="Additionally, ", + var_1=exemplar_vars[2], + var_2=templates.list_to_nl_list(exemplar_vars[:2]), + ) + if len(remaining_vars) > 0: + concluding_statement = templates.SPECIFIC_CONCL_STATEMENT.format( + already_mentioned=templates.list_to_nl_list(exemplar_vars), + remaining_vars=templates.list_to_nl_list(list(remaining_vars)), + ) + return templates.SINGLE_CORREL_SET_MAIN.format( + example_1=example_1, + optional_example_2=example_2, + optional_concluding_statement=concluding_statement, + ) + + def render_only_ind(self, correl_sets: List[Set[str]]) -> str: + """ + Describes a causal graph where we have at least two correlation + sets, all of which have only one variable, i.e. each variable + in the causal graph is independent of all other variables. The + description looks like: + ``` + In general, no discernible patterns were noticed between the variables. + For example, changes in {var_1} were not observed to reflect any changes in + {var_2}. + {optional example of other pair} + {optional concluding statement that this holds for all pairs} + ``` + """ + variables = [var for correl_set in correl_sets for var in correl_set] + num_vars = len(variables) # equal to the number of sets + # there's always at least 2 variables. + example_1 = templates.IND_VARS_EXAMPLE.format( + optional_transition="", + var_1=variables[0], + var_2=variables[1], + ) + example_2 = "" + concluding_statement = "" + if num_vars > 2: + example_2 = templates.IND_VARS_EXAMPLE.format( + optional_transition="Similarly, ", + var_1=variables[0], + var_2=variables[2], + ) + if num_vars > 3: + concluding_statement = templates.SPECIFIC_CONCL_STATEMENT.format( + already_mentioned=templates.list_to_nl_list(variables[:3]), + remaining_vars=templates.list_to_nl_list(variables[3:]), + ) + else: + concluding_statement = templates.GENERIC_CONCL_STATEMENT + + return templates.ONLY_IND_MAIN.format( + example_1=example_1, + optional_example_2=example_2, + optional_concluding_statement=concluding_statement, + ) + + def mention_unobserved_vars(self, sample: Sample) -> str: + """ + Mentions any unobserved variables that also hypothesized about. + """ + vars_to_mention = self._get_hypd_unobserved_vars(sample) + + n_vars_to_mention = len(vars_to_mention) + if n_vars_to_mention == 0: + return_string = "" + else: + be_plurality = {"singular": "is", "plural": "are"} + be_string = be_plurality["plural" if n_vars_to_mention > 1 else "singular"] + return_string = templates.UNOBS_BUT_HYP_VARS.format( + unobs_but_hyp_vars=templates.list_to_nl_list(vars_to_mention), + be_string=be_string, + ) + return return_string + + +if __name__ == "__main__": + import random + import numpy as np + + list_of_lists = [ + [{"x_1004"}, {"x_1005", "x_1006", "x_1007", "x_1008", "x_1009"}], + [{"x_1007", "x_1008", "x_1009"}, {"x_1010"}], + [{"x_1011"}, {"x_1012", "x_1013"}, {"x_1014"}], # 3 elements + [{"x_1022"}, {"x_1023", "x_1024"}, {"x_1025", "x_1026"}], + [{"x_1030"}, {"x_1031", "x_1032", "x_1033"}, {"x_1034"}, {"x_1035"}], + ] + + np_rng = np.random.default_rng(0) + renderer = PureCorrSetRenderer(random.Random(0), np_rng) + + from evals.elsuite.identifying_variables.scripts.gen_data import gen_samples + import networkx as nx + from pprint import pprint + + samples = gen_samples(10, None, np_rng) + + for sample in samples: + print("causal graph", nx.to_dict_of_lists(sample.causal_graph)) + print("hypotheses", list(sample.hypotheses.edges)) + pprint(sample.variable_metadata) + print(renderer.render_obs(sample)) + print("================") diff --git a/evals/elsuite/identifying_variables/renderers/tabular.py b/evals/elsuite/identifying_variables/renderers/tabular.py new file mode 100644 index 0000000000..0feb8b38fe --- /dev/null +++ b/evals/elsuite/identifying_variables/renderers/tabular.py @@ -0,0 +1,200 @@ +from typing import Optional, Tuple, Union, List +import json +import random + +import networkx as nx +import numpy as np +import pandas as pd + +from evals.elsuite.identifying_variables.structs import Sample +from evals.elsuite.identifying_variables.renderers.base import RendererBase +from evals.elsuite.identifying_variables.latent_funcs import ( + DISTRIBUTIONS, + LATENT_FUNC_MAP, +) +from evals.elsuite.identifying_variables.constants import NUM_OBS + + +def apply_noise( + data_df: pd.DataFrame, np_rng: np.random.Generator, snr: Optional[float] = None +) -> pd.DataFrame: + """ + Apply noise to a pandas DataFrame to achieve a specified Signal-to-Noise Ratio + (SNR). + + Args: + data_df (pd.DataFrame): The DataFrame containing the original data. + snr (float): The desired Signal-to-Noise Ratio in decibels (dB). + If None, no noise is applied. + """ + if snr is None: + return data_df + + desired_snr_linear = 10 ** (snr / 10) + + signal_powers = data_df.var() + noise_powers = signal_powers / desired_snr_linear + + noise = pd.DataFrame( + np_rng.normal(0, np.sqrt(noise_powers), data_df.shape), + columns=data_df.columns, + ) + noisy_df = data_df + noise + + return noisy_df + + +def sparsify_data( + data_df: pd.DataFrame, variable_metadata: dict, np_rng: np.random.Generator +) -> pd.DataFrame: + total_obs = data_df.shape[0] + for var in variable_metadata.keys(): + sparsity_rate = variable_metadata[var]["extra"]["sparsity_rate"] + num_missing_obs = int(sparsity_rate * total_obs) + missing_obs_indices = np_rng.choice(total_obs, num_missing_obs, replace=False) + data_df.loc[missing_obs_indices, var] = np.nan + return data_df + + +class TabularRenderer(RendererBase): + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + self.num_obs = NUM_OBS + + def _render_table(self, sample: Sample) -> pd.DataFrame: + variable_metadata = sample.variable_metadata + sample_metadata = sample.sample_metadata + n_obs_samples = self.num_obs + causal_graph = sample.causal_graph + + # "topological sort" from least to most ancestors (i.e. least to most dependent) + sorted_vars = nx.topological_sort(causal_graph) + # necessary so that we can generate data in the correct order + + data_dict = {} + for var in sorted_vars: + gen_method = variable_metadata[var]["gen_method"]["name"] + if "input_x" not in variable_metadata[var]["gen_method"]: + distr = DISTRIBUTIONS[gen_method] + distr_kwargs = variable_metadata[var]["gen_method"]["kwargs"] + data_dict[var] = distr( + num_samples=n_obs_samples, **distr_kwargs, rng=self.np_rng + ) + else: + latent_func = LATENT_FUNC_MAP[gen_method] + latent_func_kwargs = variable_metadata[var]["gen_method"]["kwargs"] + input_x = variable_metadata[var]["gen_method"]["input_x"] + data_dict[var] = latent_func(x=data_dict[input_x], **latent_func_kwargs) + + data_df = pd.DataFrame(data_dict) + + # apply noise after generating data + data_df = apply_noise(data_df, self.np_rng, sample_metadata["snr"]) + # apply sparsification after generating and noise + data_df = sparsify_data(data_df, variable_metadata, self.np_rng) + + # round to 3 decimal places + data_df = data_df.round(3) + + return data_df + + +class MarkdownTableRenderer(TabularRenderer): + """ + Renders tabular data as a markdown table with variable names as column names. + """ + + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + + def render_obs(self, sample: Sample) -> str: + data_df = self._render_table(sample) + return data_df.to_markdown(index=False) + + +class CSVTableRenderer(TabularRenderer): + """ + Renders tabular data as a comma-separated-values (CSV) file with variable names as + column names. + """ + + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + + def render_obs(self, sample: Sample) -> str: + data_df = self._render_table(sample) + return data_df.to_csv(index=False) + + +class JSONTableRenderer(TabularRenderer): + """ + Renders tabular data as a JSON object with variable names as keys and lists of + values as values. + """ + + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + + def render_obs(self, sample: Sample) -> str: + data_df = self._render_table(sample) + return json.dumps(data_df.to_dict(orient="list")) + + +class LanguageTableRenderer(TabularRenderer): + """ + Renders tabular data as a natural language description of the data. + Describing the data row by row. + """ + + def __init__(self, *args, **kwargs) -> None: + super().__init__(*args, **kwargs) + self.num_obs = 10 # set it to 10 + # realistically no one would read more than 10 rows of data one by one + + def render_obs(self, sample: Sample) -> str: + data_df = self._render_table(sample) + variables = list(data_df.columns) + rendered_obs = "" + current_step = "first" + for row in data_df.itertuples(index=False, name=None): + rendered_obs += self._render_row(row, variables, current_step) + "\n" + current_step = "next" + return rendered_obs + + def _render_row( + self, row: Tuple[Union[int, float]], variables: List[str], current_step: str + ) -> str: + string = f"On the {current_step} step, " + past_participle_verb = self.rng.choice(["measured", "recorded", "reported"]) + for value, var in zip(row, variables): + if np.isnan(value): + string += f"{var} was not {past_participle_verb}. " + else: + string += ( + f"{var} was {past_participle_verb} to be {format_number(value)}. " + ) + return string + + +def format_number(number: Union[int, float]): + """Get's rid of trailing .0's""" + if float(number).is_integer(): + return int(number) + else: + return number + + +if __name__ == "__main__": + # just for quick testing + np_rng = np.random.default_rng(0) + renderer = LanguageTableRenderer(random.Random(0), np_rng) + + from evals.elsuite.identifying_variables.scripts.gen_data import gen_samples + + samples = gen_samples(10, None, np_rng) + + for sample in samples: + print(nx.to_dict_of_lists(sample.causal_graph)) + print(sample.variable_metadata) + print(renderer.render_obs(sample)) + print("================") diff --git a/evals/elsuite/identifying_variables/renderers/templates.py b/evals/elsuite/identifying_variables/renderers/templates.py new file mode 100644 index 0000000000..c7a9000072 --- /dev/null +++ b/evals/elsuite/identifying_variables/renderers/templates.py @@ -0,0 +1,56 @@ +from typing import List + + +def list_to_nl_list(list_of_words: List[str]) -> str: + """ + Converts a list of words into a natural language list. + """ + if len(list_of_words) == 1: + return list_of_words[0] + elif len(list_of_words) == 2: + return f"{list_of_words[0]} and {list_of_words[1]}" + else: + return f"{', '.join(list_of_words[:-1])} and {list_of_words[-1]}" + + +OPENING_STATEMENT = """\ +The following is a description of the observations made about a set of variables. +""".strip() + +MANY_CORREL_SETS_MAIN = """\ +In general, there were cases where some variables changed in tandem with each other,\ + while others did not. +""".strip() + +SINGLE_CORREL_SET_MAIN = """\ +In general, all of the variables seemed to change in tandem with each other. +For example, {example_1} {optional_example_2} {optional_concluding_statement} +""".strip() + +ONLY_IND_MAIN = """\ +In general, no discernible patterns were noticed between the variables. +For example, {example_1} {optional_example_2} {optional_concluding_statement} +""".strip() + +CORREL_VARS_EXAMPLE = """\ +{optional_transition}changes in {var_1} were observed to reflect changes in {var_2} and\ + viceversa. +""".strip() + +IND_VARS_EXAMPLE = """\ +{optional_transition}changes in {var_1} were not observed to reflect any changes in\ + {var_2}. +""".strip() + +SPECIFIC_CONCL_STATEMENT = """\ +Similar observations were made for all other pairings within and across\ + {already_mentioned} and {remaining_vars}. +""".strip() + +GENERIC_CONCL_STATEMENT = """\ +Similar observations were made for all other pairings of the observed variables. +""".strip() + +UNOBS_BUT_HYP_VARS = """\ +{unobs_but_hyp_vars} {be_string} not observed but {be_string} hypothesized about. +""".strip() diff --git a/evals/elsuite/identifying_variables/scripts/data.sh b/evals/elsuite/identifying_variables/scripts/data.sh new file mode 100755 index 0000000000..418ebe3fef --- /dev/null +++ b/evals/elsuite/identifying_variables/scripts/data.sh @@ -0,0 +1,13 @@ +#!/bin/bash + +# generate datasets of size 500 and 5000 +echo "Generating default dataset: 500 samples" +python gen_data.py --n_samples 500 --jsonl_dir ../../../registry/data/identifying_variables/ +echo "Generating large dataset: 5000 samples" +python gen_data.py --n_samples 5000 --jsonl_dir ../../../registry/data/identifying_variables/ +echo "Generating default dataset: 500 samples (balanced ctrl vars)" +python gen_data.py --balanced_ctrl_vars --n_samples 500 --jsonl_dir ../../../registry/data/identifying_variables/ +echo "Generating large dataset: 5000 samples (balanced ctrl vars)" +python gen_data.py --balanced_ctrl_vars --n_samples 5000 --jsonl_dir ../../../registry/data/identifying_variables/ + +echo "Done." diff --git a/evals/elsuite/identifying_variables/scripts/gen_data.py b/evals/elsuite/identifying_variables/scripts/gen_data.py new file mode 100644 index 0000000000..14c5f78e28 --- /dev/null +++ b/evals/elsuite/identifying_variables/scripts/gen_data.py @@ -0,0 +1,467 @@ +""" +Code for generating .jsonl dataset for identifying variables eval + +Use default argparse args to replicate the dataset used for the report +""" + +from dataclasses import asdict +import os +import argparse +from typing import Dict, List, Optional, Set, Tuple, Any +import json +import copy + +from tqdm.auto import tqdm +import networkx as nx +import numpy as np + +import evals.elsuite.identifying_variables.latent_funcs as latent_funcs +from evals.elsuite.identifying_variables.graph_utils import ( + gen_random_forest, + gen_random_forest_tree_size, + find_graph_roots, + find_unconnected_nodes_pair, + find_connected_nodes_pair, +) +from evals.elsuite.identifying_variables.utils import sample_serializer +from evals.elsuite.identifying_variables.structs import Sample, Answer +import evals.elsuite.identifying_variables.constants as constants + + +def write_to_jsonl( + samples: List[Sample], + jsonl_path: str, +): + with open(jsonl_path, "w") as f: + for sample in samples: + f.write(json.dumps(asdict(sample), default=sample_serializer) + "\n") + + +def random_latent_func_meta( + np_rng: np.random.Generator, input_x: Optional[str] = None +) -> Dict: + """ + Generates random metadata for defining a latent function + + Args: + input_x (Optional[str]): Name of input variable. If None, then + the latent function is a distribution, not dependent on any input. + """ + if input_x is None: + latent_func_name = np_rng.choice(list(latent_funcs.DISTRIBUTIONS.keys())) + predefined_kwargs = latent_funcs.DISTRIBUTIONS_KWARG_MAP[latent_func_name] + kwargs = {**predefined_kwargs} + return {"name": latent_func_name, "kwargs": kwargs} + else: + latent_func_name = np_rng.choice(list(latent_funcs.LATENT_FUNC_MAP.keys())) + predefined_kwargs = latent_funcs.LATENT_FUNC_KWARG_MAP[latent_func_name] + kwargs = {} + for kwarg, min_max in predefined_kwargs.items(): + kwarg_value = np_rng.integers(min_max["min_v"], min_max["max_v"]) + while kwarg == "grad" and kwarg_value == 0: + # dont allow 0 gradient + kwarg_value = np_rng.integers(min_max["min_v"], min_max["max_v"]) + kwargs[kwarg] = kwarg_value + return {"name": latent_func_name, "input_x": input_x, "kwargs": kwargs} + + +def build_var_metadata( + causal_graph: nx.DiGraph, + sparse_var_rate: float, + np_rng: np.random.Generator, +) -> Dict: + """ + Builds the variable metadata for a sample, containing + information on how each variable is generated and which variables + it is correlated with. + + Args: + causal_graph (nx.DiGraph): Causal graph of the sample. + sparse_var_rate (float): Percentage of variables that should be sparsified. + max_sparsity (float): Maximum sparsity rate for sparse variables. + np_rng (np.random.Generator): Random number generator to be used. + """ + var_metadata = {} + + roots = find_graph_roots(causal_graph) + root_to_descendants = {r: nx.descendants(causal_graph, r) for r in roots} + node_to_root = { + n: root + for root, descendants in root_to_descendants.items() + for n in descendants + } + + for var in causal_graph: + if var in roots: + latent_func_meta = random_latent_func_meta(np_rng, input_x=None) + var_root = var + else: + parent = next(causal_graph.predecessors(var)) + latent_func_meta = random_latent_func_meta(np_rng, input_x=parent) + var_root = node_to_root[var] + # variables with a common root are correlated. Need to copy to avoid mutation + corrs: Set[str] = set(root_to_descendants[var_root]) + if var_root != var: + # remove self-correlation, add correlation to root itself + corrs.remove(var) + corrs.add(var_root) + + var_metadata[var] = { + "gen_method": latent_func_meta, + "corrs": corrs, + "extra": {"sparsity_rate": 0}, + } + + # add sparsity + var_metadata = sparsify_data(var_metadata, sparse_var_rate, np_rng) + + return var_metadata + + +def sparsify_data(var_metadata, sparse_var_rate, np_rng): + num_observed_vars = 0 + orig_var_metadata = copy.deepcopy(var_metadata) + for var in var_metadata.keys(): + if np_rng.uniform(0, 1) < sparse_var_rate: + sparsity_rate = np_rng.uniform( + low=constants.MIN_SPARSITY, high=constants.MAX_SPARSITY + ) + var_metadata[var]["extra"]["sparsity_rate"] = sparsity_rate + if sparsity_rate > constants.SPARSITY_FOR_UNOBS: + # remove unobserved variables from correlations + for corr_var in var_metadata[var]["corrs"]: + var_metadata[corr_var]["corrs"].remove(var) + var_metadata[var]["corrs"] = set() + else: + num_observed_vars += 1 + else: + num_observed_vars += 1 + + # if less than 2 observed variables, sparsification was too much, try again + if num_observed_vars < 2: + var_metadata = sparsify_data(orig_var_metadata, sparse_var_rate, np_rng) + + return var_metadata + + +def gen_sample_balanced_ctrl_vars( + signal_noise_ratio: Optional[float], np_rng: np.random.Generator +) -> Sample: + """ + Generates a sample for the dataset, containing information on how a set + of variables are interlinked, and which hypotheses are currently held. + + This differs from gen_sample in the following ways: + + To simplify: + - The total number of variables in a given sample is fixed to MAX_VARS + - The hypothesis is always valid + + The number of control variables is sampled uniformly between 0 and MAX_VARS-2 + (we subtract 2 since two variables are involved in the hypothesis) + """ + sample_metadata = {"snr": signal_noise_ratio} + + n_vars = constants.MAX_VARS + + sparse_var_rate = np_rng.uniform( + low=constants.MIN_SPARSE_VAR_RATE, high=constants.MAX_SPARSE_VAR_RATE + ) # perc of variables to sparsify + + var_ids = np_rng.choice(np.arange(1000, 10000), size=n_vars, replace=False).astype( + str + ) + var_names = [f"x_{var_id}" for var_id in var_ids] + + num_ctrl_vars = np_rng.integers(low=0, high=n_vars - 1) # high is exclusive + + causal_graph = gen_random_forest_tree_size( + nodes=var_names, tree_size=num_ctrl_vars + 2, np_rng=np_rng + ) + + variable_metadata = build_var_metadata(causal_graph, sparse_var_rate, np_rng) + + target_hypothesis = find_connected_nodes_pair(causal_graph, np_rng) + target_hyp_is_valid = ( + parse_target_hyp(target_hypothesis, variable_metadata)[0] + if target_hypothesis is not None + else None + ) + # try again if the sparsification caused the hypothesis to be invalid + if target_hypothesis is None or not target_hyp_is_valid: + return gen_sample_balanced_ctrl_vars(signal_noise_ratio, np_rng) + + n_hypotheses = np_rng.integers( + low=constants.MIN_HYPS, + high=min(constants.MAX_HYPS, n_vars - 1) + 1, + ) + hypotheses = gen_random_forest(var_names, total_edges=n_hypotheses, np_rng=np_rng) + + hypotheses = integrate_target_hyp(target_hypothesis, hypotheses, np_rng) + + gold_label, num_not_ctrl = determine_gold_label( + target_hypothesis, variable_metadata, hypotheses + ) + + return Sample( + variable_metadata=variable_metadata, + hypotheses=hypotheses, + target_hypothesis=target_hypothesis, + sample_metadata=sample_metadata, + # keep track of underlying ground truth in case want more in depth analysis + causal_graph=causal_graph, + gold_label=gold_label, + num_not_ctrl=num_not_ctrl, + ) + + +def gen_sample( + signal_noise_ratio: Optional[float], + np_rng: np.random.Generator, + valid_hyp_requested: Optional[bool] = None, +) -> Sample: + """ + Generates a sample for the dataset, containing information on how a set + of variables are interlinked, and which hypotheses are currently held. + + Args: + signal_noise_ratio (float): Signal-to-noise ratio to be applied to the + observations. If None, no noise is applied. + np_rng (np.random.Generator): Random number generator to be used. + valid_hyp_requested (Optional[bool]): Whether the target hypothesis should be + valid. If None, will be randomly chosen. + + Returns: + Sample: A sample as defined by the `Sample` dataclass. + """ + sample_metadata = {"snr": signal_noise_ratio} + + n_vars = np_rng.integers(low=constants.MIN_VARS, high=constants.MAX_VARS + 1) + sparse_var_rate = np_rng.uniform( + low=constants.MIN_SPARSE_VAR_RATE, high=constants.MAX_SPARSE_VAR_RATE + ) # perc of variables to sparsify + + var_ids = np_rng.choice(np.arange(1000, 10000), size=n_vars, replace=False).astype( + str + ) + var_names = [f"x_{var_id}" for var_id in var_ids] + + causal_graph = gen_random_forest(var_names, np_rng=np_rng) + + variable_metadata = build_var_metadata(causal_graph, sparse_var_rate, np_rng) + + n_hypotheses = np_rng.integers( + low=constants.MIN_HYPS, + high=min(constants.MAX_HYPS, n_vars - 1) + 1, + ) + hypotheses = gen_random_forest(var_names, total_edges=n_hypotheses, np_rng=np_rng) + + if valid_hyp_requested is None: + # 0.5 chance of valid hypothesis + valid_hyp_requested = np_rng.uniform(0, 1) < 0.5 + + if valid_hyp_requested: + target_hypothesis = find_connected_nodes_pair(causal_graph, np_rng) + else: + target_hypothesis = find_unconnected_nodes_pair(causal_graph) + + target_hyp_is_valid = ( + parse_target_hyp(target_hypothesis, variable_metadata)[0] + if target_hypothesis is not None + else None + ) + if target_hypothesis is None or target_hyp_is_valid != valid_hyp_requested: + return gen_sample(signal_noise_ratio, np_rng, valid_hyp_requested) + + hypotheses = integrate_target_hyp(target_hypothesis, hypotheses, np_rng) + + gold_label, num_not_ctrl = determine_gold_label( + target_hypothesis, variable_metadata, hypotheses + ) + + return Sample( + variable_metadata=variable_metadata, + hypotheses=hypotheses, + target_hypothesis=target_hypothesis, + sample_metadata=sample_metadata, + # keep track of underlying ground truth in case want more in depth analysis + causal_graph=causal_graph, + gold_label=gold_label, + num_not_ctrl=num_not_ctrl, + ) + + +def determine_gold_label( + target_hyp, variable_metadata, hypotheses +) -> Tuple[Answer, Optional[int]]: + """ + Determines the ideal `Answer` for a given sample. Additionally returns + the number of variables not controlled for, if the hypothesis is valid, + necessary for nDCG calculation. + """ + valid_hypothesis, ind_var, dep_var = parse_target_hyp(target_hyp, variable_metadata) + if not valid_hypothesis: + ctrl_vars, not_ctrls = None, None + num_not_ctrl = None + else: + ctrl_vars, not_ctrls = determine_ctrl_vars( + variable_metadata, ind_var, dep_var, hypotheses + ) + # worst case ctrl: all vars that aren't meant to be ctrld are ctrld + num_not_ctrl = len(not_ctrls) + + return ( + Answer( + valid_hypothesis=valid_hypothesis, + ind_var=ind_var, + dep_var=dep_var, + ctrl_vars=ctrl_vars, + ), + num_not_ctrl, + ) + + +def parse_target_hyp( + target_hyp: Tuple[str, str], variable_metadata: Dict[str, Any] +) -> Tuple[bool, Optional[str], Optional[str]]: + """Implements decision tree in Figure 2 from eval spec""" + proposed_ind = target_hyp[0] + proposed_dep = target_hyp[1] + + ind_unobserved = ( + variable_metadata[proposed_ind]["extra"]["sparsity_rate"] + > constants.SPARSITY_FOR_UNOBS + ) + dep_unobserved = ( + variable_metadata[proposed_dep]["extra"]["sparsity_rate"] + > constants.SPARSITY_FOR_UNOBS + ) + + # if either are unobserved, we have no evidence that they are not correlated + if ind_unobserved or dep_unobserved: + return True, proposed_ind, proposed_dep + # evidence of lack of correlation + elif proposed_dep not in variable_metadata[proposed_ind]["corrs"]: + return False, None, None + # evidence of correlation + else: + return True, proposed_ind, proposed_dep + + +def determine_ctrl_vars( + variable_metadata: Dict[str, Any], + ind_var: str, + dep_var: str, + hypotheses: nx.DiGraph, +) -> Tuple[List[str], List[str]]: + """Implements decision tree in Figure 1 from eval spec""" + ctrl_vars = [] + not_ctrls = [] + for var in variable_metadata: + if var in {ind_var, dep_var}: + not_ctrls.append(var) + elif are_correlated(var, dep_var, variable_metadata) or are_correlated( + var, ind_var, variable_metadata + ): + ctrl_vars.append(var) + elif are_correlated(var, dep_var, variable_metadata) is not None: + # don't control vars which we have observed to be uncorrelated w/ dep + not_ctrls.append(var) + else: # when dep_var or var is unobserved, no evidence of lack of correlation + # control for any var which might influence the dependent variable + dep_var_ancestors = nx.ancestors(hypotheses, dep_var) + if var in dep_var_ancestors: + ctrl_vars.append(var) + else: + not_ctrls.append(var) + + return ctrl_vars, not_ctrls + + +def are_correlated(var_1, var_2, variable_metadata) -> Optional[bool]: + """ + Returns whether two variables are correlated. If there is no evidence + of correlation, returns None. + """ + if ( + variable_metadata[var_1]["extra"]["sparsity_rate"] + > constants.SPARSITY_FOR_UNOBS + or variable_metadata[var_2]["extra"]["sparsity_rate"] + > constants.SPARSITY_FOR_UNOBS + ): + return None + return ( + var_2 in variable_metadata[var_1]["corrs"] + or var_1 in variable_metadata[var_2]["corrs"] + ) + + +def integrate_target_hyp( + target_hyp: Tuple[Any, Any], hyp_graph: nx.DiGraph, np_rng: np.random.Generator +): + """ + Integrates the target hypothesis into the hypotheses graph, respecting + the original edge count by removing a random edge if necessary. + """ + if not hyp_graph.has_edge(*target_hyp): + random_edge_to_remove = np_rng.choice(list(hyp_graph.edges)) + hyp_graph.remove_edge(*random_edge_to_remove) + hyp_graph.add_edge(*target_hyp) + return hyp_graph + + +def gen_samples( + n_samples: int, + signal_noise_ratio: Optional[float], + np_rng: np.random.Generator, + balanced_ctrl_vars: bool = False, +) -> List[Sample]: + samples = [] + if not balanced_ctrl_vars: + for _ in tqdm(range(n_samples)): + sample = gen_sample(signal_noise_ratio, np_rng) + samples.append(sample) + else: + for _ in tqdm(range(n_samples)): + sample = gen_sample_balanced_ctrl_vars(signal_noise_ratio, np_rng) + samples.append(sample) + + return samples + + +def main(args: argparse.Namespace): + np_rng = np.random.default_rng(args.seed) + samples = gen_samples(args.n_samples, args.snr, np_rng, args.balanced_ctrl_vars) + os.makedirs(args.jsonl_dir, exist_ok=True) + if not args.balanced_ctrl_vars: + jsonl_path = os.path.join(args.jsonl_dir, f"{args.n_samples}.jsonl") + else: + jsonl_path = os.path.join( + args.jsonl_dir, f"{args.n_samples}_balanced_ctrl_vars.jsonl" + ) + write_to_jsonl(samples, jsonl_path) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + + parser.add_argument("--n_samples", type=int, default=5000) + parser.add_argument( + "--snr", + type=float, + default=None, + help="signal-to-noise ratio. Default None (no noise is applied.)", + ) + parser.add_argument( + "--jsonl_dir", type=str, default="./evals/registry/data/identifying_variables/" + ) + parser.add_argument("--seed", type=int, default=20220722) + parser.add_argument( + "--balanced_ctrl_vars", + action="store_true", + help="Whether to generate samples with balanced control variables.", + default=False, + ) + args = parser.parse_args() + + main(args) diff --git a/evals/elsuite/identifying_variables/scripts/make_plots.py b/evals/elsuite/identifying_variables/scripts/make_plots.py new file mode 100644 index 0000000000..f29f781492 --- /dev/null +++ b/evals/elsuite/identifying_variables/scripts/make_plots.py @@ -0,0 +1,400 @@ +from pathlib import Path +from typing import Dict, Tuple + +import numpy as np +import pandas as pd +from tqdm.auto import tqdm + +from evals.elsuite.identifying_variables.metrics import compute_metric_posthoc +from evals.elsuite.identifying_variables.scripts.plotting_utils import ( + plot_difficulty_bars, + plot_solver_bars, +) +from evals.elsuite.identifying_variables.scripts.table_utils import ( + make_main_metric_table, +) +from evals.utils import log_utils + +NUM_REPEATS = 3 +MAIN_METRICS = [ + "ctrl_nDCG", + "ctrl_recall", + "hyp_valid_acc", + "ind_acc", + "dep_acc", + "violation_rate", +] + +SOLVERS = [ + "generation/direct/gpt-3.5-turbo", + "generation/cot/gpt-3.5-turbo", + "generation/hhh/gpt-4-base", + "generation/cot_hhh/gpt-4-base", + "generation/direct/gpt-4-1106-preview", + "generation/cot/gpt-4-1106-preview", + "generation/cot/mixtral-8x7b-instruct", + "generation/cot/llama-2-70b-chat", + "generation/cot/gemini-pro", + "identifying_variables/random", + "identifying_variables/noctrl", +] + + +RENDERERS = [ + "markdown", + "csv", + "json", + "language-tabular", + "language-corrset", + "corrset", +] + + +def initialize_default_results_dict(): + results_dict = { + metric: { + stat: { + solver: { + renderer: { + "with tree": ([] if stat == "raw" else 0), + "without tree": ([] if stat == "raw" else 0), + } + for renderer in RENDERERS + } + for solver in SOLVERS + } + for stat in ["raw", "mean", "sem"] + } + for metric in MAIN_METRICS + } + return results_dict + + +def handle_cot_double_sampling(sampling_entries, solver): + if "cot" in solver: + sampling_entries = [ + entry + for entry in sampling_entries + if ( + # for chat models we filter like this + isinstance(entry["prompt"], list) + and entry["prompt"][-1]["content"].startswith( + "Given the above reasoning" + ) + or ( + # for base models we need to filter like this + isinstance(entry["prompt"], str) + and "Given the above reasoning" in entry["prompt"] + ) + ) + ] + return sampling_entries + + +def handle_posthoc_metrics(final_results: Dict, log_path: Path, solver: str): + """ + Computes and includes missing metrics from log file if they are not present + """ + metric_entries = log_utils.extract_individual_results(log_path) + sampling_entries = log_utils.extract_individual_results(log_path, "sampling") + # filter out cot double samplings + sampling_entries = handle_cot_double_sampling(sampling_entries, solver) + # this is necessary because we originally didnt compute recall in the eval + for metric in MAIN_METRICS: + if metric not in final_results.keys(): + final_results[metric] = compute_metric_posthoc( + metric, metric_entries, sampling_entries + ) + + return final_results + + +def populate_default_results_dict(results_dict, results_dir): + for log in tqdm(results_dir.glob("*.log"), total=222): + spec = log_utils.extract_spec(log) + solver = spec["completion_fns"][0] + run_config = spec["run_config"] + renderer = run_config["eval_spec"]["args"]["renderer"] + show_tree = "show_tree=True" in run_config["command"] + tree_key = "with tree" if show_tree else "without tree" + if renderer not in RENDERERS and solver != "identifying_variables/random": + continue + if solver not in SOLVERS: + continue + + final_results = log_utils.extract_final_results(log) + final_results = handle_posthoc_metrics(final_results, log, solver) + + for metric, value in final_results.items(): + if metric in MAIN_METRICS: + results_dict[metric]["raw"][solver][renderer][tree_key].append(value) + raw = results_dict[metric]["raw"][solver][renderer][tree_key] + results_dict[metric]["mean"][solver][renderer][tree_key] = np.mean(raw) + results_dict[metric]["sem"][solver][renderer][tree_key] = np.std( + raw + ) / np.sqrt(NUM_REPEATS) + for metric in results_dict.keys(): + del results_dict[metric]["raw"] + return results_dict + + +def make_default_tables(results_dict: Dict, save_dir: Path): + for metric in tqdm(MAIN_METRICS): + make_main_metric_table(results_dict, metric, SOLVERS, RENDERERS, save_dir) + + +def extract_default_results_dict(results_dir: Path): + results_dict = initialize_default_results_dict() + results_dict = populate_default_results_dict(results_dict, results_dir) + + return results_dict + + +def make_default_plots(results_dict: Dict, save_dir: Path): + all_solvers = list(results_dict["ctrl_nDCG"]["mean"].keys()) + bar_solvers, baseline_solvers = all_solvers[:-2], all_solvers[-2:] + + metrics = ["ctrl_nDCG", "ctrl_recall"] + metric_labels = ["Control Variable Retrieval nDCG*", "Control Variable Recall"] + fig_heights = [6, 5] + + for metric, metric_label, fig_height in tqdm( + zip(metrics, metric_labels, fig_heights) + ): + plot_solver_bars( + bar_solvers, + baseline_solvers, + results_dict[metric], + metric_label, + fig_height, + save_dir / f"{metric}.png", + ) + + +def extract_large_results_dict(results_dir: Path) -> Dict: + ctrl_nDCG_bins = list(range(0, 9)) + results_dict = {} + for log in tqdm(results_dir.glob("*.log"), total=12): + spec = log_utils.extract_spec(log) + final_results = log_utils.extract_final_results(log) + solver = spec["completion_fns"][0] + renderer = spec["split"] + key = f"{solver};{renderer}" + if key not in results_dict: + results_dict[key] = { + bbin: {"raw": [], "mean": None, "sem": None} for bbin in ctrl_nDCG_bins + } + + for bbin in ctrl_nDCG_bins: + results_dict[key][bbin]["raw"].append( + final_results[f"ctrl_nDCG-n_ctrl_vars-{bbin}"] + ) + for key in results_dict.keys(): + for bbin in ctrl_nDCG_bins: + mean = np.mean(results_dict[key][bbin]["raw"]) + sem = np.std(results_dict[key][bbin]["raw"]) / 3 + results_dict[key][bbin]["mean"] = mean + results_dict[key][bbin]["sem"] = sem + del results_dict[key][bbin]["raw"] + + return results_dict + + +def make_large_plot(large_results_dir: Dict, save_dir: Path): + ctrl_vars_bins = list(range(0, 9)) + plot_difficulty_bars( + large_results_dir, ctrl_vars_bins, save_dir / "ctrl_nDCG_difficulty.png" + ) + + +def np_nan_if_none(input_num): + if input_num is None: + return np.nan + else: + return input_num + + +def zero_if_none(input_num): + if input_num is None: + return 0 + else: + return input_num + + +def round_if_not_nan(input_num): + if np.isnan(input_num): + return input_num + else: + return round(input_num) + + +def make_token_per_sample_df(solver_to_eval, solver_to_tokens) -> pd.DataFrame: + tokens_per_sample_df = pd.DataFrame( + index=solver_to_eval.keys(), + columns=[ + "input tokens/sample", + "output tokens/sample", + "total tokens/sample", + ], + ) + for solver in solver_to_tokens.keys(): + # print(solver_to_tokens[solver]) + input_mean = np.nanmean(solver_to_tokens[solver]["input"]) + output_mean = np.nanmean(solver_to_tokens[solver]["output"]) + total_mean = np.nanmean(solver_to_tokens[solver]["total"]) + # print([input_mean, output_mean, total_mean]) + tokens_per_sample_df.loc[solver] = [ + round_if_not_nan(input_mean), + round_if_not_nan(output_mean), + round_if_not_nan(total_mean), + ] + solver_to_index = { + "generation/hhh/gpt-4-base": "HHH GPT-4-base (corrset, no tree)", + "generation/direct/gpt-3.5-turbo": "Direct GPT-3.5-turbo (corrset, no tree)", + "generation/direct/gpt-4-1106-preview": "Direct GPT-4-1106-preview (corrset, no tree)", + "generation/cot_hhh/gpt-4-base": "CoT HHH GPT-4-base (language-tabular, with tree)", + "generation/cot/gpt-3.5-turbo": "CoT GPT-3.5-turbo (language-tabular, with tree)", + "generation/cot/gpt-4-1106-preview": "CoT GPT-4-1106-preview (language-tabular, with tree)", + } + tokens_per_sample_df = tokens_per_sample_df.rename(index=solver_to_index) + return tokens_per_sample_df + + +def count_tokens(results_dir: Path, total) -> Tuple[Dict, pd.DataFrame]: + eval_names = [ + "identifying_variables.corrset.default", + "identifying_variables.language-tabular.default", + ] + solver_names = [ + "generation/hhh/gpt-4-base", + "generation/direct/gpt-3.5-turbo", + "generation/direct/gpt-4-1106-preview", + "generation/cot_hhh/gpt-4-base", + "generation/cot/gpt-3.5-turbo", + "generation/cot/gpt-4-1106-preview", + ] + solver_to_eval = { + solver: eval_names[0] if "cot" not in solver else eval_names[1] + for solver in solver_names + } + solver_to_tree = { + solver: False if "cot" not in solver else True for solver in solver_names + } + solver_to_tokens = { + solver: {"input": [], "output": [], "total": []} for solver in solver_names + } + total_input = 0 + total_output = 0 + for log in tqdm(results_dir.glob("*.log"), total=total): + spec = log_utils.extract_spec(log) + solver = spec["completion_fns"][0] + if solver not in solver_names: + print(f"Skipping {solver}: token counting not supported.") + continue + eval_name = spec["eval_name"] + seed = spec["run_config"]["seed"] + tree = "show_tree=True" in spec["run_config"]["command"] + samplings = log_utils.extract_individual_results(log, "sampling") + samplings = handle_cot_double_sampling(samplings, solver) + for sampling in samplings: + usage = sampling["usage"] + if ( + solver in solver_to_eval + and eval_name == solver_to_eval[solver] + and seed == 1 + and tree != solver_to_tree[solver] + ): + solver_to_tokens[solver]["input"].append( + np_nan_if_none(usage["prompt_tokens"]) + ) + solver_to_tokens[solver]["output"].append( + np_nan_if_none(usage["completion_tokens"]) + ) + solver_to_tokens[solver]["total"].append( + np_nan_if_none(usage["total_tokens"]) + ) + total_input += zero_if_none(usage["prompt_tokens"]) + total_output += zero_if_none(usage["completion_tokens"]) + + total_tokens = {"input": total_input, "output": total_output} + tokens_per_sample_df = make_token_per_sample_df(solver_to_eval, solver_to_tokens) + + return total_tokens, tokens_per_sample_df + + +def make_total_tokens_table(default_total: Dict, large_total: Dict) -> pd.DataFrame: + """ + Makes a dataframe where the index is "default" "large" and the columns are + "input", "output"; showing the total number of input and output tokens for + our experiments on each dataset. + """ + total_tokens_df = pd.DataFrame( + { + "input": [default_total["input"], large_total["input"]], + "output": [default_total["output"], large_total["output"]], + }, + index=["default", "large"], + ) + return total_tokens_df + + +def make_token_count_tables( + default_results_dir: Path, large_results_dir: Path, save_dir: Path +): + default_total_tokens, default_per_sample_tokens_df = count_tokens( + default_results_dir, total=222 + ) + large_total_tokens, _ = count_tokens(large_results_dir, total=12) + + total_tokens_df = make_total_tokens_table(default_total_tokens, large_total_tokens) + + # save the tables + total_tokens_df.to_csv(save_dir / "total_tokens.csv") + default_per_sample_tokens_df.to_csv(save_dir / "per_sample_tokens.csv") + + +def main(default_results_dir: Path, large_results_dir: Path, save_dir: Path): + save_dir.mkdir(parents=True, exist_ok=True) + + print("Parsing default dataset results...") + default_results_dict = extract_default_results_dict(default_results_dir) + print("Making default dataset tables...") + make_default_tables(default_results_dict, save_dir) + print("Making default dataset plots...") + make_default_plots(default_results_dict, save_dir) + + print("Parsing large dataset results...") + large_results_dict = extract_large_results_dict(large_results_dir) + print("Making large dataset plot...") + make_large_plot(large_results_dict, save_dir) + + print("Making token count tables...") + make_token_count_tables(default_results_dir, large_results_dir, save_dir) + print("Done.") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description="Process results") + parser.add_argument( + "--default_results_dir", + type=str, + help="Path to directory containing .log files from experiments on default dataset", + ) + parser.add_argument( + "--large_results_dir", + type=str, + help="Path to directory containing .log files from experiments on large dataset", + ) + parser.add_argument( + "--save_dir", type=str, help="Path to directory to save plots and tables to" + ) + + args = parser.parse_args() + + main( + Path(args.default_results_dir), + Path(args.large_results_dir), + Path(args.save_dir), + ) diff --git a/evals/elsuite/identifying_variables/scripts/plotting_utils.py b/evals/elsuite/identifying_variables/scripts/plotting_utils.py new file mode 100644 index 0000000000..1c80aab042 --- /dev/null +++ b/evals/elsuite/identifying_variables/scripts/plotting_utils.py @@ -0,0 +1,163 @@ +from typing import Dict, Iterable, List +from pathlib import Path + +import numpy as np +import matplotlib.pyplot as plt +import seaborn as sns + + +renderers_of_interest = ["csv", "language-corrset"] + +renderer_to_label = { + "csv": "CSV observations", + "language-corrset": "Correlation set", +} + +cmap = plt.get_cmap("Paired") +colors = np.array([cmap(i) for i in range(len(renderers_of_interest))]) +renderer_to_color = {r: c for r, c in zip(renderers_of_interest, colors)} + +solver_to_label = { + "generation/direct/gpt-3.5-turbo": "Direct gpt-3.5-turbo", + "generation/cot/gpt-3.5-turbo": "CoT gpt-3.5-turbo", + "generation/hhh/gpt-4-base": "HHH gpt-4-base", + "generation/cot_hhh/gpt-4-base": "CoT HHH gpt-4-base", + "generation/direct/gpt-4-1106-preview": "Direct gpt-4-1106-preview", + "generation/cot/gpt-4-1106-preview": "CoT gpt-4-1106-preview", + "generation/cot/mixtral-8x7b-instruct": "CoT mixtral-8x7b-instruct\n(Correlation set only)", + "generation/cot/llama-2-70b-chat": "CoT llama-2-70b-chat\n(Correlation set only)", + "generation/cot/gemini-pro": "CoT gemini-pro-1.0\n(Correlation set only)", + "identifying_variables/random": "Random baseline", + "identifying_variables/noctrl": "NoCtrl baseline", +} + +baseline_to_linestyle = { + "identifying_variables/random": "--", + "identifying_variables/noctrl": "-.", +} + +cmap = plt.get_cmap("Set2") +bline_colors = np.array( + [cmap(i) for i in range(0, len(baseline_to_linestyle.keys()) + 0)] +) +baseline_to_color = { + key: color for key, color in zip(baseline_to_linestyle.keys(), bline_colors) +} + + +def plot_solver_bars( + bar_solvers: List[str], + baseline_solvers: List[str], + metric_results: Dict, + metric_label: str, + fig_height: int, + output_path: Path, +): + """ + Plots a side-by-side bar plot of the metric results, showing the + solvers on the x axis and the metric value on the y axis. + + Args: + bar_solvers: The names of solvers to plot. + baseline_solvers: The names of the baseline solvers to plot. + metric_results: A dictionary with k: v of format solver : {mean: value, sem: value} + metric_label: The label for the y axis + fig_height: the height of the figure in inches + output_path: the path to save the figure to + """ + sns.set_context("paper") + sns.set_style("whitegrid") + + bar_width = 0.3 + positions = np.arange(len(bar_solvers)) + + f, ax = plt.subplots(1, 1, dpi=300, figsize=(9, fig_height)) + + for i, renderer in enumerate(renderers_of_interest): + bars = [ + metric_results["mean"][solver][renderer]["without tree"] + for solver in bar_solvers + ] + errors = [ + metric_results["sem"][solver][renderer]["without tree"] + for solver in bar_solvers + ] + + ax.bar( + positions + bar_width * i, + bars, + bar_width, + yerr=errors, + label=renderer_to_label[renderer], + color=renderer_to_color[renderer], + ) + + for baseline_solver in baseline_solvers: + mean = metric_results["mean"][baseline_solver]["corrset"]["without tree"] + sem = metric_results["sem"][baseline_solver]["corrset"]["without tree"] + ax.axhline( + mean, + label=solver_to_label[baseline_solver], + color=baseline_to_color[baseline_solver], + linestyle=baseline_to_linestyle[baseline_solver], + ) + ax.axhspan( + mean - sem, mean + sem, alpha=0.1, color=baseline_to_color[baseline_solver] + ) + + ax.set_xticks( + positions + bar_width / 2, + [solver_to_label[s] for s in bar_solvers], + rotation=45, + ha="right", + ) + ax.tick_params( + axis="x", which="both", bottom=True + ) # Show both major and minor xticks + ax.set_ylabel(metric_label) + ax.set_ylim(-0.005, 1) + ax.xaxis.grid(False) + ax.legend() + f.set_tight_layout(True) + plt.savefig(output_path, dpi=300, bbox_inches="tight") + + +def plot_difficulty_bars(results_dict: Dict, bins: Iterable[int], output_path: Path): + sns.set_context("paper") + sns.set_style("whitegrid") + + f, ax = plt.subplots(1, 1, dpi=300, figsize=(7, 4)) + + positions = np.arange(len(bins)) + bar_width = 0.4 + + for i, key in enumerate(sorted(results_dict.keys())): + solver, renderer = key.split(";") + bars = [results_dict[key][bbin]["mean"] for bbin in bins] + errors = [results_dict[key][bbin]["sem"] for bbin in bins] + if solver == "generation/direct/gpt-4-1106-preview": + label = renderer_to_label[renderer] + color = renderer_to_color[renderer] + ax.bar( + positions + bar_width * i, + bars, + bar_width, + yerr=errors, + label=label, + color=color, + ) + + ax.set_xlabel("Number of necessary control variables") + ax.set_ylabel("Control Variable Retrieval nDCG*") + + ax.set_xlim(-0.3, 8.7) + ax.set_ylim(0, 1) + ax.xaxis.grid(False) + ax.legend() + ax.set_xticks(positions + bar_width / 2, bins) + f.set_tight_layout(True) + plt.savefig( + output_path, + dpi=300, + bbox_inches="tight", + ) diff --git a/evals/elsuite/identifying_variables/scripts/run_experiments.sh b/evals/elsuite/identifying_variables/scripts/run_experiments.sh new file mode 100755 index 0000000000..fae5ceb93b --- /dev/null +++ b/evals/elsuite/identifying_variables/scripts/run_experiments.sh @@ -0,0 +1,105 @@ +#!/bin/bash + +# Function to display usage +usage() { + echo "Usage: $0 -s size -l logdir" + echo " -s size Specify the size of the experiments (options: 'balanced-hypotheses', 'balanced-ctrl', 'balanced-hypotheses-large', 'balanced-ctrl-large')" + echo " -l logdir Specify the directory for log files" + exit 1 +} + +# Check if no arguments were provided +if [ $# -eq 0 ]; then + usage + exit 1 +fi + +# Parse command-line options +while getopts 's:l:' flag; do + case "${flag}" in + s) size=${OPTARG} ;; + l) logdir=${OPTARG} ;; + *) usage ;; + esac +done + +# Check if mandatory arguments were provided +if [ -z "$size" ] || [ -z "$logdir" ]; then + usage + exit 1 +fi + +logdirbase=$logdir +NUM_REPEATS=3 + +# Function to run experiments +run_experiments() { + local size=$1 + local logpathbase="${logdirbase}/${size}" + local start_time=$SECONDS + + # Define RENDERERS and SOLVERS array based on size + declare -a RENDERERS + declare -a SOLVERS + if [ "$size" == "balanced-hypotheses" ]; then + RENDERERS=("markdown" "csv" "json" "language-tabular" "language-corrset" "corrset") + SOLVERS=("generation/direct/gpt-3.5-turbo" + "generation/cot/gpt-3.5-turbo" + "generation/hhh/gpt-4-base" + "generation/cot_hhh/gpt-4-base" + "generation/direct/gpt-4-1106-preview" + "generation/cot/gpt-4-1106-preview") + elif [ "$size" == "balanced-ctrl" ]; then + RENDERERS=("csv" "language-corrset") + SOLVERS=("generation/direct/gpt-3.5-turbo" + "generation/cot/gpt-3.5-turbo" + "generation/hhh/gpt-4-base" + "generation/cot_hhh/gpt-4-base" + "generation/direct/gpt-4-1106-preview" + "generation/cot/gpt-4-1106-preview") + else + RENDERERS=("csv" "language-corrset") + SOLVERS=("generation/direct/gpt-4-1106-preview") + fi + + # Main loop + for ((i = 1; i <= NUM_REPEATS; i++)); do + for solver in "${SOLVERS[@]}"; do + for renderer in "${RENDERERS[@]}"; do + run_solver $solver $renderer $size $i "$logpathbase" + done + done + run_solver "identifying_variables/random" "corrset" $size $i "$logpathbase" + run_solver "identifying_variables/noctrl" "corrset" $size $i "$logpathbase" + done + + local end_time=$SECONDS + echo "Done running experiments for $size size, all logs in $logpathbase" + echo "Total execution time: $((end_time - start_time)) seconds." +} + +# Function to run a single solver +run_solver() { + local solver=$1 + local renderer=$2 + local size=$3 + local seed=$4 + local logpathbase=$5 + local solver_dotted=${solver//\//.} + + local record_path="${logpathbase}/${solver_dotted}_${renderer}_${size}_${seed}" + echo "Running $solver with $renderer renderer and $size data size; seed $seed" + + local sub_start_time=$(date +%s) + oaieval "$solver" "identifying_variables.${renderer}.${size}" --record_path "$record_path.log" --seed $seed + local sub_end_time=$(date +%s) + echo "${solver_dotted}_${renderer}_${size} execution time: $((sub_end_time - sub_start_time)) seconds." + + skip_tree_solvers=("identifying_variables/random" "identifying_variables/noctrl") + if [[ ! "${skip_tree_solvers[@]}" =~ "$solver" ]] && [ "$size" == "balanced-hypotheses" ]; then + echo "Now repeating with show_tree=True" + oaieval "$solver" "identifying_variables.${renderer}.${size}" --extra_eval_params show_tree=True --record_path "${record_path}_tree.log" --seed $seed + fi +} + +run_experiments "${size}" diff --git a/evals/elsuite/identifying_variables/scripts/table_utils.py b/evals/elsuite/identifying_variables/scripts/table_utils.py new file mode 100644 index 0000000000..3991cd469b --- /dev/null +++ b/evals/elsuite/identifying_variables/scripts/table_utils.py @@ -0,0 +1,66 @@ +from typing import Dict, List +from pathlib import Path + +import numpy as np +import pandas as pd + + +def make_main_metric_table( + results_dict: Dict, + metric: str, + solvers: List[str], + renderers: List[str], + save_dir: Path, +): + """ + Makes and saves a table containing the information of performance of + each solver for each renderer for each variant of the eval on + a given metric. + - Table rows are solvers; they are multi-rows, so each row has two subrows: with + tree and without tree + - Table columns are renderers; they are multi-columns, so each column has two + subcolumns: mean and sem (standard error of the mean) + + Args: + results_dict: dictionary containing the results of the eval. See + `initialize_default_results_dict` and `populate_default_results_dict` in + `process_results.py`. + metric: the name of the metric we want to make the table for + solvers: list of solvers we want to include in the table + renderers: list of renderers we want to include in the table + save_dir: directory to save the table in (as a CSV file) + """ + + # only keep keep metric in results_dict + filtered_results_dict = results_dict[metric] + # flatten into tuples + data_tuples = [] + for stat, solver_data in filtered_results_dict.items(): + for solver, renderer_data in solver_data.items(): + for renderer, tree_data in renderer_data.items(): + for tree_type, value in tree_data.items(): + if value is not None: + data_tuples.append((solver, tree_type, renderer, stat, value)) + + df = pd.DataFrame( + data_tuples, columns=["Solver", "Tree", "Renderer", "Stat", "Value"] + ) + df = df.pivot_table( + index=["Solver", "Tree"], columns=["Renderer", "Stat"], values="Value" + ) + # sorting by solvers, renderers (for some reason ordering is lost in the above process) + new_index = [ + (solver, tree) for solver in solvers for tree in ["with tree", "without tree"] + ] + new_columns = pd.MultiIndex.from_product( + [renderers, df.columns.levels[1]], names=df.columns.names + ) + df = df.reindex(new_index, columns=new_columns) + + # delete the with tree rows for the treeless solvers + for solver in solvers[-2:]: + df.drop((solver, "with tree"), inplace=True) + + # save table + save_path = save_dir / f"{metric}_table.csv" + df.to_csv(save_path) diff --git a/evals/elsuite/identifying_variables/solvers.py b/evals/elsuite/identifying_variables/solvers.py new file mode 100644 index 0000000000..c6010c74da --- /dev/null +++ b/evals/elsuite/identifying_variables/solvers.py @@ -0,0 +1,48 @@ +import random + +from evals.solvers.solver import Solver, SolverResult +from evals.task_state import TaskState + + +class RandomSolver(Solver): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + def _solve(self, task_state: TaskState) -> SolverResult: + valid_hyp = random.uniform(0, 1) < 0.5 + + variables = task_state.current_state["variables"] + n_vars_to_sample = random.randint(2, len(variables)) + ind_var, dep_var, *ctrl_vars = random.sample(variables, n_vars_to_sample) + if len(ctrl_vars) == 0: + ctrl_vars = "none" + else: + ctrl_vars = ", ".join(ctrl_vars) + + solver_string = f"[@ANSWER valid_hyp: {valid_hyp}; independent: {ind_var}; dependent: {dep_var}; control: {ctrl_vars}]" + + return SolverResult(output=solver_string) + + +class NoCtrl(Solver): + """ + Solver that always returns no control variables + (i.e. "none", interpreted as an empty list by the eval) + what it returns for the other variables is arbitrary + """ + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + def _solve(self, task_state: TaskState) -> SolverResult: + # we don't care about valid_hyp and ind/dep vars for this solver + # it's only used for the ctrl variables subtask + valid_hyp = True + variables = task_state.current_state["variables"] + ind_var, dep_var = random.sample(variables, 2) + + # it just always returns no control variables + ctrl_vars = "none" + solver_string = f"[@ANSWER valid_hyp: {valid_hyp}; independent: {ind_var}; dependent: {dep_var}; control: {ctrl_vars}]" + + return SolverResult(output=solver_string) diff --git a/evals/elsuite/identifying_variables/structs.py b/evals/elsuite/identifying_variables/structs.py new file mode 100644 index 0000000000..90b47b96b0 --- /dev/null +++ b/evals/elsuite/identifying_variables/structs.py @@ -0,0 +1,49 @@ +"""Custom data structures for the eval""" +from dataclasses import dataclass +from typing import Dict, List, Optional, Tuple + +import networkx as nx + + +@dataclass +class Answer: + valid_hypothesis: bool + ind_var: Optional[str] + dep_var: Optional[str] + ctrl_vars: Optional[List[str]] + + +@dataclass +class Sample: + """ + A sample of the dataset for the eval. + + Args: + variable_metadata (Dict) : A dictionary mapping each variable name to its metadata. + Each variable's metadata is a dictionary containing: + - 'gen_method': A dictionary specifying the generation method for the + variable, including: + - 'name': Name of the latent function or distribution. + - 'input_x': Name of the input variable, if applicable. + - 'kwargs': Additional arguments for the latent function. + - 'corrs': A set of variables correlated with this variable. + hypotheses (nx.DiGraph): A directed acyclic graph (DAG) representing the hypotheses. + target_hypothesis (Tuple[str, str]) A tuple (independent_variable, dependent_variable) + representing the hypothesis of interest. + sample_metadata (Dict): A dictionary with additional metadata, including: + - 'num_obs_samples': Number of observations generated per variable. + - 'snr': Signal-to-noise ratio applied to the observations. + causal_graph (nx.DiGraph): A randomly generated DAG representing the underlying + causal relationships among variables. Represented as nx.DiGraph. + gold_label (Answer): The gold label for the sample. + num_not_ctrl (Optional[int]): The number of variables not controlled for. None + if the hypothesis is invalid. + """ + + variable_metadata: Dict + hypotheses: nx.DiGraph + target_hypothesis: Tuple[str, str] + sample_metadata: Dict + causal_graph: nx.DiGraph + gold_label: Answer + num_not_ctrl: Optional[int] diff --git a/evals/elsuite/identifying_variables/utils.py b/evals/elsuite/identifying_variables/utils.py new file mode 100644 index 0000000000..6918926bdf --- /dev/null +++ b/evals/elsuite/identifying_variables/utils.py @@ -0,0 +1,91 @@ +import re +from typing import Dict + +import networkx as nx +import numpy as np + +from evals.elsuite.identifying_variables.structs import Answer, Sample +from evals.solvers.solver import SolverResult + + +def parse_solver_preds(solver_result: SolverResult) -> Answer: + solver_string = solver_result.output.strip().lower() + + pattern = ( + r"\[@answer " # Matches the beginning of the answer + r"valid_hyp: (true|false|True|False)" # valid hyp part + r"(?:; independent: ([^;]*))?" # Optionally matches the independent part + r"(?:; dependent: ([^;]*))?" # Optionally matches the dependent part + r"(?:; control: ([^\]]*))?" # Optionally matches the control part + r"\]" # Matches the end of the answer + ) + + match = re.search(pattern, solver_string) + + if match: + valid_hyp = match.group(1).lower() == "true" + if not valid_hyp: + return Answer( + valid_hypothesis=False, + ind_var=None, + dep_var=None, + ctrl_vars=None, + ) + ind_var = match.group(2) + ind_var = ind_var if ind_var is not None else "WRONG" + dep_var = match.group(3) + dep_var = dep_var if dep_var is not None else "WRONG" + ctrl_vars = match.group(4) + if ctrl_vars is not None: + ctrl_vars = ctrl_vars.split(",") + ctrl_vars = [var.strip() for var in ctrl_vars] + if ctrl_vars[0].lower().strip("\"'`«»<>") == "none": + ctrl_vars = [] + else: + ctrl_vars = ["WRONG"] + return Answer( + valid_hypothesis=True, + ind_var=ind_var, + dep_var=dep_var, + ctrl_vars=ctrl_vars, + ) + else: + raise ValueError("Invalid solver output") + + +def sample_serializer(obj): + """ + Custom serializer to pass to json.dumps when + saving a sample dictionary to jsonl + """ + if isinstance(obj, set): + return list(obj) + elif isinstance(obj, nx.DiGraph): + return nx.to_dict_of_lists(obj) + elif isinstance(obj, np.integer): + return int(obj) + elif isinstance(obj, np.floating): + return float(obj) + + +def json_to_sample(serialized_sample: Dict) -> Sample: + """Reads sample from jsonl into Sample dataclass""" + hypotheses = nx.from_dict_of_lists(serialized_sample["hypotheses"], create_using=nx.DiGraph) + causal_graph = nx.from_dict_of_lists(serialized_sample["causal_graph"], create_using=nx.DiGraph) + gold_label = Answer(**serialized_sample["gold_label"]) + + # convert corrs in variable_metadata from lists to sets + for var in serialized_sample["variable_metadata"]: + serialized_sample["variable_metadata"][var]["corrs"] = set( + serialized_sample["variable_metadata"][var]["corrs"] + ) + + return Sample( + variable_metadata=serialized_sample["variable_metadata"], + hypotheses=hypotheses, + target_hypothesis=serialized_sample["target_hypothesis"], + sample_metadata=serialized_sample["sample_metadata"], + causal_graph=causal_graph, + gold_label=gold_label, + num_not_ctrl=serialized_sample["num_not_ctrl"], + ) diff --git a/evals/elsuite/incontext_rl/README.md b/evals/elsuite/incontext_rl/README.md new file mode 100644 index 0000000000..69dbde303b --- /dev/null +++ b/evals/elsuite/incontext_rl/README.md @@ -0,0 +1,74 @@ +# In-Context RL + +This eval tests models' ability to solve RL environments simply by interacting with them in-context, without dedicated training or fine-tuning. + +## Usage + +Run with: + +```bash +oaieval incontext_rl +``` + +For examples of tested solvers, see [`./scripts/run_experiments.sh`](./scripts/run_experiments.sh). + +## Dataset + +The eval is currently set up to test models on the following canonical RL environments: +1. [FrozenLake-v1](https://gymnasium.farama.org/environments/toy_text/frozen_lake/) (non-slippery version, default map), 4x4 gridworld where the agent has to reach the goal without falling into traps. +2. [CliffWalking-v0](https://gymnasium.farama.org/environments/toy_text/cliff_walking/). 4x12 gridworld where the agent has to reach the other side of the map without falling off a cliff. +3. [BanditTwoArmedHighLowFixed-v1](https://github.com/james-aung/gymasium-bandits). Stochastic two-armed bandit setup where Arm 1 pays out 80% of the time with reward 1, and Arm 2 pays out 20% of the time with reward 1. +4. [BanditTenArmedRandomFixed-v1](https://github.com/james-aung/gymasium-bandits). Stochastic ten-armed bandit setup where each arm has some randomly-initialized probability of payout. + +Besides these four environments, our eval is also built to be compatible with any environments that have discrete action and observation spaces using the Gymnasium API. Future work may generalize our eval to work with environments with other types of action/observation spaces. + +## Evaluation Process + +Each run of the eval tests the model on all four environments in the dataset, and has the model take steps in each environment until 200 steps are taken or the model’s context limit is reached. + +At each step, the eval provides the following to the model: +- The next observation and the reward from the last action. The model is also told when the environment has reset due to its action leading to a termination. +- How many of the maximum number of steps it has already taken. +- The total reward it has accumulated so far across all episodes. + +If an episode ends, the environment resets and a new episode begins. + +If the eval receive 4 responses in a row where we cannot parse an action selection, we end the evaluation for that environment. (This provides a natural end for runs where the model’s context window is exceeded.) + + +## Prompts + +We refer readers to the [`./defaults.py`](./defaults.py) file for the `TASK_DESCRIPTION` and other prompts used in the eval. + +## Metrics + +We provide the following metrics per evaluated environment: + +| **Metric** | **Notes** | +|---|---| +| `average_episode_reward` | The average reward achieved per episode | +| `total_steps` | The number of steps taken across all episodes before the environment sample ended | +| `invalid_response_rate` | % of responses that were in an invalid format for the eval | + + +## Token Usage Estimates + + +| Model | Token Usage Per Run | +|---|---| +| **gpt-3.5-turbo** | 4200000 ± 400000 | +| **gpt-4-turbo-preview** | 21900000 ± 10100000 | +| **mixtral-8x7b** | 2700000 ± 800000 | + + +## Future modifications + +- Extend the eval to work with other observation and action spaces beyond Discrete spaces + +## Version History + +- v0: Initial version released + +## Contribution Statement + +Eval design, implementation, and results evaluation were primarily conducted by James Aung. Chan Jun Shern was responsible for code reviews throughout the implementation process, along with fine-grained feedback on the project in general. Additional guidance was provided by Steven Adler, who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation. \ No newline at end of file diff --git a/evals/elsuite/incontext_rl/anti-cot_solver.py b/evals/elsuite/incontext_rl/anti-cot_solver.py new file mode 100644 index 0000000000..40b3997e3c --- /dev/null +++ b/evals/elsuite/incontext_rl/anti-cot_solver.py @@ -0,0 +1,38 @@ +from typing import Any +from evals.solvers.solver import NestedSolver, Solver, SolverResult, SolverSpec +from evals.task_state import Message, TaskState + +ANTI_COT_TEMPLATE = "RESPOND ONLY WITH YOUR FINAL ANSWER IN THE FORMAT REQUESTED. DO NOT OUTPUT ANY ADDITIONAL REASONING OR TEXT." + +class AntiCoTSolver(NestedSolver): + """ + Instructs the model to not do any further reasoning and just respond with the final answer. + """ + + def __init__( + self, + solver: SolverSpec, + registry: Any = None, + ): + super().__init__(solver=solver) + + @property + def solver(self) -> Solver: + return self.get_solver("solver") + + def _solve( + self, + task_state: TaskState, + **kwargs, + ) -> SolverResult: + task_state.messages += ( + [ + Message(role="system", content=ANTI_COT_TEMPLATE), + ] + ) + solver_result = self.solver(task_state=task_state, **kwargs) + return solver_result + + @property + def name(self) -> str: + return f"Anti-CoT_{self.solver.name}" diff --git a/evals/elsuite/incontext_rl/baselines.py b/evals/elsuite/incontext_rl/baselines.py new file mode 100644 index 0000000000..34f65d6caf --- /dev/null +++ b/evals/elsuite/incontext_rl/baselines.py @@ -0,0 +1,118 @@ +import random + +import numpy as np + +from evals.elsuite.incontext_rl.eval import CurrentState +from evals.record import record_sampling +from evals.solvers.solver import Solver, SolverResult +from evals.task_state import TaskState + + +class RandomSolver(Solver): + def __init__(self, *args, **kwargs): + pass + + def _solve( + self, + task_state: TaskState, + **kwargs, + ) -> SolverResult: + + cs: CurrentState = task_state.current_state + + try: + action = cs.action_space.sample() + response = f"[SELECT: {action}]" + except Exception as e: + response = f"Error: {e}" + + record_sampling( + prompt=cs.observations[-1], + sampled=response, + model="incontext_rl_random", + ) + + return SolverResult(response) + + +class QlearningSolver(Solver): + def __init__( + self, + learning_rate=0.7, + gamma=0.95, + epsilon=1.0, + min_epsilon=0.05, + max_epsilon=1.0, + decay_rate=0.0005, + *args, + **kwargs, + ): + super().__init__(*args, **kwargs) + self.learning_rate = learning_rate + self.gamma = gamma + self.epsilon = epsilon + self.min_epsilon = min_epsilon + self.max_epsilon = max_epsilon + self.decay_rate = decay_rate + self.q_table = None + + def initialize_q_table(self, observation_space_size, action_space_size): + self.q_table = np.zeros((observation_space_size, action_space_size)) + + def select_action(self, state, action_space): + if random.uniform(0, 1) < self.epsilon: + return action_space.sample() # Explore action space + else: + return np.argmax(self.q_table[state][:]) # Exploit learned values + + def update_q_table(self, state, action, reward, next_state): + next_max = np.max(self.q_table[next_state]) + self.q_table[state, action] = self.q_table[state, action] + self.learning_rate * ( + reward + self.gamma * next_max - self.q_table[state, action] + ) + + def reduce_epsilon(self, episode_number): + self.epsilon = self.min_epsilon + (self.max_epsilon - self.min_epsilon) * np.exp( + -self.decay_rate * episode_number + ) + + def _solve(self, task_state: TaskState, **kwargs) -> SolverResult: + + cs: CurrentState = task_state.current_state + + # TODO these might not be true if environment is not discrete + assert ( + cs.observation_space_n is not None + ), "Environment must have discrete observation space" + assert cs.action_space_n is not None, "Environment must have discrete action space" + + if self.q_table is None: + print("Initializing Q-table") + self.initialize_q_table( + observation_space_size=cs.observation_space_n, action_space_size=cs.action_space_n + ) + + # This shouln't run on the first step + if len(cs.actions) >= 1 and len(cs.rewards) >= 1 and len(cs.observations) >= 2: + print(cs.actions) + self.update_q_table( + state=cs.observations[-2], + action=cs.actions[-1], + reward=cs.rewards[-1], + next_state=cs.observations[-1], + ) + print( + f"The last action {cs.actions[-1]} resulted in reward {cs.rewards[-1]}. We went from state {cs.observations[-2]} to state {cs.observations[-1]}" + ) + self.reduce_epsilon(episode_number=len(cs.episode_end_steps)) + + action = self.select_action(state=cs.observations[-1], action_space=cs.action_space) + response = f"[SELECT: {action}]" + + record_sampling( + prompt=cs.observations[-1], + sampled=response, + model="incontext_rl_qlearning", + ) + + return SolverResult(response) diff --git a/evals/elsuite/incontext_rl/defaults.py b/evals/elsuite/incontext_rl/defaults.py new file mode 100644 index 0000000000..2f1cb06fc2 --- /dev/null +++ b/evals/elsuite/incontext_rl/defaults.py @@ -0,0 +1,30 @@ +from string import Template + +task_description_template = Template("""$explanations + +You have the following $action_space actions available to you: $actions. + +On each turn, you will receive an observation from the environment, which maybe be one of the following $observation_space possible observations: +$observations +Each action you take will give you a reward (which might be negative) and may move you to a new state. + +You should take actions such that you accumulate the highest possible reward across all your actions. + +This is how we will play the game: +1. You will consider all the information you have received to decide upon the best action you can take now. +2. You will select your action from the list above by specifying the number key of the action in the command [SELECT: x], where x is the number key of the action. +3. Your selected action will be taken. +4. As a result of your action, you will be given an observation from the environment and you may receive some reward. +5. Repeat from step 1. +""") + +step_counter = Template("Total actions taken so far: $step_count") +reward_counter = Template("Total reward so far: $reward_count") +reset_msg = Template("""After the game reset you are now in $observation. +Please pick an action, providing your reasoning. You must format your final action choice as [SELECT: x]""") +step_result = Template("""You took Action $action. You are now in $next_observation. +The last step you did provided reward: $reward. +Please pick an action, providing your reasoning. You must format your final action choice as [SELECT: x]""") +step_result_reset = Template("""You took Action $action. You arrived at $next_observation. +The last step made the game reset. +The last step you did provided reward: $reward.""") \ No newline at end of file diff --git a/evals/elsuite/incontext_rl/env_setup.py b/evals/elsuite/incontext_rl/env_setup.py new file mode 100644 index 0000000000..31ffcba534 --- /dev/null +++ b/evals/elsuite/incontext_rl/env_setup.py @@ -0,0 +1,12 @@ +""" +Optional setup scripts for specific environments. +""" + +def setup_GymnasiumBandits(): + import gymnasium_bandits + return + +ENV_SETUP_FUNCS = { + "BanditTwoArmedHighLowFixed-v0": setup_GymnasiumBandits, + "BanditTenArmedRandomFixed-v0": setup_GymnasiumBandits, +} \ No newline at end of file diff --git a/evals/elsuite/incontext_rl/eval.py b/evals/elsuite/incontext_rl/eval.py new file mode 100644 index 0000000000..a1fac2101e --- /dev/null +++ b/evals/elsuite/incontext_rl/eval.py @@ -0,0 +1,299 @@ +import logging +import random +import re +from dataclasses import dataclass, field +from typing import Any, List, Optional + +import gymnasium as gym +import numpy as np + +import evals +from evals.api import CompletionFn +from evals.elsuite.incontext_rl.defaults import ( + reset_msg, + reward_counter, + step_counter, + step_result, + step_result_reset, + task_description_template, +) +from evals.elsuite.incontext_rl.env_setup import ENV_SETUP_FUNCS +from evals.eval import SolverEval +from evals.solvers.solver import Solver +from evals.task_state import Message, TaskState + +logger = logging.getLogger(__name__) + + +@dataclass +class CurrentState: + action_space: gym.Space + observation_space: gym.Space + action_space_n: int + observation_space_n: int + invalid_responses: int = 0 + total_responses: int = 0 + actions: List = field(default_factory=list) + rewards: List[float] = field(default_factory=list) + observations: List = field(default_factory=list) + episode_end_steps: List[int] = field(default_factory=list) + + +class InContextRl(SolverEval): + def __init__( + self, + completion_fns: list[CompletionFn], + max_steps: int = 200, # maximum possible steps per sample, optional + max_invalid_responses: int = 4, # maximum invalid responses from Solver before terminating sample + max_num_messages_allowed: int = 2048, # maximum number of messages allowed by OpenAI API + use_explanations: bool = False, # Whether to include a key for how to understand action and observation spaces + *args, + **kwargs, + ): + super().__init__(completion_fns, *args, **kwargs) + self.max_steps = max_steps + self.max_invalid_responses = max_invalid_responses + self.use_explanations = use_explanations + self.max_num_messages_allowed = max_num_messages_allowed + + def eval_sample(self, solver: Solver, sample: Any, rng: random.Random): + + # Validate sample + required_keys = ["env", "env_id", "explanations"] + assert all( + key in sample for key in required_keys + ), f"Sample missing required keys: {required_keys}" + assert isinstance(sample["env"], gym.Env) + assert isinstance(sample["env_id"], str) + assert isinstance(sample["explanations"], str) + + env = sample["env"] + ts = TaskState( + task_description=self._generate_task_description(env, sample), + messages=[], + current_state=CurrentState( + action_space=env.action_space, + observation_space=env.observation_space, + action_space_n=env.action_space.n, # TODO might not be available for all envs, check when adding a continuous env + observation_space_n=env.observation_space.n, # TODO might not be available for all envs, check when adding a continuous env + ), + ) + + # Reset environment and update task state + observation, _ = env.reset(seed=42) + ts.current_state.observations.append(observation) + + # Tell model starting observation and ask it to pick an action + self._add_reset_message_to_task_state(ts, observation, sample) + + for _ in range(self.max_steps): + self._add_recap_message_to_task_state( + ts, ts.current_state.actions, ts.current_state.rewards + ) + + action = self._try_get_valid_action(solver, ts, env.action_space.n) + + if action is None: + logger.info("Ending sample since couldn't parse an action.") + break + else: + next_observation, reward, terminated, truncated, _ = env.step(action) + ts.current_state.actions.append(action) + ts.current_state.rewards.append(float(reward)) + ts.current_state.observations.append(next_observation) + + if terminated or truncated: + # Tell model that episode ended and what reward was received + content = self._format_step_message( + action, next_observation, reward, sample, terminated=True + ) + ts.messages += [Message(role="user", content=content)] + + # Log what step the episode ended on + ts.current_state.episode_end_steps.append(len(ts.current_state.actions)) + + # Reset environment + observation, _ = env.reset(seed=42) + ts.current_state.observations.append(observation) + + # Tell model new observation after reset and ask it to pick an action + self._add_reset_message_to_task_state(ts, observation, sample) + else: + content = self._format_step_message(action, next_observation, reward, sample) + ts.messages += [Message(role="user", content=content)] + + env.close() + + episode_rewards = self._calculate_episode_rewards( + ts.current_state.episode_end_steps, ts.current_state.rewards + ) + evals.record.record_metrics( + environment=f"{env.spec.id} {env.spec.kwargs}", + explanations=self.use_explanations, + total_return=sum(ts.current_state.rewards), + total_steps=len(ts.current_state.actions), + actions=ts.current_state.actions, + rewards=ts.current_state.rewards, + episode_rewards=episode_rewards, + average_episode_reward=float(np.mean(episode_rewards)), + average_reward_last_5_episodes=float(np.mean(episode_rewards[-5:])), + average_reward_last_10_episodes=float(np.mean(episode_rewards[-10:])), + average_reward_last_20_episodes=float(np.mean(episode_rewards[-20:])), + average_reward_last_50_episodes=float(np.mean(episode_rewards[-50:])), + invalid_response_rate=ts.current_state.invalid_responses + / ts.current_state.total_responses + if ts.current_state.total_responses > 0 + else 0, + episode_end_steps=ts.current_state.episode_end_steps, + ) + + def run(self, recorder: evals.record.Recorder): + samples = self.get_samples() + for sample in samples: + # Create environments and pass them to each thread via the sample + # (gym envs don't like being created in the thread itself) + sample["env"] = self._make_env(sample) + self.eval_all_samples(recorder, samples) + + metrics = recorder.get_metrics() + + results = [] + + for metric in metrics: + env_result = { + "env": metric["environment"], + "metrics": { + "explanations": metric["explanations"], + "average_episode_reward": metric["average_episode_reward"], + "average_reward_last_5_episodes": metric["average_reward_last_5_episodes"], + "average_reward_last_10_episodes": metric["average_reward_last_10_episodes"], + "average_reward_last_20_episodes": metric["average_reward_last_20_episodes"], + "average_reward_last_50_episodes": metric["average_reward_last_50_episodes"], + "episode_rewards": metric["episode_rewards"], + "total_return": metric["total_return"], + "total_steps": metric["total_steps"], + "actions": metric["actions"], + "rewards": metric["rewards"], + "invalid_response_rate": metric["invalid_response_rate"], + "episode_end_steps": metric["episode_end_steps"], + }, + } + results.append(env_result) + + final_result = {"environments": results} + return final_result + + def _make_env(self, sample: dict) -> gym.Env: + env_id = sample["env_id"] + env_args = sample.get("env_args", {}) + if env_id in ENV_SETUP_FUNCS: + # Optional setup scripts for specific environments + ENV_SETUP_FUNCS[env_id]() + return gym.make(env_id, **env_args) + + def _generate_task_description(self, env: gym.Env, sample: dict) -> str: + + actions = [str(action) for action in range(env.action_space.n)] + observations = [ + f"Observation {observation}" for observation in range(env.observation_space.n) + ] + explanations = ( + sample["explanations"] if self.use_explanations else "You are playing a game." + ) + + return task_description_template.substitute( + action_space=env.action_space.n, + actions=actions, + observation_space=env.observation_space.n, + observations=observations, + explanations=explanations, + ) + + def _try_get_valid_action( + self, solver: Solver, task_state: TaskState, action_space: int + ) -> Optional[int]: + number_of_attempts = 0 + while number_of_attempts < self.max_invalid_responses: + if len(task_state.messages) > self.max_num_messages_allowed: + logger.info( + f"Exceeded maximum number of messages allowed ({self.max_num_messages_allowed})." + ) + return None + solver_response = solver(task_state).output + action = self._parse_action(solver_response) + task_state.messages += [Message(role="assistant", content=solver_response)] + task_state.current_state.total_responses += 1 + # Check if action is valid + if action not in range( + action_space + ): # TODO this might not work for non-discrete action spaces, check with more complex env + task_state.messages += [ + Message( + role="user", + content="Invalid action. Please provide ONE valid action by outputting your selection in the format [SELECT: x]. Only output this selection ONCE.", + ) + ] + task_state.current_state.invalid_responses += 1 + number_of_attempts += 1 + else: + return action + # If the loop exits due to reaching max invalid attempts, log and return None + logger.info(f"Exceeded maximum invalid action attempts ({self.max_invalid_responses}).") + return None + + def _parse_action(self, raw_response: str) -> Optional[int]: + pattern = r"\[SELECT: (\d+)\]" + matches = re.findall(pattern, raw_response) + + actions = [int(match) for match in matches] + if not actions: + logger.info(f"No action selections found in response: {raw_response}") + return None + if len(actions) > 1: + logger.info(f"Multiple action selections found in response: {raw_response}") + return None + return actions[0] + + def _add_message_to_task_state(self, task_state: TaskState, role: str, content: str) -> None: + """ + Adds a message to the task state, combining it with the previous message if they are from the same role. + """ + if task_state.messages and task_state.messages[-1].role == role: + task_state.messages[-1].content += "\n\n" + content + else: + task_state.messages.append(Message(role=role, content=content)) + + def _add_reset_message_to_task_state( + self, task_state: TaskState, observation: int, sample: dict + ) -> None: + content = reset_msg.substitute(observation=f"Observation {observation}") + self._add_message_to_task_state(task_state, "user", content) + + def _add_recap_message_to_task_state( + self, task_state: TaskState, actions: List, rewards: List[float] + ) -> None: + step_count = step_counter.substitute(step_count=len(actions)) + reward_count = reward_counter.substitute(reward_count=sum(rewards)) + content = "\n".join([step_count, reward_count]) + self._add_message_to_task_state(task_state, "user", content) + + def _format_step_message( + self, action: int, observation: int, reward: float, sample: dict, terminated: bool = False + ) -> str: + observation_desc = f"Observation {observation}" + if terminated: + template = step_result_reset + else: + template = step_result + return template.substitute(action=action, next_observation=observation_desc, reward=reward) + + def _calculate_episode_rewards(self, episode_end_steps, rewards): + episode_rewards = [] + if not episode_end_steps: # Handle case where there was only 1 episode + return [sum(rewards)] + start_index = 0 + for end_index in episode_end_steps: + episode_reward = sum(rewards[start_index:end_index]) + episode_rewards.append(episode_reward) + start_index = end_index + return episode_rewards diff --git a/evals/elsuite/incontext_rl/requirements.txt b/evals/elsuite/incontext_rl/requirements.txt new file mode 100644 index 0000000000..2712d1140b --- /dev/null +++ b/evals/elsuite/incontext_rl/requirements.txt @@ -0,0 +1,3 @@ +# Additional requirements for specific environments +gymnasium +git+https://github.com/james-aung/gymnasium-bandits \ No newline at end of file diff --git a/evals/elsuite/incontext_rl/scripts/plot_experiments.py b/evals/elsuite/incontext_rl/scripts/plot_experiments.py new file mode 100644 index 0000000000..9e8e27f82b --- /dev/null +++ b/evals/elsuite/incontext_rl/scripts/plot_experiments.py @@ -0,0 +1,363 @@ +import json +import numpy as np +import matplotlib.pyplot as plt +from scipy.stats import sem +import pandas as pd +from pathlib import Path +import matplotlib.colors as mcolors +import argparse +import seaborn as sns + +from evals.utils.log_utils import extract_spec, get_final_results_from_dir + +WINDOW_SIZES = { + "FrozenLake-v1 {'map_name': '4x4', 'is_slippery': False}": 20, + "BanditTwoArmedHighLowFixed-v0 {}": 40, + "BanditTenArmedRandomFixed-v0 {}": 40, + "CliffWalking-v0 {}": 20, + "FrozenLake-v1 {'map_name': '4x4', 'is_slippery': False, 'desc': ['SHFF', 'FFFF', 'FFGH', 'HFHF']}": 20, + "default": 20, +} + +PRETTY_MODEL_NAMES = { + 'generation/direct/gpt-4-turbo-preview': 'GPT-4 Turbo Preview', + 'incontext_rl/random': 'Random Strategy', + 'generation/direct/gpt-3.5-turbo': 'GPT-3.5 Turbo', + 'incontext_rl/qlearning_scratch': 'Q-Learning from scratch', + 'incontext_rl/qlearning_trained': 'Q-Learning trained', + 'generation/direct/gemini-pro': 'Gemini Pro 1.0', + 'generation/direct/mixtral-8x7b-instruct': 'Mixtral 8x7b', +} + +PRETTY_ENV_TITLES = { + "FrozenLake-v1 {'map_name': '4x4', 'is_slippery': False}": 'Frozen Lake (4x4, Non-slippery)', + "BanditTwoArmedHighLowFixed-v0 {}": "Two-Armed Bandit", + "BanditTenArmedRandomFixed-v0 {}": "Ten-Armed Bandit", + "CliffWalking-v0 {}": "Cliff Walking", + "FrozenLake-v1 {'map_name': '4x4', 'is_slippery': False, 'desc': ['SFFF', 'FHFH', 'FFFH', 'GFFH']}": 'Frozen Lake Custom Map (4x4, Non-slippery)', +} + +MODEL_STYLES = { + 'generation/direct/gpt-4-turbo-preview': {'line_style': '-', 'color': 'purple', 'alpha': 0.7}, + 'incontext_rl/random': {'line_style': ':', 'color': 'grey', 'alpha': 0.7}, + 'generation/direct/gpt-3.5-turbo': {'line_style': '-', 'color': 'green', 'alpha': 0.7}, + 'incontext_rl/qlearning_scratch': {'line_style': '--', 'color': 'grey', 'alpha': 0.7}, + 'incontext_rl/qlearning_trained': {'line_style': '-', 'color': 'black', 'alpha': 0.7}, + 'generation/direct/gemini-pro': {'line_style': '-', 'color': 'blue', 'alpha': 0.7}, + 'generation/direct/mixtral-8x7b-instruct': {'line_style': '-', 'color': 'orange', 'alpha': 0.7}, + 'default': {'line_style': '-', 'color': 'black', 'alpha': 0.5}, +} + +def calculate_episode_rewards(row: pd.Series) -> list: + """ + Calculate the rewards for each episode based on the episode end steps and rewards. + """ + episode_end_steps = row['episode_end_steps'] + rewards = row['rewards'] + episode_rewards = [] + if not episode_end_steps: # Handle case where there was only 1 episode + return [sum(rewards)] + start_index = 0 + for end_index in episode_end_steps: + episode_reward = sum(rewards[start_index:end_index]) + episode_rewards.append(episode_reward) + start_index = end_index + return episode_rewards + +def calculate_rolling_average(episode_rewards: list, window_size: int) -> list: + """ + Calculate the rolling average of the episode rewards using a specified window size. + """ + window_size = int(window_size) + rolling_averages = [] + for i in range(len(episode_rewards)): + # Calculate the start index for the window; ensure it's not negative + start_index = max(0, i - window_size + 1) + # Calculate the running average for the current window + window_average = np.mean(episode_rewards[start_index:i+1]) + rolling_averages.append(window_average) + return rolling_averages + +def calculate_custom_episode_end_steps_for_cliffwalking(rewards: list, existing_end_steps: list) -> list: + """ + Calculate episode end steps based on rewards and append to existing end steps. + An episode also ends when the reward is -100 i.e. when the agent falls off the cliff. + + Args: + rewards (list): List of rewards for each step in an episode. + existing_end_steps (list): List of already identified episode end steps. + + Returns: + list: Updated list of indices representing the end of each episode. + """ + new_end_steps = [i + 1 for i, reward in enumerate(rewards) if reward == -100] + # Combine existing and new end steps, remove duplicates, and sort + combined_end_steps = sorted(set(existing_end_steps + new_end_steps)) + return combined_end_steps + +def extract_results(datadir: Path) -> pd.DataFrame: + """ + Extracts results from the specified directory and returns a DataFrame. + + Args: + datadir (Path): Path to the directory containing the experiment results. + + Returns: + pd.DataFrame: DataFrame containing the experiment results. + """ + print(f"Extracting results from directory: {datadir}") + df_rows = [] + final_results = get_final_results_from_dir(datadir) + if not final_results: + print("No results found in directory.") + raise ValueError("No results found in directory.") + + for path, results in final_results.items(): + print(f"Processing file: {path}") + spec = extract_spec(path) + if not spec: + raise ValueError(f"No spec found for {path}") + model = spec.get("completion_fns", [None])[0] + base_eval = spec.get("base_eval") + if not model or base_eval is None: + raise ValueError(f"Missing model or base_eval in spec for {path}") + + environments = results.get('environments', []) + for env in environments: + metrics = env.get('metrics', {}) + flattened_metrics = {f"{k}": v for k, v in metrics.items()} # Flatten metrics into separate columns + print(f"Extracted {env['env']} metrics for model: {model}") + + # Calculate custom episode end steps for CliffWalking environment + if env['env'] == "CliffWalking-v0 {}": + rewards = metrics.get('rewards', []) + existing_end_steps = metrics.get('episode_end_steps', []) + episode_end_steps = calculate_custom_episode_end_steps_for_cliffwalking(rewards, existing_end_steps) + flattened_metrics['episode_end_steps'] = episode_end_steps + + df_rows.append({"model": model, "base_eval": base_eval, "environment": env['env'], **flattened_metrics}) + + df = pd.DataFrame(df_rows) + + if 'episode_rewards' not in df.columns: + df['episode_rewards'] = df.apply(calculate_episode_rewards, axis=1) + + # For plots + df['cumulative_episode_rewards'] = df['episode_rewards'].apply(np.cumsum) + df['average_episode_reward'] = df['episode_rewards'].apply(np.mean) + df['window_size'] = df['environment'].map(WINDOW_SIZES).fillna(WINDOW_SIZES.get('default', 20)) + df['rolling_average_rewards'] = df.apply(lambda row: calculate_rolling_average(row['episode_rewards'], row['window_size']), axis=1) + + # We also calculate the rolling average across different window sizes + df['rolling_average_rewards_5_episodes'] = df.apply(lambda row: calculate_rolling_average(row['episode_rewards'], 5), axis=1) + df['rolling_average_rewards_10_episodes'] = df.apply(lambda row: calculate_rolling_average(row['episode_rewards'], 10), axis=1) + df['rolling_average_rewards_20_episodes'] = df.apply(lambda row: calculate_rolling_average(row['episode_rewards'], 20), axis=1) + df['rolling_average_rewards_50_episodes'] = df.apply(lambda row: calculate_rolling_average(row['episode_rewards'], 50), axis=1) + + # We also calculate the average reward for the last 5, 10, 20, and 50 episodes. For older runs, we may not have this information. + if 'average_reward_last_5_episodes' not in df.columns: + df['average_reward_last_5_episodes'] = df['episode_rewards'].apply(lambda rewards: np.mean(rewards[-5:])) + if 'average_reward_last_10_episodes' not in df.columns: + df['average_reward_last_10_episodes'] = df['episode_rewards'].apply(lambda rewards: np.mean(rewards[-10:])) + if 'average_reward_last_20_episodes' not in df.columns: + df['average_reward_last_20_episodes'] = df['episode_rewards'].apply(lambda rewards: np.mean(rewards[-20:])) + if 'average_reward_last_50_episodes' not in df.columns: + df['average_reward_last_50_episodes'] = df['episode_rewards'].apply(lambda rewards: np.mean(rewards[-50:])) + + print(f"Extraction complete. {len(df_rows)} rows in DataFrame.") + return df + +def plot_rewards(df, environment, reward_type, out_dir, window_size=None): + """ + Generalized function to plot episode, cumulative, or running average rewards for different models + on the same graph for a specific environment. It automatically determines the plot type (line or scatter) + based on the number of episodes and includes the 95% confidence intervals for line plots. + + Args: + df (pd.DataFrame): DataFrame containing the experiment results. + environment (str): Name of the environment to plot. + reward_type (str): Type of reward to plot. Must be one of 'episode_rewards', 'cumulative_episode_rewards', or 'rolling_average_rewards'. + out_dir (Path): Path to the directory to save the plots. + window_size (int): Window size for calculating rolling averages. If None, the window size will be determined based on the environment. + """ + valid_reward_types = ['episode_rewards', 'cumulative_episode_rewards', 'rolling_average_rewards'] + if reward_type not in valid_reward_types: + raise ValueError(f"Invalid reward_type. Expected one of {valid_reward_types}, got {reward_type}") + + # Filter the DataFrame for the specific environment + filtered_df = df[df['environment'] == environment] + + # Explode the specified reward list into separate rows and prepare for plotting + rewards_df = filtered_df.explode(reward_type).reset_index() # Each row will be a single episode + rewards_df['episode'] = rewards_df.groupby(['model', 'index']).cumcount() + 1 # Add episode number as a column + rewards_df['reward'] = rewards_df[reward_type] # Rename the column for clarity + + truncate_per_model = True + if environment == "CliffWalking-v0 {}": + truncate_per_model = False # Hacky workaround to make better plots since some models only have 1 episode on CliffWalking + + if truncate_per_model: + filtered_rewards_df = pd.DataFrame() + for model, group in rewards_df.groupby('model'): + # Count the number of runs for each episode number + episode_counts = group.groupby('episode').size() + # Check if there are at least 3 runs for any episode number + if episode_counts.max() >= 3: + # Find the maximum episode number where at least 3 runs are available + max_episode_with_at_least_3_runs = episode_counts[episode_counts >= 3].index.max() + # Filter the group DataFrame to only include data up to this episode number + model_filtered = group[group['episode'] <= max_episode_with_at_least_3_runs] + else: + # If there are fewer than 3 runs for all episodes, include all data for this model + model_filtered = group + # Append the filtered data for the current model to the overall filtered DataFrame + filtered_rewards_df = pd.concat([filtered_rewards_df, model_filtered], ignore_index=True) + rewards_df = filtered_rewards_df + + plt.figure(figsize=(10, 5)) + ax = plt.gca() + + # Determine the plot type based on the number of episodes + num_episodes = len(rewards_df['episode'].unique()) + if num_episodes > 1: + # Iterate over each unique model in the DataFrame + for model in rewards_df['model'].unique(): + # Filter the DataFrame for the current model + model_df = rewards_df[rewards_df['model'] == model] + # Get the custom style for the current model using the helper function + custom_style = MODEL_STYLES.get(model, MODEL_STYLES['default']) + pretty_model_name = PRETTY_MODEL_NAMES.get(model, model) + # Plot the data for the current model on the same axes with custom settings + lineplot = sns.lineplot(data=model_df, x='episode', y='reward', estimator='mean', errorbar=('ci', 95), + linestyle=custom_style['line_style'], color=custom_style['color'], + alpha=custom_style['alpha'], label=pretty_model_name, ax=ax, + err_kws={'alpha': 0.035}) + # Add labels to the final value on the x axis + for line in lineplot.get_lines(): + x, y = line.get_data() + if len(x) > 0: # Check if there is data to plot + ax.text(x[-1], y[-1], f"{y[-1]:.2f}", color=line.get_color(), fontsize=9) + else: + # For a single episode, use scatter plot, differentiating models by color + scatterplot = sns.scatterplot(data=rewards_df, x='episode', y='reward', hue='model', ax=ax) + # Add labels to the final value on the x axis + for line in scatterplot.collections: + offsets = line.get_offsets() + if offsets.size > 0: # Check if there are points to plot + last_point = offsets[-1] + ax.text(last_point[0], last_point[1], f"{last_point[1]:.2f}", fontsize=9) + + pretty_env_title = PRETTY_ENV_TITLES.get(environment, environment) + plt.title(f'{reward_type.replace("_", " ").title()} in {pretty_env_title} (Window Size: {window_size})' if reward_type == 'rolling_average_rewards' else f'{reward_type.replace("_", " ").title()} in {pretty_env_title}') + plt.xlabel('Episode') + plt.ylabel('Reward') + plt.legend(title='Model', bbox_to_anchor=(1.05, 1), loc='upper left') + plt.xlim(1, num_episodes) + plt.tight_layout() + plot_dir = out_dir / reward_type + plot_dir.mkdir(parents=True, exist_ok=True) + plt.savefig(plot_dir / f'{environment}.png') + plt.show() + +def calculate_rolling_averages(df: pd.DataFrame, max_items: int = 200): + """ + Calculate the averaged final and max rolling averages for the first N items in each model and environment. + Args: + df (pd.DataFrame): DataFrame containing the experiment results. + max_items (int): Maximum number of items to consider for calculating rolling averages. + Returns: + dict: Dictionary containing the averaged final and max rolling averages for each model and environment. + """ + + model_env_averages_info = {} + for model in df['model'].unique(): + model_df = df[df['model'] == model] + model_env_averages_info[model] = {} + all_final_rolling_averages = [] # To store all final rolling averages across environments for each model + for env in model_df['environment'].unique(): + env_df = model_df[model_df['environment'] == env] + # Determine the last shared episode across all runs for the current model and environment, limited to the first max_items items + max_shared_episode = min(max_items, env_df['rolling_average_rewards'].apply(lambda rewards: len(rewards[:max_items])).min()) + # Truncate each run's rolling_average_rewards to the max shared episode and then calculate averages + truncated_averages = env_df['rolling_average_rewards'].apply(lambda rewards: rewards[:max_shared_episode]) + final_rolling_averages = round(truncated_averages.apply(lambda rewards: rewards[-1] if len(rewards) > 0 else None).mean(), 2) + max_rolling_averages = round(truncated_averages.apply(lambda rewards: max(rewards) if len(rewards) > 0 else None).mean(), 2) + + all_final_rolling_averages.append(final_rolling_averages) # Append the final rolling average for the current environment + + model_env_averages_info[model][env] = { + 'average_final_rolling_averages': final_rolling_averages, + 'average_max_rolling_averages': max_rolling_averages, + } + + # Calculate the average final rolling average across all environments for the current model + average_final_across_envs = round(sum(all_final_rolling_averages) / len(all_final_rolling_averages), 2) if all_final_rolling_averages else None + model_env_averages_info[model]['average_final_rolling_averages_across_envs'] = average_final_across_envs + return model_env_averages_info + +def json_of_results(df: pd.DataFrame, out_dir: Path): + """ + JSON dump of the results. + + Each model will have the following information, grouping by environment: + - Average episode reward + - Last rolling average reward for each of 5, 10, 20, and 50 episodes + - Max rolling average reward across the 5, 10, 20, and 50 episodes + - Invalid response rate + + Where there are multiple runs for a model and environment, the average of the above values will be calculated. + """ + + model_info = {} + for model in df['model'].unique(): + model_df = df[df['model'] == model] + model_info[model] = {} + for env in model_df['environment'].unique(): + env_df = model_df[model_df['environment'] == env] + # Calculate the average rolling averages across all runs for each window size, then find the max + average_rolling_averages_5 = env_df['rolling_average_rewards_5_episodes'].apply(pd.Series).mean().max() + average_rolling_averages_10 = env_df['rolling_average_rewards_10_episodes'].apply(pd.Series).mean().max() + average_rolling_averages_20 = env_df['rolling_average_rewards_20_episodes'].apply(pd.Series).mean().max() + average_rolling_averages_50 = env_df['rolling_average_rewards_50_episodes'].apply(pd.Series).mean().max() + + model_info[model][env] = { + 'average_episode_reward': round(env_df['average_episode_reward'].mean(), 2), + 'average_reward_last_5_episodes': round(env_df['average_reward_last_5_episodes'].mean(), 2), + 'average_reward_last_10_episodes': round(env_df['average_reward_last_10_episodes'].mean(), 2), + 'average_reward_last_20_episodes': round(env_df['average_reward_last_20_episodes'].mean(), 2), + 'average_reward_last_50_episodes': round(env_df['average_reward_last_50_episodes'].mean(), 2), + 'max_rolling_average_rewards_5_episodes': round(average_rolling_averages_5, 2), + 'max_rolling_average_rewards_10_episodes': round(average_rolling_averages_10, 2), + 'max_rolling_average_rewards_20_episodes': round(average_rolling_averages_20, 2), + 'max_rolling_average_rewards_50_episodes': round(average_rolling_averages_50, 2), + 'invalid_response_rate': round(env_df['invalid_response_rate'].mean(), 2), + } + with open(out_dir / 'model_info.json', 'w') as f: + json.dump(model_info, f, indent=4) + +def main(log_dir: str = None, out_dir: str = None): + + parser = argparse.ArgumentParser() + parser.add_argument("--log_dir", "-d", type=str, required=not log_dir) + parser.add_argument("--out_dir", "-o", type=str, required=not out_dir) + args = parser.parse_args() + log_dir = Path(log_dir) if log_dir else Path(args.log_dir) + out_dir = Path(out_dir) if out_dir else Path(args.out_dir) + + # Extract results from directory + df = extract_results(log_dir) + + # # Plot episode rewards with 95% confidence intervals + for env in df['environment'].unique(): + plot_rewards(df, env, 'episode_rewards', out_dir) + plot_rewards(df, env, 'cumulative_episode_rewards', out_dir) + window_size = df[df['environment'] == env]['window_size'].iloc[0] + plot_rewards(df, env, 'rolling_average_rewards', out_dir, window_size) + + # JSON dump of the results + json_of_results(df, out_dir) + + +if __name__ == "__main__": + main() + diff --git a/evals/elsuite/incontext_rl/scripts/qlearning_baseline.ipynb b/evals/elsuite/incontext_rl/scripts/qlearning_baseline.ipynb new file mode 100644 index 0000000000..bb2dc7ae44 --- /dev/null +++ b/evals/elsuite/incontext_rl/scripts/qlearning_baseline.ipynb @@ -0,0 +1,402 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install gymnasium\n", + "!pip install numpy\n", + "!pip install git+https://github.com/james-aung/gymnasium-bandits\n", + "!pip install tqdm" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import gymnasium as gym\n", + "import random\n", + "import json\n", + "\n", + "import gymnasium_bandits" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [], + "source": [ + "# Training parameters\n", + "n_training_episodes = 10000 # Total training episodes\n", + "n_training_steps = 200 # Total training steps\n", + "learning_rate = 0.7 # Learning rate\n", + "\n", + "# Evaluation parameters\n", + "reward_window_size = 25 # Number of steps to consider when calculating average reward\n", + "\n", + "# Environment parameters\n", + "gamma = 0.95 # Discounting rate\n", + "\n", + "# Exploration parameters\n", + "max_epsilon = 1.0 # Exploration probability at start\n", + "min_epsilon = 0.05 # Minimum exploration probability\n", + "decay_rate = 0.0005 # Exponential decay rate for exploration prob" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [], + "source": [ + "def initialize_q_table(state_space, action_space):\n", + " Qtable = np.zeros((state_space, action_space))\n", + " return Qtable" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [], + "source": [ + "def greedy_policy(Qtable, state):\n", + " # Exploitation: take the action with the highest state, action value\n", + " action = np.argmax(Qtable[state][:])\n", + "\n", + " return action" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [], + "source": [ + "def epsilon_greedy_policy(Qtable, state, epsilon):\n", + " # Randomly generate a number between 0 and 1\n", + " random_num = random.uniform(0,1)\n", + " # if random_num > greater than epsilon --> exploitation\n", + " if random_num > epsilon:\n", + " # Take the action with the highest value given a state\n", + " # np.argmax can be useful here\n", + " action = greedy_policy(Qtable, state)\n", + " # else --> exploration\n", + " else:\n", + " action = env.action_space.sample()\n", + "\n", + " return action" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [], + "source": [ + "def train(n_training_steps, min_epsilon, max_epsilon, decay_rate, env, Qtable, reward_window_size=25):\n", + "\n", + " actions, rewards = [], []\n", + " total_steps = 0\n", + " episode_end_steps = []\n", + "\n", + " for _ in range(n_training_steps):\n", + " if total_steps >= n_training_steps:\n", + " break\n", + " # Reduce epsilon (because we need less and less exploration)\n", + " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*len(episode_end_steps))\n", + " # Reset the environment\n", + " state, info = env.reset()\n", + " terminated = False\n", + " truncated = False\n", + "\n", + " while not terminated and not truncated and total_steps < n_training_steps:\n", + " # Choose the action At using epsilon greedy policy\n", + " action = epsilon_greedy_policy(Qtable, state, epsilon)\n", + "\n", + " # Take action At and observe Rt+1 and St+1\n", + " new_state, reward, terminated, truncated, info = env.step(action)\n", + "\n", + " actions.append(int(action))\n", + " rewards.append(float(reward))\n", + " total_steps += 1\n", + "\n", + " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n", + " Qtable[state][action] = Qtable[state][action] + learning_rate * (reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action])\n", + "\n", + " # Our next state is the new state\n", + " state = new_state\n", + "\n", + " if terminated or truncated:\n", + " episode_end_steps.append(total_steps)\n", + "\n", + " training_summary = {\n", + " \"reward_window_size\": reward_window_size,\n", + " \"average_reward_at_end\": sum(rewards[-reward_window_size:])/reward_window_size,\n", + " \"total_reward\": sum(rewards),\n", + " \"total_steps\": len(actions),\n", + " \"actions\": list(actions),\n", + " \"rewards\": list(rewards),\n", + " \"episode_end_steps\": episode_end_steps,\n", + " }\n", + " \n", + " print(f\"Training completed for {env.spec.id} at {total_steps} steps.\")\n", + " \n", + " return training_summary" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [], + "source": [ + "def evaluate(env, Qtable, n_evaluation_steps=200):\n", + "\n", + " actions, rewards = [], []\n", + " total_steps = 0\n", + " episode_end_steps = []\n", + "\n", + " while total_steps < n_evaluation_steps:\n", + " # Reset the environment at the start of each new episode\n", + " state, info = env.reset()\n", + " episode_end_steps.append(total_steps)\n", + " terminated = False\n", + " truncated = False\n", + "\n", + " while not terminated and not truncated and total_steps < n_evaluation_steps:\n", + " # Choose the action At using greedy policy\n", + " action = greedy_policy(Qtable, state)\n", + "\n", + " # Take action At and observe Rt+1 and St+1\n", + " new_state, reward, terminated, truncated, info = env.step(action)\n", + "\n", + " actions.append(int(action))\n", + " rewards.append(float(reward))\n", + " total_steps += 1\n", + "\n", + " # Our next state is the new state\n", + " state = new_state\n", + "\n", + "\n", + " evaluation_summary = {\n", + " \"average_reward_at_end\": sum(rewards[-25:])/min(25, len(rewards)),\n", + " \"total_reward\": sum(rewards),\n", + " \"total_steps\": len(actions),\n", + " \"actions\": list(actions),\n", + " \"rewards\": list(rewards),\n", + " \"episode_end_steps\": episode_end_steps,\n", + " }\n", + " \n", + " print(f\"Evaluation completed for {env.spec.id} at {total_steps} steps.\")\n", + " \n", + " return evaluation_summary" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [], + "source": [ + "frozenlake = gym.make(\"FrozenLake-v1\", is_slippery=False)\n", + "frozenlakecustom = gym.make(\"FrozenLake-v1\", is_slippery=False, desc =['SFFF', 'FHFH', 'FFFH', 'GFFH'])\n", + "twobandits = gym.make(\"BanditTwoArmedHighLowFixed-v0\")\n", + "tenbandits = gym.make(\"BanditTenArmedRandomFixed-v0\")\n", + "cliffwalking = gym.make(\"CliffWalking-v0\")\n", + "\n", + "envs = [frozenlake, frozenlakecustom, twobandits, tenbandits, cliffwalking]\n" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training FrozenLake-v1 with args {'map_name': '4x4', 'is_slippery': False}...\n", + "Training completed for FrozenLake-v1 at 200 steps.\n", + "Training FrozenLake-v1 with args {'map_name': '4x4', 'is_slippery': False, 'desc': ['SFFF', 'FHFH', 'FFFH', 'GFFH']}...\n", + "Training completed for FrozenLake-v1 at 200 steps.\n", + "Training BanditTwoArmedHighLowFixed-v0 with args {}...\n", + "Training completed for BanditTwoArmedHighLowFixed-v0 at 200 steps.\n", + "Training BanditTenArmedRandomFixed-v0 with args {}...\n", + "Training completed for BanditTenArmedRandomFixed-v0 at 200 steps.\n", + "Training CliffWalking-v0 with args {}...\n", + "Training completed for CliffWalking-v0 at 200 steps.\n" + ] + } + ], + "source": [ + "from datetime import datetime\n", + "\n", + "environment_results = []\n", + "\n", + "for env in envs:\n", + " print(f\"Training {env.spec.id} with args {env.spec.kwargs}...\")\n", + " Qtable = initialize_q_table(env.observation_space.n, env.action_space.n)\n", + " results = train(n_training_steps, min_epsilon, max_epsilon, decay_rate, env, Qtable, reward_window_size)\n", + " # train(n_training_steps, min_epsilon, max_epsilon, decay_rate, env, Qtable, reward_window_size)\n", + " # results = evaluate(env, Qtable)\n", + " env_result = {\"env\": f\"{env.spec.id} {env.spec.kwargs}\", \"metrics\": results}\n", + " environment_results.append(env_result)\n", + "\n", + "current_time = datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\")\n", + "spec = {\"spec\": {\"completion_fns\": [\"incontext_rl/qlearning\"], \"eval_name\": \"incontext_rl.v0\", \"base_eval\": \"incontext_rl\", \"split\": \"v0\", \"created_at\": current_time}}\n", + "final_report = {\"final_report\": {\"environments\": environment_results}}\n", + "\n", + "with open('./logs/qlearning_incontext_rl.log', 'w') as f:\n", + " json.dump(spec, f)\n", + " f.write(\"\\n\")\n", + " json.dump(final_report, f)\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 76, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Plotting running average reward for FrozenLake-v1\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABvNklEQVR4nO3deVhU1RsH8O+wDZsgioAiiuCCCyqCILjgQqKSRlkumVtlWmoqarmTlaKlRqVpamaZ5ZaWmkv+cN9ScV9wQVBT2VxAQLaZ8/uDGB0ZkMFhLjDfz/PM88ycOffe916Yue+ce865MiGEABEREZEBMZI6ACIiIiJ9YwJEREREBocJEBERERkcJkBERERkcJgAERERkcFhAkREREQGhwkQERERGRwmQERERGRwmAARERGRwWECRKRDMpkMn3zyidRhkAHbu3cvZDIZ9u7dK3Uoklm5ciVkMhni4+OlDoXKMSZAVO4VfJkVPExMTODs7IwhQ4bg9u3bUodX7ly6dAkymQzm5uZ4+PCh1OGUK0OGDFH7X5LL5WjYsCFmzJiBrKwsqcOTxIULF/DWW2/B2dkZcrkctWrVwoABA3DhwgWpQ1PTsWNHtb9dUQ/+AKGSMpE6AKKS+vTTT1GvXj1kZWXh6NGjWLlyJQ4ePIjz58/D3Nxc6vAAAI8fP4aJibQfq19++QVOTk548OABNmzYgHfffVfSeMobuVyO5cuXAwBSU1Px559/4rPPPkNsbCxWr14tcXT6tXHjRvTv3x/VqlXDO++8g3r16iE+Ph4//PADNmzYgDVr1uDVV1+VOkwAwNSpU9X+l48fP45vvvkGU6ZMQePGjVXlzZs3R9OmTdGvXz/I5XIpQqWKQhCVcz/++KMAII4fP65W/vHHHwsAYu3atRJFVv4olUrh6uoqwsLCxKuvvio6duwoSRwZGRmSbPd5Bg8eLKysrNTKlEqlaNOmjZDJZCIhIUGiyEpOqVSKzMzMIt/fs2ePACD27NlT7HquXbsmLC0thYeHh0hKSlJ7Lzk5WXh4eAgrKysRGxuri7BLLD09vUT11q9fX6L9JCoKL4FRhdW+fXsAQGxsrKqsY8eO6NixY6G6Q4YMgaurq+p1fHw8ZDIZ5s2bh6VLl8Ld3R1yuRytW7fG8ePHCy1rbW2N27dvIzQ0FNbW1qhRowYmTJgAhUKhVvfZJvhPPvkEMpkM165dw5AhQ1C1alXY2tpi6NChyMzMVFv28ePH+PDDD2Fvb48qVaqgV69euH37tlbN+ocOHUJ8fDz69euHfv36Yf/+/fj3339V77/88stwc3PTuKy/vz98fHzUyn755Rd4e3vDwsIC1apVQ79+/XDr1i21Oh07dkSzZs0QHR2NDh06wNLSElOmTAEA/PnnnwgJCUGtWrUgl8vh7u6Ozz77rNBxA4BFixbBzc0NFhYW8PX1xYEDBzT+PbOzsxEeHo769etDLpfDxcUFH330EbKzs0t0jJ4lk8nQrl07CCFw/fp1tfe2b9+O9u3bw8rKClWqVEFISIjapaHNmzdDJpPh7NmzqrLff/8dMpkMr732mtq6GjdujL59+6pe//jjj+jcuTMcHBwgl8vRpEkTLF68uFB8rq6uePnll7Fz5074+PjAwsIC33//PQDg33//RWhoKKysrODg4IBx48aV+Dh8+eWXyMzMxNKlS1GjRg219+zt7fH9998jIyMDX3zxBQBgw4YNkMlk2LdvX6F1ff/995DJZDh//ryqLCYmBq+//jqqVasGc3Nz+Pj4YPPmzWrLFVze3rdvHz744AM4ODigdu3aJYq/OJr6ABUcx71796qOo6enp6qv1MaNG+Hp6Qlzc3N4e3vj1KlThdZbkn2iioMJEFVYBV9udnZ2pV7Hr7/+ii+//BLDhw/H559/jvj4eLz22mvIzc1Vq6dQKBAcHIzq1atj3rx5CAwMxPz587F06dISbadPnz549OgRIiIi0KdPH6xcuRIzZ85UqzNkyBB8++236NGjB+bOnQsLCwuEhIRotT+rV6+Gu7s7WrdujZ49e8LS0hK//fab6v2+ffsiLi6uUJJ348YNHD16FP369VOVzZo1C4MGDUKDBg2wYMECjB07FlFRUejQoUOhvkX37t1D9+7d0bJlS0RGRqJTp04A8k9E1tbWCAsLw9dffw1vb2/MmDEDkyZNUlt+8eLFGDVqFGrXro0vvvgC7du3R2hoqFryBgBKpRK9evXCvHnz0LNnT3z77bcIDQ3FV199pZZcaEvT/9KqVasQEhICa2trzJ07F9OnT8fFixfRrl07Vf127dpBJpNh//79quUOHDgAIyMjHDx4UFWWnJyMmJgYdOjQQW2f69atiylTpmD+/PlwcXHBBx98gEWLFhWK7/Lly+jfvz9eeuklfP3112jZsiUeP36MLl26YOfOnRg1ahSmTp2KAwcO4KOPPirRPm/ZsgWurq6qHxLP6tChA1xdXfHXX38BgOpYrFu3rlDdtWvXomnTpmjWrBmA/H5Fbdq0waVLlzBp0iTMnz8fVlZWCA0NxaZNmwot/8EHH+DixYsa/zd06dq1a3jzzTfRs2dPRERE4MGDB+jZsydWr16NcePG4a233sLMmTMRGxuLPn36QKlUqpbVdp+oApC6CYroeQougf3vf/8TycnJ4tatW2LDhg2iRo0aQi6Xi1u3bqnqBgYGisDAwELrGDx4sKhbt67qdVxcnAAgqlevLu7fv68q//PPPwUAsWXLFrVlAYhPP/1UbZ1eXl7C29tbrQyACA8PV70ODw8XAMTbb7+tVu/VV18V1atXV72Ojo4WAMTYsWPV6g0ZMqTQOouSk5MjqlevLqZOnaoqe/PNN0WLFi1Ur1NTU4VcLhfjx49XW/aLL74QMplM3LhxQwghRHx8vDA2NhazZs1Sq3fu3DlhYmKiVh4YGCgAiCVLlhSKSdOlmuHDhwtLS0uRlZUlhBAiOztbVK9eXbRu3Vrk5uaq6q1cuVIAUPt7rlq1ShgZGYkDBw6orXPJkiUCgDh06FBRh0cI8eQSWHJyskhOThbXrl0T8+bNEzKZTDRr1kwolUohhBCPHj0SVatWFcOGDVNbPiEhQdja2qqVN23aVPTp00f1ulWrVuKNN94QAMSlS5eEEEJs3LhRABBnzpwp9tgEBwcLNzc3tbK6desKAGLHjh1q5ZGRkQKAWLdunaosIyND1K9f/7mXhh4+fCgAiFdeeaXIOkII0atXLwFApKWlCSGE6N+/v3BwcBB5eXmqOnfv3hVGRkZqn48uXboIT09P1d9YiPxLdwEBAaJBgwaqsoLPdrt27dTWWRLFXQIrWG9cXJyqrOA4Hj58WFW2c+dOAUBYWFio/veFEOL7778vtO6S7hNVHGwBogojKCgINWrUgIuLC15//XVYWVlh8+bNL9Rk3rdvX7Vf/QW/hp+9FAIAI0aMUHvdvn17jfU00bTsvXv3kJaWBgDYsWMHgPxfwk8bPXp0idYP5F+uuXfvHvr3768q69+/P86cOaO6bGNjY4Pu3btj3bp1EEKo6q1duxZt2rRBnTp1AORfDlAqlejTpw9SUlJUDycnJzRo0AB79uxR27ZcLsfQoUMLxWRhYaF6/ujRI6SkpKB9+/bIzMxETEwMAODEiRO4d+8ehg0bptaBfMCAAYVa99avX4/GjRvDw8NDLa7OnTsDQKG4NMnIyECNGjVQo0YN1K9fHxMmTEDbtm3x559/QiaTAQB27dqFhw8fon///mrbMTY2hp+fn9p22rdvjwMHDqj28cyZM3jvvfdgb2+vKj9w4ACqVq2qaiF59tikpqYiJSUFgYGBuH79OlJTU9VirlevHoKDg9XKtm3bhpo1a+L1119XlVlaWuK999577jF49OgRAKBKlSrF1it4v+D/tG/fvkhKSlIbYr9hwwYolUpVC9z9+/exe/duVatnwbG7d+8egoODcfXq1UKjN4cNGwZjY+Pnxv2imjRpAn9/f9VrPz8/AEDnzp1V//tPlxd8vkuzT1T+cRQYVRiLFi1Cw4YNkZqaihUrVmD//v0vPMrj6S894MklkAcPHqiVm5ubF+onYWdnV6heabZjY2ODGzduwMjICPXq1VOrV79+/RKtH8jvr1OvXj3I5XJcu3YNAODu7g5LS0usXr0as2fPBpB/Evvjjz9w5MgRBAQEIDY2FtHR0YiMjFSt6+rVqxBCoEGDBhq3ZWpqqvba2dkZZmZmhepduHAB06ZNw+7du1Un0QIFJ/kbN25o3FcTExO1flsFcV26dKnQ36JAUlKSxvKnmZubY8uWLQDy+9B88cUXSEpKUktIrl69CgCqxOpZNjY2quft27fHkiVLcO3aNcTGxkImk8Hf31+VGA0bNgwHDhxA27ZtYWT05DfnoUOHEB4ejiNHjhTqD5aamgpbW1vV62f/L4D841a/fn1V0lagUaNGzz0GBYlNQSJUlGcTpW7dusHW1hZr165Fly5dAOQnzy1btkTDhg0B5F9mEkJg+vTpmD59usb1JiUlwdnZudj9KwvPfg4LjrGLi4vG8oLPd2n2ico/JkBUYfj6+qo66YaGhqJdu3Z48803cfnyZVhbWwPI79D6dMtGAU2dbgEU+avz2XW86K/Tkm6ntNLS0rBlyxZkZWVpTFp+/fVXzJo1CzKZTNU3aN26dQgICMC6detgZGSEN954Q1VfqVRCJpNh+/btGmMvON4Fnk4eCjx8+BCBgYGwsbHBp59+Cnd3d5ibm+PkyZP4+OOP1fpXlJRSqYSnpycWLFig8f1nT2SaGBsbIygoSPU6ODgYHh4eGD58uKpDa0Fsq1atgpOTU6F1PN1S1a5dOwDA/v37cf36dbRq1QpWVlZo3749vvnmG6Snp+PUqVOYNWuWapnY2Fh06dIFHh4eWLBgAVxcXGBmZoZt27bhq6++KnRsNB3fF2Fra4uaNWuqdd7W5OzZs3B2dlYlfHK5XNXn5bvvvkNiYiIOHTqkSq6BJ8duwoQJhVqtCjyb7Op6/4pS1OfweZ/P0uwTlX9MgKhCMjY2RkREBDp16oSFCxeqOk7a2dlpvCxV0MpQXtWtWxdKpRJxcXFqCUxBS87zbNy4EVlZWVi8eDHs7e3V3rt8+TKmTZuGQ4cOoV27drCyssLLL7+M9evXY8GCBVi7di3at2+PWrVqqZZxd3eHEAL16tVT/bLX1t69e3Hv3j1s3LhRrfNvXFycWr26deuq9rWg8zQA5OXlIT4+Hs2bN1eL68yZM+jSpUuhlo/SqlmzJsaNG4eZM2fi6NGjaNOmDdzd3QEADg4OasmSJnXq1EGdOnVw4MABXL9+XXUZtUOHDggLC8P69euhUCjUjsGWLVuQnZ2NzZs3q7VKlOQSXoG6devi/PnzEEKoHYvLly+XaPmXX34Zy5Ytw8GDB1VJ3NMOHDiA+Ph4DB8+XK28b9+++OmnnxAVFYVLly5BCKHWAb1glKGpqelzj11FURn3iTgKjCqwjh07wtfXF5GRkapZfN3d3RETE4Pk5GRVvTNnzuDQoUNShVkiBb8qv/vuO7Xyb7/9tkTL//LLL3Bzc8OIESPw+uuvqz0mTJgAa2trtUn++vbtizt37mD58uU4c+ZMoRFUr732GoyNjTFz5sxCrVRCCNy7d++5MRX8qn56+ZycnEL76OPjg+rVq2PZsmXIy8tTla9evbrQJcY+ffrg9u3bWLZsWaHtPX78GBkZGc+NS5PRo0fD0tISc+bMAZD/97CxscHs2bMLjQgEoPb/BeRfBtu9ezeOHTumSoBatmyJKlWqYM6cObCwsIC3t7eqvqZjk5qaih9//LHEMffo0QN37tzBhg0bVGUFw9pLYuLEibCwsMDw4cML/T3v37+PESNGwNLSEhMnTlR7LygoCNWqVcPatWuxdu1a+Pr6ql3CcnBwQMeOHfH999/j7t27hbb77LGrCCrjPhFbgKiCmzhxIt544w2sXLkSI0aMwNtvv40FCxYgODgY77zzDpKSkrBkyRI0bdq0UB+U8sTb2xu9e/dGZGQk7t27hzZt2mDfvn24cuUKABTb2nHnzh3s2bMHH374ocb35XI5goODsX79enzzzTcwNTVFjx49UKVKFUyYMAHGxsbo3bu32jLu7u74/PPPMXnyZMTHxyM0NBRVqlRBXFwcNm3ahPfeew8TJkwodp8CAgJgZ2eHwYMH48MPP4RMJsOqVasKJVRmZmb45JNPMHr0aHTu3Bl9+vRBfHw8Vq5cCXd3d7V9HzhwINatW4cRI0Zgz549aNu2LRQKBWJiYrBu3TrVXDnaql69OoYOHYrvvvsOly5dQuPGjbF48WIMHDgQrVq1Qr9+/VCjRg3cvHkTf/31F9q2bYuFCxeqlm/fvj1Wr16tmlMIyE9yAgICsHPnTnTs2FGtj1TXrl1hZmaGnj17Yvjw4UhPT8eyZcvg4OCg8QSrybBhw7Bw4UIMGjQI0dHRqFmzJlatWgVLS8sSLd+gQQP89NNPGDBgADw9PQvNBJ2SkoLffvtN1RpWwNTUFK+99hrWrFmDjIwMzJs3r9C6Fy1ahHbt2sHT0xPDhg2Dm5sbEhMTceTIEfz77784c+ZMiWIsTyrjPhk8/Q88I9JOUTNBCyGEQqEQ7u7uwt3dXTWM9pdffhFubm7CzMxMtGzZUuzcubPIYfBffvlloXXimWHnmmYPFuLJEPfili2ok5ycrHGfnh6mm5GRIUaOHCmqVasmrK2tRWhoqLh8+bIAIObMmVPk8Zk/f74AIKKiooqsUzCk/M8//1SVDRgwQAAQQUFBRS73+++/i3bt2gkrKythZWUlPDw8xMiRI8Xly5dVdQIDA0XTpk01Ln/o0CHRpk0bYWFhIWrVqiU++ugj1dDjZ4cvf/PNN6Ju3bpCLpcLX19fcejQIeHt7S26deumVi8nJ0fMnTtXNG3aVMjlcmFnZye8vb3FzJkzRWpqapH7IkTRf0shhIiNjRXGxsZi8ODBqrI9e/aI4OBgYWtrK8zNzYW7u7sYMmSIOHHihNqyFy5cEABE48aN1co///xzAUBMnz690PY2b94smjdvLszNzYWrq6uYO3euWLFihcbh2yEhIRpjvnHjhujVq5ewtLQU9vb2YsyYMWLHjh1azZB89uxZ0b9/f1GzZk1hamoqnJycRP/+/cW5c+eKXGbXrl0CgJDJZGrTUDwtNjZWDBo0SDg5OQlTU1Ph7OwsXn75ZbFhwwZVneI+289TmmHwmo4jADFy5Ei1sqK+H0qyT1RxyITQUS9MItK506dPw8vLC7/88gsGDBggdTh6pVQqUaNGDbz22msaL3kREb0I9gEiKiceP35cqCwyMhJGRkZqHWgro6ysrEKXxn7++Wfcv39f461NiIheFPsAEZUTX3zxBaKjo9GpUyeYmJhg+/bt2L59O957770SDe+uyI4ePYpx48bhjTfeQPXq1XHy5En88MMPaNasmdrwfCIiXeElMKJyYteuXZg5cyYuXryI9PR01KlTBwMHDsTUqVPV5p2pjOLj4/Hhhx/i2LFjuH//PqpVq4YePXpgzpw5cHBwkDo8IqqEmAARERGRwWEfICIiIjI4TICIiIjI4FTujgWlpFQqcefOHVSpUkVn0+0TERFR2RJC4NGjR6hVq5bazYc1YQKkwZ07dyr9qBsiIqLK6tatW6hdu3axdZgAaVClShUA+Qew4C7IREREVL6lpaXBxcVFdR4vDhMgDQoue9nY2DABIiIiqmBK0n2FnaCJiIjI4DABIiIiIoPDBIiIiIgMDhMgIiIiMjhMgIiIiMjgSJ4ALVq0CK6urjA3N4efnx+OHTtWZN0LFy6gd+/ecHV1hUwmQ2RkpMZ6t2/fxltvvYXq1avDwsICnp6eOHHiRBntAREREVU0kiZAa9euRVhYGMLDw3Hy5Em0aNECwcHBSEpK0lg/MzMTbm5umDNnDpycnDTWefDgAdq2bQtTU1Ns374dFy9exPz582FnZ1eWu0JEREQViKR3g/fz80Pr1q2xcOFCAPm3oHBxccHo0aMxadKkYpd1dXXF2LFjMXbsWLXySZMm4dChQzhw4ECp40pLS4OtrS1SU1M5DxAREVEFoc35W7IWoJycHERHRyMoKOhJMEZGCAoKwpEjR0q93s2bN8PHxwdvvPEGHBwc4OXlhWXLlhW7THZ2NtLS0tQeREREVHlJlgClpKRAoVDA0dFRrdzR0REJCQmlXu/169exePFiNGjQADt37sT777+PDz/8ED/99FORy0RERMDW1lb14H3AiIiIKjfJO0HrmlKpRKtWrTB79mx4eXnhvffew7Bhw7BkyZIil5k8eTJSU1NVj1u3bukxYiIiItI3yRIge3t7GBsbIzExUa08MTGxyA7OJVGzZk00adJEraxx48a4efNmkcvI5XLVfb94/y8iIqLKT7IEyMzMDN7e3oiKilKVKZVKREVFwd/fv9Trbdu2LS5fvqxWduXKFdStW7fU6yQyZFm5Ckg4VoKIqExIejf4sLAwDB48GD4+PvD19UVkZCQyMjIwdOhQAMCgQYPg7OyMiIgIAPkdpy9evKh6fvv2bZw+fRrW1taoX78+AGDcuHEICAjA7Nmz0adPHxw7dgxLly7F0qVLpdlJogrMddJfqufxc0IkjISISLckHQYPAAsXLsSXX36JhIQEtGzZEt988w38/PwAAB07doSrqytWrlwJAIiPj0e9evUKrSMwMBB79+5Vvd66dSsmT56Mq1evol69eggLC8OwYcNKHBOHwRMB52+n4uVvD6peV5Gb4NzMYAkjIqLSyslTwsyk0nX7LUSb87fkCVB5xARId5LSsnAtOR0B7vYlqv/zkXhsPXsXywf7wMbctIyjMwzv/XwCh2PvIXp6EOQmxhrrZOUqAAByEyNk5ijQNHynxnpnP+nKv4senLz5AK99dxhHJ3eBk6252ntCCNSbvA1AyVrl8hRKGBvJIJPJSrz9XIUSpsaV/2RpKJ5uyb02qztMXuBve+fhY1xJfISOjRyKrJOSno1b9zPRtJYtTI21+997UUyAXlBlT4DWnbgF1+pW8K1XTafrzcjOK/LECQBbR7dDM2fbIt9/+kMKAKvf9cOwn0/gwEedUN1arvZeTp4SWXkK3E/PQcd5ewEAcRE9NH7Qvtp1BRZmxhgR6K7F3mhHCAGFUpTqi0UIga5f7Udv79oYEeiu05NPnkKJ+lO3q15/0NEdH3XzUKtz9Po99Ft6tMTrNDc1Qsxn3XUSX0nlKpQY9MMxHLl+Dwc+6gSXapY6WW92ngI37mWi61f7AagnFEqlQFaeAk1m5P9Pazpx5CqU+H5fLDo0rIHmtau+UCxCCOQpBYxkMrhP2aYqfzqmcWtPY9Op22rLxc7ugew8BSzNCvdo6L/0KI5cvwcA+HFIa3TyKPqkdfNeJl5ZdBAPMnNVZXsndMQXO2PwmldtjF9/BqmPc+FbrxrWDX/STzP5UTaCFuzD7vGBhT6nZWn4qhPYeSER4T2bYGjbwlcG7mfkYNSvJ9G1iSO6NatZKJF81obofzFz8wVMCG6EQf51X+ikvfdyEob8eLxQ+fYx7dG4pv7OKc9+/p+lzWXt7DwFGk3boXrtVacqNn3QVmPdZ7/Li/puLgtMgF5QZUuAFEoBYyMZbt3PRPsv9qjKf3/fH951CydB2jSVKpUC/8Tdxy9Hb+Cvc3dLtMz+iZ1Qp7r6CaztnN24/fBxiZZ/nnOfdEUVc1Nk5uRh7fFbmLklv99Y2EsNsWDXFYx/qSGGdXDDgl1X8HLzmmhayxbGRoU/nFm5Cmw7dxd+btXhXNWi2G0WfOCXD/JBUBNHjXWe/uX+09u+sDQzRm07C/hH7C5Ut569FfZM6IjY5HSYGRuhRhU5Vh25gQFt6mg80T0vrqf1blUb8/u0KLbO03q2qIUtZ+6olV2f3QNGGo7Z03IVSqw4GIeI7TGqsonBjTCyU32N9bPzFIi+8QCt6tjB2EimlgTq+gv1zsPHCJhT+Lif/aQrTI2M0DR8B5Qavhnj54SoPk8AELHtEr7ff71Use2OScQHq09izXv+aOlStci/w3cDWuGD1SdLsFfAxU+DVQmbJp++0hSD/F0LlWfm5BW73LPOfdIVnp/8Xah8QZ8WeK1VbdXrP0/fRjUrM/xwMA4fd/PQ2ck/KS0LvrOfDKD5tr8XeraopXo97Y9z+OVo4ZG/xZ3wnz7+zlUtsHtCIEyMjHD234d49bvDGtfz5+nbGLPmNABgTJcGGOhfF7kKpcbPdIH+vnUQ8Zpn8TuohQ9WR8O9hjXGd22kVh6x/RK+33e9iKWeKOqY5CqUaDB1O0Z2cseiPbEa62wZ1Q6etdV/1N7PyEGrz3aVeDu6xgToBVWmBKje5L9Q3F/46X/KTvP2Ii4lQ/X6VS9nfNW3pVr9PTFJGLqy8C+b0tj2YXs0qWUDhVKo/eLVBYcqciQ9ytZqmWc/oJpOSFXMTXDuE/V+MAeuJmPgD09u4qvpg67LfdTmi6Sok2rs7B4wNpIhK1cBj+k7NNYBgFq25jg0qTMG/nAMB6+lqL0XNT4Q9apbaUyEntca+Ow+PJ0cFtg3sSPqVrfCmmM3MWnjuULreF6L4tPSs/PQrJh4SuPK593RcJrmX9fnZwbDWl50ovrs/v7+vj96Ly79DPjaKsn/+ousu7j/q82j2qq1lj3IyIHXZ7tUydmemCRcvJuGNm7V0Lx2VZy7nYomNW1gbmqM5Qeu4/O/Lmlc76/v+iGgvj0W7r6KeX9fKTa+AkIIBH65FzfvZ5ZuZ0vp6RjyFEoAwIkbD9CidlXcz8xBbp4SrvZWz13P08fjvQ5umNKjseo9bf+mO8a2h4fTk/OdNss/nfS7T9kGhaZfDwBq2prj73EdUKUML6MzAXpBlSUBmrD+DDZE//vcegUfxuL+4UM8a5a4hUcb60f4440l+vviL85nrzTFwP9+HZ+59RCvLDqksV4zZxt0beKED7s0QOiiQzh966Ha+8+2AimVAm0iorROyIrSvoE9Vr3j99x6TWbsQGaOosj34+eEFPs3f7Yl44eDcfhs68VC9dxrWGFKj8ZYc/wWdl1MLPR+UcYFNcSYoAYAgEm/n8Wa49pPQFqSZLDgl6y+xc8JgRACbyw5glAvZ0z74zyA/Fam5hpaT7SxeVRb9Fqo+f+zJOa/0QK9vWtDCIGLd9MQ8s3B5y+kQ9/090LXJo44fzsVrz/1+Y/5rFuxCbkujO5cH++2c0NGTp7GVkB9CHupIXZeSMD6Ef5Ftry94V0b65/6/i5I8ID8pOnc7dRCLVMF+vq4YO0J9c/T3N6e+Pj3wj8knrZ9THt0/7p099GMnxPy3EtuBSZ19yizLglMgF5QZUiA0rJytfqSrVPNUie/gqpamuLhU30IYj7rhplbLuC3YyU7uV2b1R2yZ/pAFCVqfCC6zN9X6lj14XlJxouY9WozDPDTPL/Vs9frY2f3wD/X7+HN5f8Uub5jU7sg7XEeUh/naLw0Cui2pQAoWfJdnJ1jO8CthhVm/Hket+4/xokb95GVm/+Lev/ETvh060X879LzkzIPpyqISXik8b1Wdari5M2HWsf249DWGKqhH8iL2vRBALzq2OFRVq7Gy1DPip3dA1/sjCl0OWTpQG+8typa4zJXPu+OrDwF9l5ORgMHa3T/+gBqVJEjWUdJvL7ERfQAgEKti/pQ0MXgfkYOAuZEqf4vX0RrVzusHxGg9efl1PSXYGdlplouLqJHqY7J9dk9IJPp5nhWtTTF1tHtUNtON/35CjABekGVIQEq6gPSrakTlgz01vmJzLdeNSx6sxVqVJHjcY4Cvx27iZeaOKo6qwohIJPJ8MnmC1h5OL7I9TzbPH058REeZeXBp64dZDIZ/rl+D0v2xWLhm61g9dQlhqRHWcjJU6Ld3D2aVovwnk1UfYGkdm1Wd8Tfy8DpW6mYsP4MgPz9PnQtBQOKSVBKoqiEoiSJRklaUzRdqiqOiZEMV2d1x6Fr91CzqnmhhFXbBLFbUyfsuFD6ewU+66NujfB+oDtkMlmRxwx4/qXfY1O6wMLMuEQJSUkc+KgTattZoN7kbXirTR189kqzYvsVPft3KThRPb3Ms30ANSm4NFqUY3H30ef7Jy02UeMDUUVugmpWZiX65a8rte0sENm3JXxcqxX5/+Nd1w6/vx8A4PmdgYH8HwA1rPMvnUf+7yp+O6beh2haSGP8euwmridnqJUXJFlZuUr871IiRv92Cp09HLBiSOtC2yirH0PFKepzPWXTOfz6T9F3SHjW0y3CJ+Lvq7XcafLnyLZo4VIV3SL3F/njorj4SosJ0Auq6AlQamYuWnz65IvY09kWW0a3U6tTcN1dk+f9Oqhb3RJRYYGlHkqZkJqFNhFRhcpjPusGc1PNw7RL6ukvmI+6NYKlqTEG+rsW+aV+LO4+oi4lFurIWsDSzLjYy0hPq1vdEjfuFd+K9uswP7UpAQoSw2fVn7INeUqBs590xfh1Z7DvSjLWvtemyCbvAqvf9YOrvRXaPtW0//QX1817mejwZeGToLZfQgv+voxvdl97br2rs7qrdWbecuYORv926snrUe3Qc6H65ZcdY9ujW2ThZviS9lsqTsxn3QAAVxIfwdPZttCxf5SVi9nbYjCzV9NCAwFuP3yM3t8dRkJaFqaFNMbsbZdUHaULjl/0jQfovbj4v9HT4ueE4O8LCXhvVTSWDvRG16alvw1QibaXkqEaNfmsRW+2Qkjzms9dxxc7YnA9OQNfvNFcbUoETSf3gv89IQSEAH47fhNTN50vUaxFffae/Z54trUTAC5/3q3QlA+aOqwXKOr/P0+hxM9HbiCgfnV4ONlACIHzt9PQyKlKqYZ3F5cAvdKyFv48fafI94tz+fNuhY4BkN8pvrhBE3kKJYxkMrgV0eJuYiTD0LauGPdSQ43rKS6xfPqYFtdayQSonKnICdCzozkKRhNpoumf98jkzqhpawGFUuC3YzfRs0Ut/Hw4Hg42crzh7fLckT/aEEJg69m7OBx7D7NfLf4Xbkll5ylw8U4aWtSuqlWsRX0xxc8JwcGrKahqaYqmtWw0JoYHPuoE56oWMDKS4cdDcUW2NBV0+n4RXebvRewzv0Cf53kdXp/3q784Z249xOWER/jo97MA8vsQeDhVKfZvWVxLhHsNK0SN74g//hvq3b6BPR5k5qC+QxW1eqN+PYmtZ0veJ82nrh3e6+Cm8wRDoRQweqalRdP/0o6x7VHbzlLVEXvFEB8ENnQo9XF/EQN/+AcHrqp3aH/Duza+fKNFEUuUnEIp8NKCfahubYb1IwI01tF0fL7t76WWGANP/m/fWv6PqgN+gHt1/Dqsjcb1rjwUh0+2XMRH3Rrhg46aRxruv5KMQSuOqZXpe4bze+nZiNgeU6h/ZvycECSmZeHQtRRsO3cXnTwc0KtFLfx19i5e966tMdGYFtIYQwJcYWJshKS0LKRn56HzU62sJd23Z0fVHZ7UGU425iX6DtX09zw5/SVUszLTWD/yf1cQ+b+rAICFb3rh5ea1NNYrLSZAL6iiJkDPztwLPP8DsP7ELUzccLbE9Surw9dSCvWP+aa/F3q1KPzhzM5T4Ld/biIlPQfjuzYsdLLPVShx4Goy3l55QlXWq0UtfNPfSyex/nL0hqpD7fNc+bx7oZaMgkttBf0JdCFXoYSJFpPtFZVwlnQIuVIpCv1qrW5lhnsZOYXqvtKyFr7up5tjX1IF+2ckAy5++uItm7r07OUyfX/m/zx9G3O3x2DfR50KzXdVcNyipwWpzSmk64kZ1x6/iRl/XsDJ6S+pXUrXt4KW1EOTOj93qo3Xvjuk1hetqM9KVq4CF+6kwsvFTqsfgQqlgAzQ+kfuOyuPIyomCf1au+D2w8cYEeiOtvWLn/g2KS0LmTmKEo100xYToBdUEROgooZZl+TLbfOZO5i26RyOTQ0qV1/UUlEqBRRCVKiZcDUlFKM61ceE4EYaaktvzvYYLNmnPrfIs8NwS6KgRezZS21Hr9/D0v3XsfitVkXOfl2WMnPycOFOGrzraHcS0hchBDJzFJKe/DURQkApIEnLWEXwe/S/+PtiApa85a3X2ZUrEiZAL6giJkCaToDPm4uEKpdTNx+o9REq7615WbkKTNxwFlvO3MHBjzvpfDQIERkeJkAvqCIlQEV1Qvu6X0u80tJZgohISgqlwPh1pzHQv26RQ9mJiCorbc7fbB6o4DQlP+uG++v8Pl9UMRgbyRCp5/4uREQVEROgSuZF7/RLRERkCHimrMB+emZCwd/fD2DyQ0REVAJsAarAwjdfUD1/dhQMERERFY1nzAoq9XGu2msmP0RERCXHs2YFNX7dGdXzaSGNJYyEiIio4mECVEE9fYfrd9u7SRgJERFRxcMEiIiIiAwOEyAiIiIyOEyAKqBchVL1fGQndwkjISIiqpiYAFVAm0/fUT1/06+uhJEQERFVTEyAKqApm86pnjtXtZAwEiIiooqJCVAFlJ2nfH4lIiIiKhIToApmT0yS1CEQERFVeEyAKpiNp26rnn/Vt4WEkRAREVVcTIAqmC1nnnSAftWrtoSREBERVVxMgIiIiMjgMAGqoHj/LyIiotJjAlSBKJRC9byNW3UJIyEiIqrYmABVIBtP/qt67mJnKWEkREREFRsToApk9T83Vc+rmJtIGAkREVHFxgSoAjl966HquZGRTLpAiIiIKjgmQERERGRwmABVEIeupUgdAhERUaXBBKiC+Cfuvur5hZnBEkZCRERU8TEBqiCuJDxSPbeSswM0ERHRi2ACVEHE38uQOgQiIqJKo1wkQIsWLYKrqyvMzc3h5+eHY8eOFVn3woUL6N27N1xdXSGTyRAZGVnsuufMmQOZTIaxY8fqNmg9i3mqBYiIiIhejOQJ0Nq1axEWFobw8HCcPHkSLVq0QHBwMJKSkjTWz8zMhJubG+bMmQMnJ6di1338+HF8//33aN68eVmELglPZ1upQyAiIqrwJE+AFixYgGHDhmHo0KFo0qQJlixZAktLS6xYsUJj/datW+PLL79Ev379IJfLi1xveno6BgwYgGXLlsHOzq6swte7qpamUodARERU4UmaAOXk5CA6OhpBQUGqMiMjIwQFBeHIkSMvtO6RI0ciJCREbd1Fyc7ORlpamtqjPNkdk6h67lWn8iRzREREUpE0AUpJSYFCoYCjo6NauaOjIxISEkq93jVr1uDkyZOIiIgoUf2IiAjY2tqqHi4uLqXedlmI/N9V1fOxXRpIGAkREVHlIPklMF27desWxowZg9WrV8Pc3LxEy0yePBmpqamqx61bt8o4Su2c/TdV9Zy3wCAiInpxkk4oY29vD2NjYyQmJqqVJyYmPreDc1Gio6ORlJSEVq1aqcoUCgX279+PhQsXIjs7G8bGxmrLyOXyYvsTERERUeUiaQuQmZkZvL29ERUVpSpTKpWIioqCv79/qdbZpUsXnDt3DqdPn1Y9fHx8MGDAAJw+fbpQ8lPe5SqUUodARERU6Ug+pXBYWBgGDx4MHx8f+Pr6IjIyEhkZGRg6dCgAYNCgQXB2dlb158nJycHFixdVz2/fvo3Tp0/D2toa9evXR5UqVdCsWTO1bVhZWaF69eqFyiuCLWfuqJ7z6hcREZFuSJ4A9e3bF8nJyZgxYwYSEhLQsmVL7NixQ9Ux+ubNmzAyetJQdefOHXh5ealez5s3D/PmzUNgYCD27t2r7/DL3K37j1XPr3zeXcJIiIiIKg+ZEEJIHUR5k5aWBltbW6SmpsLGxkbSWHovPozoGw8AAPFzQiSNhYiIqDzT5vxd6UaBVTYFMz+bGvP6FxERka4wASrnLtzJHwLvW6+axJEQERFVHkyAyrnj8fmXvw5duydxJERERJUHE6ByLO+pIfBedapKFwgREVElwwSoHMvIUaief/l65bmjPRERkdSYAJVj528/uQWGew1rCSMhIiKqXJgAlWNVzJ9M0ySTcRQYERGRrjABKseuJqZLHQIREVGlxASoHLtxL0PqEIiIiColJkDl2KbTt6UOgYiIqFJiAlSOPX0fMCIiItIdJkBERERkcJgAVQC9W9WWOgQiIqJKhQlQBWBixCHwREREusQEqJy6dT9T9dzV3krCSIiIiCofJkDl1OWER6rn3Zo5SRgJERFR5cMEqJwyfuqyl2t1SwkjISIiqnyYAJVTS/dfVz3nbTCIiIh0iwlQOZWQliV1CERERJUWE6ByytLMWOoQiIiIKi0mQOWUi11+v5/adhYSR0JERFT5MAEqp878+xAAUN/BWtpAiIiIKiEmQOXU3dT8PkB7LydLHAkREVHlwwSIiIiIDA4ToHKOfYCIiIh0jwlQOWdnaSZ1CERERJUOE6By6FFWruq5vTUTICIiIl1jAlQO5SqE6vmk7o0ljISIiKhyYgJUDmXlKlTPGzlVkTASIiKiyokJUDl09Po9qUMgIiKq1ExKUmnz5s0lXmGvXr1KHQzle/pGqERERKR7JUqAQkND1V7LZDIIIdReF1AoFKAXE5PwSOoQiIiIKrUSXQJTKpWqx99//42WLVti+/btePjwIR4+fIht27ahVatW2LFjR1nHaxDa1q8udQhERESVWolagJ42duxYLFmyBO3atVOVBQcHw9LSEu+99x4uXbqk0wANUUuXqjh07R6seEd4IiKiMqF1J+jY2FhUrVq1ULmtrS3i4+N1EBLZWpgCALo2dZI4EiIiospJ6wSodevWCAsLQ2JioqosMTEREydOhK+vr06DM1RXEtMBAOamHKRHRERUFrQ+w/7www+4e/cu6tSpg/r166N+/fqoU6cObt++jR9++KEsYjQ4uy7mJ5d/nLojcSRERESVk9Z9gBo0aICzZ89i165diImJAQA0btwYQUFBaqPBqPRSH+ffCsPEmMeTiIioLGiVAOXm5sLCwgKnT59G165d0bVr17KKiwB09nCQOgQiIqJKSatLYKampqhTpw7n+tETJkBERERlQ+s+QFOnTsWUKVNw//59nQWxaNEiuLq6wtzcHH5+fjh27FiRdS9cuIDevXvD1dUVMpkMkZGRhepERESgdevWqFKlChwcHBAaGorLly/rLF59aejI+4ARERGVBa0ToIULF2L//v2oVasWGjVqhFatWqk9tLV27VqEhYUhPDwcJ0+eRIsWLRAcHIykpCSN9TMzM+Hm5oY5c+bAyUnzMPF9+/Zh5MiROHr0KHbt2oXc3Fx07doVGRkZWsenb0/PsH3jXqaEkRAREVVeWneCfva2GC9qwYIFGDZsGIYOHQoAWLJkCf766y+sWLECkyZNKlS/devWaN26NQBofB9AoRmpV65cCQcHB0RHR6NDhw46jV/XTt58qHruVsNKukCIiIgqMa0ToPDwcJ1tPCcnB9HR0Zg8ebKqzMjICEFBQThy5IjOtpOamgoAqFatmsb3s7OzkZ2drXqdlpams21r627qY9VzN3smQERERGVB0pn2UlJSoFAo4OjoqFbu6OiIhIQEnWxDqVRi7NixaNu2LZo1a6axTkREBGxtbVUPFxcXnWy7NBTKJ5fATIw5ESIREVFZ0PoMq1AoMG/ePPj6+sLJyQnVqlVTe5Q3I0eOxPnz57FmzZoi60yePBmpqamqx61bt/QYoboT8Q8k2zYREZGh0DoBmjlzJhYsWIC+ffsiNTUVYWFheO2112BkZIRPPvlEq3XZ29vD2NhY7bYaQP6tNYrq4KyNUaNGYevWrdizZw9q165dZD25XA4bGxu1h1S86lSVbNtERESGQusEaPXq1Vi2bBnGjx8PExMT9O/fH8uXL8eMGTNw9OhRrdZlZmYGb29vREVFqcqUSiWioqLg7++vbWgqQgiMGjUKmzZtwu7du1GvXr1Sr0vfEtKypA6BiIio0tO6E3RCQgI8PT0BANbW1qoOxi+//DKmT5+udQBhYWEYPHgwfHx84Ovri8jISGRkZKhGhQ0aNAjOzs6IiIgAkN9x+uLFi6rnt2/fxunTp2FtbY369esDyL/s9euvv+LPP/9ElSpVVP2JbG1tYWFhoXWM+nTwaorUIRAREVV6WidAtWvXVt0M1d3dHX///TdatWqF48ePQy6Xax1A3759kZycjBkzZiAhIQEtW7bEjh07VB2jb968CSOjJw1Vd+7cgZeXl+r1vHnzMG/ePAQGBmLv3r0AgMWLFwMAOnbsqLatH3/8EUOGDNE6Rn2yszIDAFSRa/2nISIiohKSiadn3iuBSZMmwcbGBlOmTMHatWvx1ltvwdXVFTdv3sS4ceMwZ86csopVb9LS0mBra4vU1FS99wdynfSX6nn8nBC9bpuIiKgi0+b8rXUzw9MJTt++fVG3bl0cPnwYDRo0QM+ePbWPljSq72AtdQhERESV1gtfZ2nTpg3atGmji1joKdeS0qUOgYiIqNLSOgGqU6cOOnbsiMDAQHTs2BHu7u5lEZfBm/FyE6lDICIiqrS0HgY/e/ZsmJubY+7cuWjQoAFcXFzw1ltvYdmyZbh69WpZxGiQHucqpA6BiIio0tK6Beitt97CW2+9BQC4e/cu9u3bh61bt+KDDz6AUqmEQsETty40rSXdZIxERESVXan6AGVmZuLgwYPYu3cv9uzZg1OnTqFZs2aFhp2Tdp4ekKfd2DwiIiLShtYJUEBAAE6dOoXGjRujY8eOmDRpEjp06AA7O7uyiM+gJD96ckf66tZmEkZCRERUuWndBygmJgZWVlbw8PCAh4cHGjduzORHR7LzlKrnjWvyEhgREVFZ0ToBunfvHnbv3o02bdpg586daNu2LZydnfHmm29i2bJlZRGjwZDJnjw3Ndb6T0NEREQlpPVM0E8TQiA6OhoLFy7E6tWrK00naKlmgj56/R76Lc2/oSxngSYiItJOmc4EffLkSezduxd79+7FwYMH8ejRI3h6emL06NEIDAwsddAE3LyXKXUIREREBkHrBMjX1xdeXl4IDAzEsGHD0KFDB9ja2pZFbAantl3+nerlJrz8RUREVJa0ToDu37+v9xuEGork9PxRYA0ceR8wIiKisqR1U4ONjQ0ePnyI5cuXY/Lkybh//z6A/Etjt2/f1nmAhqSg43NccobEkRAREVVuWrcAnT17Fl26dEHVqlURHx+PYcOGoVq1ati4cSNu3ryJn3/+uSziNAgJqVkAgNb1qkkcCRERUeWmdQtQWFgYhg4diqtXr8Lc3FxV3qNHD+zfv1+nwRmaWw/yO0FfTeSd4ImIiMqS1gnQ8ePHMXz48ELlzs7OSEhI0ElQhmrd8VsAgNsPH0scCRERUeWmdQIkl8uRlpZWqPzKlSuoUaOGToIyVB0a5h+/To14HImIiMqS1glQr1698OmnnyI3NxcAIJPJcPPmTXz88cfo3bu3zgM0JGf/TQUAVLeWSxwJERFR5aZ1AjR//nykp6fDwcEBjx8/RmBgIOrXrw9ra2vMmjWrLGI0GAWXvjZE/ytxJERERJWb1qPAbG1tsWvXLhw8eBBnz55Feno6WrVqhaCgoLKIj4iIiEjntE6ACrRr1w7t2rVTvT558iRmzJiBrVu36iQwQza5u4fUIRAREVVqWl0C27lzJyZMmIApU6bg+vXrAICYmBiEhoaidevWUCqVZRKkofF3ry51CERERJVaiVuAfvjhB9Wkhw8ePMDy5cuxYMECjB49Gn379sX58+fRuHHjsoy1UhNCqJ6bmxpLGAkREVHlV+IWoK+//hpz585FSkoK1q1bh5SUFHz33Xc4d+4clixZwuTnBWXnPWk9q1XVQsJIiIiIKr8SJ0CxsbF44403AACvvfYaTExM8OWXX6J27dplFpwhSX6UrXpuzrvBExERlakSn2kfP34MS0tLAPlz/8jlctSsWbPMAjNkJsZMgIiIiMqSVqPAli9fDmtrawBAXl4eVq5cCXt7e7U6H374oe6iMyA5ivxLYFXMSz0wj4iIiEqoxGfbOnXqYNmyZarXTk5OWLVqlVodmUzGBKiU0h7nz6xtwQ7QREREZa7ECVB8fHwZhkF5yvxRYElP9QUiIiKissHOJuVEyn+JT5OaNhJHQkREVPkxASonHucqAADp2XkSR0JERFT5MQEqJ3acTwAA3LyfKXEkRERElR8ToHIiK4+3ESEiItIXJkDlRO5/CVADB2uJIyEiIqr8SpUAxcbGYtq0aejfvz+SkpIAANu3b8eFCxd0GpwhOXL9HgDgalK6xJEQERFVflonQPv27YOnpyf++ecfbNy4Eenp+SfsM2fOIDw8XOcBGoqOjWoAAF735q1FiIiIyprWCdCkSZPw+eefY9euXTAzM1OVd+7cGUePHtVpcIbE9L/bX9TnJTAiIqIyp3UCdO7cObz66quFyh0cHJCSkqKToAzR/YwcAIBM4jiIiIgMgdYJUNWqVXH37t1C5adOnYKzs3Opgli0aBFcXV1hbm4OPz8/HDt2rMi6Fy5cQO/eveHq6gqZTIbIyMgXXmd5YGdpCgBQCCFxJERERJWf1glQv3798PHHHyMhIQEymQxKpRKHDh3ChAkTMGjQIK0DWLt2LcLCwhAeHo6TJ0+iRYsWCA4OVnWuflZmZibc3NwwZ84cODk56WSd5cGdh1kAgFq2FhJHQkREVPlpnQDNnj0bHh4ecHFxQXp6Opo0aYIOHTogICAA06ZN0zqABQsWYNiwYRg6dCiaNGmCJUuWwNLSEitWrNBYv3Xr1vjyyy/Rr18/yOVynayzPLh4Nw0AcOFOqsSREBERVX4lvhlqATMzMyxbtgzTp0/H+fPnkZ6eDi8vLzRo0EDrjefk5CA6OhqTJ09WlRkZGSEoKAhHjhzRen2lXWd2djays5/chDQtLa1U29YFDyfeC4yIiKisaZ0AFahTpw7q1KnzQhtPSUmBQqGAo6OjWrmjoyNiYmL0ts6IiAjMnDmzVNsjIiKiikfrBCgsLExjuUwmg7m5OerXr49XXnkF1apVe+Hg9GXy5Mlq+5WWlgYXFxdJYjEx5jgwIiKisqZ1AnTq1CmcPHkSCoUCjRo1AgBcuXIFxsbG8PDwwHfffYfx48fj4MGDaNKkSbHrsre3h7GxMRITE9XKExMTi+zg/DylWadcLi+yP5G++bhWnMSRiIiootK6E/Qrr7yCoKAg3LlzB9HR0YiOjsa///6Ll156Cf3798ft27fRoUMHjBs37rnrMjMzg7e3N6KiolRlSqUSUVFR8Pf31za0MltnWXuUlat6zvYfIiKisqd1C9CXX36JXbt2wcbmSWddW1tbfPLJJ+jatSvGjBmDGTNmoGvXriVaX1hYGAYPHgwfHx/4+voiMjISGRkZGDp0KABg0KBBcHZ2RkREBID8Ts4XL15UPb99+zZOnz4Na2tr1K9fv0TrLG/+vvCktaqalVkxNYmIiEgXtE6AUlNTkZSUVOjyVnJysmr0VNWqVZGTk1Oi9fXt2xfJycmYMWMGEhIS0LJlS+zYsUPVifnmzZswMnrSUHXnzh14eXmpXs+bNw/z5s1DYGAg9u7dW6J1ljc2Fqaq52bGpbo/LREREWlBJoR2Uw8PGDAAR44cwfz589G6dWsAwPHjxzFhwgQEBARg1apVWLNmDebNm4cTJ06USdBlLS0tDba2tkhNTVVr6Sorh6+l4M3l/wAA4ueElPn2iIiIKiNtzt9atwB9//33GDduHPr164e8vLz8lZiYYPDgwfjqq68AAB4eHli+fHkpQjdM2XlKAICHUxWJIyEiIjIMWrcAFUhPT8f169cBAG5ubrC2rjx3Mdd3C1DYutPYePI2ALYAERERlVaZtgAVsLa2RvPmzUu7OD3lcsIjqUMgIiIyKKVKgE6cOIF169bh5s2bhTo7b9y4USeBGZK29e1x4U4aung4SB0KERGRQdB6yNGaNWsQEBCAS5cuYdOmTcjNzcWFCxewe/du2NralkWMlZ65Sf6f4XGuQuJIiIiIDEOp7gb/1VdfYcuWLTAzM8PXX3+NmJgY9OnT54XvDWaoCjphOVe1kDQOIiIiQ6F1AhQbG4uQkPyOumZmZsjIyIBMJsO4ceOwdOlSnQdoCHZdzJ8I8UpSusSREBERGQatEyA7Ozs8epTfadfZ2Rnnz58HADx8+BCZmZm6jc5AxPzXCfrMrYfSBkJERGQgtO4E3aFDB+zatQuenp544403MGbMGOzevRu7du1Cly5dyiJGg9GtaeluAEtERETa0ToBWrhwIbKysgAAU6dOhampKQ4fPozevXtj2rRpOg/QkDzILNntQ4iIiOjFaJUA5eXlYevWrQgODgYAGBkZYdKkSWUSmCHq78tO5ERERPqgVR8gExMTjBgxQtUCRLp1OZETIhIREemD1p2gfX19cfr06TIIhVrU5jxKRERE+qB1H6APPvgAYWFhuHXrFry9vWFlZaX2Pm+PUXp3HrJljYiISB+0ToD69esHAPjwww9VZTKZDEIIyGQyKBSczbi0ujXjKDAiIiJ90DoBiouLK4s4DFZaVq7quamx1lckiYiIqBS0ToDq1q1bFnEYrMzsJy1mthamEkZCRERkOErV5LBq1Sq0bdsWtWrVwo0bNwAAkZGR+PPPP3UanCFQCqF6bmbCFiAiIiJ90PqMu3jxYoSFhaFHjx54+PChqs9P1apVERkZqev4Kr1b93n7ECIiIn3TOgH69ttvsWzZMkydOhXGxsaqch8fH5w7d06nwRmCqpZmUodARERkcLROgOLi4uDl5VWoXC6XIyMjQydBGZJchRIA4GRjLnEkREREhkPrBKhevXoaJ0LcsWMHGjdurIuYDErOfwmQqYlM4kiIiIgMh9ajwMLCwjBy5EhkZWVBCIFjx47ht99+Q0REBJYvX14WMVZq6Vl5ADgEnoiISJ+0ToDeffddWFhYYNq0acjMzMSbb76JWrVq4euvv1ZNkkglp1DmjwK7/eCxxJEQEREZDq0TIAAYMGAABgwYgMzMTKSnp8PBwUHXcRmMvP8SoDrVLCWOhIiIyHBofd3l888/V80GbWlpyeTnBWXn5U8jUM2Ko8GIiIj0ResEaP369ahfvz4CAgLw3XffISUlpSziMhgFl77kpsbPqUlERES6onUCdObMGZw9exYdO3bEvHnzUKtWLYSEhODXX39FZiYn9dOWtXn+Vci4lHSJIyEiIjIcpRp61LRpU8yePRvXr1/Hnj174OrqirFjx8LJiXcz19bjnPxLYF4udhJHQkREZDheeOy1lZUVLCwsYGZmhtzc3OcvQGo2nboNANhzOUniSIiIiAxHqRKguLg4zJo1C02bNoWPjw9OnTqFmTNnIiEhQdfxVXqpj/OTRhMjToRIRESkL1oPg2/Tpg2OHz+O5s2bY+jQoejfvz+cnZ3LIjaD8O9/naAdqvBWGERERPqidQLUpUsXrFixAk2aNFErVyqV2LZtG15++WWdBWcITIxkyFMKVLfmMHgiIiJ90ToBmjVrltrra9euYcWKFVi5ciWSk5PZD0hLwc2c8NfZu+jaxFHqUIiIiAxGqfoAPX78GD///DM6dOiARo0a4fDhw5gxYwb+/fdfXcdX6f119i4A4PZD3gqDiIhIX7RKgI4fP47hw4fDyckJkZGReOWVVyCTyfDdd99hxIgRcHRkK0Zpbf0vESIiIqKyV+JLYM2bN0daWhrefPNNHD58GE2bNgUATJo0qcyCMyTjuzaSOgQiIiKDUeIWoMuXL6NDhw7o1KlToQ7Q9OJy8pRSh0BERGQwSpwAXb9+HY0aNcL777+P2rVrY8KECTh16hRkMs5fowuWZrwXGBERkb6UOAFydnbG1KlTce3aNaxatQoJCQlo27Yt8vLysHLlSly5cqUs46z0mjnbSB0CERGRwSjVKLDOnTvjl19+wd27d7Fw4ULs3r0bHh4eaN68eamCWLRoEVxdXWFubg4/Pz8cO3as2Prr16+Hh4cHzM3N4enpiW3btqm9n56ejlGjRqF27dqwsLBAkyZNsGTJklLFVpaEEChoQLM003pGAiIiIiqlF7oXmK2tLT744AOcOHECJ0+eRMeOHbVex9q1axEWFobw8HCcPHkSLVq0QHBwMJKSNN8b6/Dhw+jfvz/eeecdnDp1CqGhoQgNDcX58+dVdcLCwrBjxw788ssvuHTpEsaOHYtRo0Zh8+bNpd3VMpGZo4AQ+c9tLUylDYaIiMiAyIQoOAVLw8/PD61bt8bChQsB5M8o7eLigtGjR2scYda3b19kZGRg69atqrI2bdqgZcuWqlaeZs2aoW/fvpg+fbqqjre3N7p3747PP//8uTGlpaXB1tYWqampsLEpu0tTKenZ8Pn8fwCAuIge7E9FRET0ArQ5f7/w3eBfRE5ODqKjoxEUFKQqMzIyQlBQEI4cOaJxmSNHjqjVB4Dg4GC1+gEBAdi8eTNu374NIQT27NmDK1euoGvXrhrXmZ2djbS0NLWHPmT/N/LLzNiIyQ8REZEeSZoApaSkQKFQFJpA0dHRscg7yyckJDy3/rfffosmTZqgdu3aMDMzQ7du3bBo0SJ06NBB4zojIiJga2ureri4uLzgnpVMwdD3HAWHwBMREemTpAlQWfn2229x9OhRbN68GdHR0Zg/fz5GjhyJ//3vfxrrT548GampqarHrVu39BLnxTv6aWkiIiIidZIOPbK3t4exsTESExPVyhMTE+Hk5KRxGScnp2LrP378GFOmTMGmTZsQEhICIH8W69OnT2PevHmFLp8BgFwuh1wu18UuacXJ1lzv2yQiIqJSJEDffPONxnKZTAZzc3PUr18fHTp0gLHx8yf2MzMzg7e3N6KiohAaGgogvxN0VFQURo0apXEZf39/REVFYezYsaqyXbt2wd/fHwCQm5uL3NxcGBmpN24ZGxtDqSxfl5oUyvz+5272VhJHQkREZFi0ToC++uorJCcnIzMzE3Z2dgCABw8ewNLSEtbW1khKSoKbmxv27NlTor40YWFhGDx4MHx8fODr64vIyEhkZGRg6NChAIBBgwbB2dkZERERAIAxY8YgMDAQ8+fPR0hICNasWYMTJ05g6dKlAAAbGxsEBgZi4sSJsLCwQN26dbFv3z78/PPPWLBggba7W6ay8xQAgJv3MyWOhIiIyLBo3Qdo9uzZaN26Na5evYp79+7h3r17uHLlCvz8/PD111/j5s2bcHJywrhx40q0vr59+2LevHmYMWMGWrZsidOnT2PHjh2qjs43b97E3btP7pQeEBCAX3/9FUuXLkWLFi2wYcMG/PHHH2jWrJmqzpo1a9C6dWsMGDAATZo0wZw5czBr1iyMGDFC290tUwevpQAA8pSSzkRARERkcLSeB8jd3R2///47WrZsqVZ+6tQp9O7dG9evX8fhw4fRu3dvtcSlItHXPEBrjt3EpI3nAADxc0LKbDtERESGoEznAbp79y7y8vIKlefl5amGoteqVQuPHj3SdtUGx8wk//C3b2AvcSRERESGResEqFOnThg+fDhOnTqlKjt16hTef/99dO7cGQBw7tw51KtXT3dRVlJ3U7MAcDg8ERGRvmmdAP3www+oVq0avL29VcPHfXx8UK1aNfzwww8AAGtra8yfP1/nwVY2a47fBADcy8iROBIiIiLDovUoMCcnJ+zatQsxMTG4cuUKAKBRo0Zo1KiRqk6nTp10F2ElFtrSGd/uvgZPZ1upQyEiIjIopZ4I0cPDAx4eHrqMxeDkKvL7n9fjPEBERER6pXUCpFAosHLlSkRFRSEpKanQ5IK7d+/WWXCV3ZJ9sQCAzWfu4Jv+XhJHQ0REZDi0ToDGjBmDlStXIiQkBM2aNeNdzF+AV52qOHXzIXzq2kkdChERkUHROgFas2YN1q1bhx49epRFPAbFxc4Sp24+hJ9bNalDISIiMihajwIzMzND/fr1yyIWg7P5zB0AwKI9sRJHQkREZFi0ToDGjx+Pr7/+GlpOIE3F8HVlCxAREZE+aX0J7ODBg9izZw+2b9+Opk2bwtTUVO39jRs36iw4Q+FSzVLqEIiIiAyK1glQ1apV8eqrr5ZFLAarhQvnASIiItInrROgH3/8sSziMEgOVeRIepQNN3trqUMhIiIyKFr3ASLdSUnPBgAYG3EqASIiIn0qUQtQq1atEBUVBTs7O3h5eRU798/Jkyd1FlxlZ2+d3wJkLS/1hNxERERUCiU6877yyiuQy+UAgNDQ0LKMx6Bk5+XPom1hZixxJERERIalRAlQeHi4xuf0YlIf5wIAzE15JZKIiEifSn3tJScnR+O9wOrUqfPCQRmCp+dRMjVmAkRERKRPWidAV65cwTvvvIPDhw+rlQshIJPJoFAodBZcZZanfJIAmZvwEhgREZE+aZ0ADR06FCYmJti6dStq1qzJm6GWUq7iScuZqQmPIRERkT5pnQCdPn0a0dHR8PDwKIt4DEZuHi+BERERSUXrM2+TJk2QkpJSFrEYlOT0LNVzE84DREREpFdaJ0Bz587FRx99hL179+LevXtIS0tTe1DJmBk/6ffDy4hERET6pfUlsKCgIABAly5d1MrZCVo7Of8dJwtTdoAmIiLSN60ToD179pRFHAbnWlIGAOBxLhNGIiIifdM6AQoMDCyLOAwO7/9FREQknVJNhPjw4UMcO3ZM40SIgwYN0klglV3Bpa9GjlUkjoSIiMjwaJ0AbdmyBQMGDEB6ejpsbGzUOvDKZDImQCX0IDMHAHA58ZHEkRARERkerUeBjR8/Hm+//TbS09Px8OFDPHjwQPW4f/9+WcRYKa3+54bUIRARERksrROg27dv48MPP4SlpWVZxGMwAhs6SB0CERGRwdI6AQoODsaJEyfKIhaDYmthCgB4qYmjxJEQEREZHq37AIWEhGDixIm4ePEiPD09YWpqqvZ+r169dBZcZZaZkwcAMONtMIiIiPRO6wRo2LBhAIBPP/200HucCLHkElLzb4WhFOI5NYmIiEjXtE6Anh32TqVTcAks9XGuxJEQEREZHl5/kcih2PwbynImaCIiIv3TugVI06Wvp82YMaPUwRiSf+Lypwy4nMB5gIiIiPRN6wRo06ZNaq9zc3MRFxcHExMTuLu7MwEqoR6eNfHX2bsY6F9X6lCIiIgMjtYJ0KlTpwqVpaWlYciQIXj11Vd1EpQhkP83+sveSi5xJERERIZHJ32AbGxsMHPmTEyfPl0XqzMIp/99CAAwM2E3LCIiIn3T2dk3NTUVqamppVp20aJFcHV1hbm5Ofz8/HDs2LFi669fvx4eHh4wNzeHp6cntm3bVqjOpUuX0KtXL9ja2sLKygqtW7fGzZs3SxVfWahX3QoAR4ERERFJQetLYN98843aayEE7t69i1WrVqF79+5aB7B27VqEhYVhyZIl8PPzQ2RkJIKDg3H58mU4OBS+XcThw4fRv39/RERE4OWXX8avv/6K0NBQnDx5Es2aNQMAxMbGol27dnjnnXcwc+ZM2NjY4MKFCzA3N9c6vrKSo8ifTsClmoXEkRARERkemRDazcRXr149tddGRkaoUaMGOnfujMmTJ6NKlSpaBeDn54fWrVtj4cKFAPLnGXJxccHo0aMxadKkQvX79u2LjIwMbN26VVXWpk0btGzZEkuWLAEA9OvXD6ampli1apVWsRRIS0uDra0tUlNTYWNjU6p1PE+fJUdwLP4+vhvQCj08a5bJNoiIiAyJNudvrS+BxcXFqT1iY2Nx9OhRzJ49GyYm2jUo5eTkIDo6GkFBQU8CMjJCUFAQjhw5onGZI0eOqNUH8u9PVlBfqVTir7/+QsOGDREcHAwHBwf4+fnhjz/+0G5Hy1j0zQcAeCsMIiIiKejk7JudnY0FCxYUah16npSUFCgUCjg6qt8Q1NHREQkJCRqXSUhIKLZ+UlIS0tPTMWfOHHTr1g1///03Xn31Vbz22mvYt29fkfGnpaWpPcpaAwdrAE8uhREREZH+lDgBys7OxuTJk+Hj44OAgABVi8qKFStQr149fPXVVxg3blxZxVliBbfqeOWVVzBu3Di0bNkSkyZNwssvv6y6RPasiIgI2Nraqh4uLi5lHmeeMv/Ko52lWZlvi4iIiNSVOAGaMWMGFi9eDFdXV8THx+ONN97Ae++9h8jISCxYsADx8fH4+OOPtdq4vb09jI2NkZiYqFaemJgIJycnjcs4OTkVW9/e3h4mJiZo0qSJWp3GjRsXOQps8uTJqlFsqampuHXrllb7URp5/7X8mBrLynxbREREpK7ECdD69evx888/Y8OGDfj777+hUCiQl5eHM2fOoF+/fjA2NtZ642ZmZvD29kZUVJSqTKlUIioqCv7+/hqX8ff3V6sPALt27VLVNzMzQ+vWrXH58mW1OleuXEHduppnXZbL5bCxsVF7lLVcRX4LkCn7ABEREeldiXst//vvv/D29gYANGvWDHK5HOPGjYNM9mItGGFhYRg8eDB8fHzg6+uLyMhIZGRkYOjQoQCAQYMGwdnZGREREQCAMWPGIDAwEPPnz0dISAjWrFmDEydOYOnSpap1Tpw4EX379kWHDh3QqVMn7NixA1u2bMHevXtfKFZdepSVP/+PCVuAiIiI9K7ECZBCoYCZ2ZP+KiYmJrC2tn7hAPr27Yvk5GTMmDEDCQkJaNmyJXbs2KHq6Hzz5k0YGT1pJQkICMCvv/6KadOmYcqUKWjQoAH++OMP1RxAAPDqq69iyZIliIiIwIcffohGjRrh999/R7t27V44Xl1Jy8oDABgbMQEiIiLStxLPA2RkZITu3btDLs+/d9WWLVvQuXNnWFlZqdXbuHGj7qPUM33MA9R0xg5k5Ciwe3wg3Gq8eCJJRERk6LQ5f5e4BWjw4MFqr996663SRUcAnvQBMjfVvu8UERERvZgSJ0A//vhjWcZhUIQQqvl/2AmaiIhI/3j2lUBB6w/Au8ETERFJgWdfCRSMAAMA9oEmIiLSPyZAEni617m1XLv7pxEREdGLYwIkgZy8J7NAv+g8SkRERKQ9JkASKEiAeCd4IiIiafAMLIGMnPxJEEs0ARMRERHpHBMgCRRMPZmZo5A2ECIiIgPFBEgC2Xn5iU/d6pYSR0JERGSYmABJIJt9gIiIiCTFM7AE7qXnAAA4AIyIiEgaTIAkUHD7izsPsySOhIiIyDAxAZJAwX3APJ1tJY6EiIjIMDEBkkBWbn4naLkpDz8REZEUeAaWwL8PHgMAjNkJiIiISBJMgCRQ1cIUAHAnlX2AiIiIpMAESAIFfYCa1LSROBIiIiLDxARIAqp7gZnw8BMREUmBZ2AJXE9OBwDImQARERFJgmdgCdhZmQEA7qY+ljgSIiIiw8QESAIFN0N1tbeSNhAiIiIDxQRIArn/dYK2NDWROBIiIiLDxARIAgUJkKkJ5wEiIiKSAhMgCVxOeASAd4MnIiKSCs/AErC3lgMAHmTmSBwJERGRYWICJAEjo/xLX0425hJHQkREZJiYAEkg778+QHITY4kjISIiMkxMgCSQp8wfB29izE7QREREUmACJIF/rt8HACj+S4SIiIhIv5gASaDgZqj7riRLHAkREZFhYgIkoZeb15Q6BCIiIoPEBEgC7jXyb4Fha2EmcSRERESGiQmQBGKTMwAAZrwbPBERkSR4BpaAQ5X8iRCNOAiMiIhIEkyAJFBwLzBrOW+GSkREJAUmQBLIyctPgHgJjIiISBo8A0sgI0cBADDlzVCJiIgkwTOwngnxZPJDJkBERETS4BlYz7L/u/wFABZmvBcYERGRFMpFArRo0SK4urrC3Nwcfn5+OHbsWLH1169fDw8PD5ibm8PT0xPbtm0rsu6IESMgk8kQGRmp46hL5/F/l78AwJx9gIiIiCQh+Rl47dq1CAsLQ3h4OE6ePIkWLVogODgYSUlJGusfPnwY/fv3xzvvvINTp04hNDQUoaGhOH/+fKG6mzZtwtGjR1GrVq2y3o0Sy8jJAwDIZIAJL4ERERFJQvIz8IIFCzBs2DAMHToUTZo0wZIlS2BpaYkVK1ZorP/111+jW7dumDhxIho3bozPPvsMrVq1wsKFC9Xq3b59G6NHj8bq1athamqqj10pkYIRYIL3QSUiIpKMpAlQTk4OoqOjERQUpCozMjJCUFAQjhw5onGZI0eOqNUHgODgYLX6SqUSAwcOxMSJE9G0adPnxpGdnY20tDS1R1kpuBFqdSveBoOIiEgqkiZAKSkpUCgUcHR0VCt3dHREQkKCxmUSEhKeW3/u3LkwMTHBhx9+WKI4IiIiYGtrq3q4uLhouSclV9ACJGf/HyIiIslUurNwdHQ0vv76a6xcuRIyWcnuNTF58mSkpqaqHrdu3Sqz+FIf5wJg/x8iIiIpSXoWtre3h7GxMRITE9XKExMT4eTkpHEZJyenYusfOHAASUlJqFOnDkxMTGBiYoIbN25g/PjxcHV11bhOuVwOGxsbtUdZMfovKbt5P7PMtkFERETFkzQBMjMzg7e3N6KiolRlSqUSUVFR8Pf317iMv7+/Wn0A2LVrl6r+wIEDcfbsWZw+fVr1qFWrFiZOnIidO3eW3c6UUHZe/jD45rVtJY6EiIjIcEl+N86wsDAMHjwYPj4+8PX1RWRkJDIyMjB06FAAwKBBg+Ds7IyIiAgAwJgxYxAYGIj58+cjJCQEa9aswYkTJ7B06VIAQPXq1VG9enW1bZiamsLJyQmNGjXS785pkJ37333AeAmMiIhIMpInQH379kVycjJmzJiBhIQEtGzZEjt27FB1dL558yaMjJ4kCwEBAfj1118xbdo0TJkyBQ0aNMAff/yBZs2aSbULWrmbmgWAt8EgIiKSkkwIzkjzrLS0NNja2iI1NVXn/YHWnbiFjzachaONHP9MCXr+AkRERFQi2py/2QyhZ3mK/HyzRe2q0gZCRERkwJgA6VnufxMhmnIeICIiIsnwLKxnBQkQO0ETERFJh2dhPbuekgEAMDUu2SSNREREpHtMgPTM3loOgBMhEhERSYkJkJ4VTITYrBYnQiQiIpIKEyA9i0vOvwRmbmoscSRERESGiwmQnsn/S3zSs/MkjoSIiMhwMQHSMxOj/M7PdpZmEkdCRERkuJgA6VlOXv4w+KqWphJHQkREZLiYAOnZ1aRHAAA5J0IkIiKSDM/CeiY3ye8DlP1fSxARERHpHxMgPSuYALFgPiAiIiLSPyZAenYvIwcAYG1uInEkREREhosJkJ7duJc/A7SpEW+FQUREJBUmQHrmXNUCAGDGTtBERESS4VlYzwo6P1vJeQmMiIhIKkyA9OxhZn4fILYAERERSYdnYT3LUwoAgJkxDz0REZFUeBbWI+V/yQ8AWJjxZqhERERSYQKkR1l5CtVzSyZAREREkmECpEeZOU8SIHMTJkBERERSYQKkRzlP3f7CiPMAERERSYYJkB7lKvITIGsOgSciIpIUEyA9KkiACu4HRkRERNJgAqRHBZMgmnIIPBERkaR4JtajR1l5AAClEM+pSURERGWJCZAeFVz4SknPkTQOIiIiQ8cESI9y/usD5OFUReJIiIiIDBsTID16mJkLAJDzPmBERESS4plYj47H3wfAG6ESERFJjWdiPfKrVx1WZsYI8awpdShEREQGjTPy6VFI85oIac7kh4iISGpsASIiIiKDwwSIiIiIDA4TICIiIjI4TICIiIjI4DABIiIiIoPDBIiIiIgMTrlIgBYtWgRXV1eYm5vDz88Px44dK7b++vXr4eHhAXNzc3h6emLbtm2q93Jzc/Hxxx/D09MTVlZWqFWrFgYNGoQ7d+6U9W4QERFRBSF5ArR27VqEhYUhPDwcJ0+eRIsWLRAcHIykpCSN9Q8fPoz+/fvjnXfewalTpxAaGorQ0FCcP38eAJCZmYmTJ09i+vTpOHnyJDZu3IjLly+jV69e+twtIiIiKsdkQgghZQB+fn5o3bo1Fi5cCABQKpVwcXHB6NGjMWnSpEL1+/bti4yMDGzdulVV1qZNG7Rs2RJLlizRuI3jx4/D19cXN27cQJ06dZ4bU1paGmxtbZGamgobG5tS7hkRERHpkzbnb0lbgHJychAdHY2goCBVmZGREYKCgnDkyBGNyxw5ckStPgAEBwcXWR8AUlNTIZPJULVqVZ3ETURERBWbpLfCSElJgUKhgKOjo1q5o6MjYmJiNC6TkJCgsX5CQoLG+llZWfj444/Rv3//IrPB7OxsZGdnq16npaVpsxtERERUwUjeB6gs5ebmok+fPhBCYPHixUXWi4iIgK2trerh4uKixyiJiIhI3yRNgOzt7WFsbIzExES18sTERDg5OWlcxsnJqUT1C5KfGzduYNeuXcVeC5w8eTJSU1NVj1u3bpVyj4iIiKgikDQBMjMzg7e3N6KiolRlSqUSUVFR8Pf317iMv7+/Wn0A2LVrl1r9guTn6tWr+N///ofq1asXG4dcLoeNjY3ag4iIiCovSfsAAUBYWBgGDx4MHx8f+Pr6IjIyEhkZGRg6dCgAYNCgQXB2dkZERAQAYMyYMQgMDMT8+fMREhKCNWvW4MSJE1i6dCmA/OTn9ddfx8mTJ7F161YoFApV/6Bq1arBzMzsuTEVDIxjXyAiIqKKo+C8XaIB7qIc+Pbbb0WdOnWEmZmZ8PX1FUePHlW9FxgYKAYPHqxWf926daJhw4bCzMxMNG3aVPz111+q9+Li4gQAjY89e/aUKJ5bt24VuQ4++OCDDz744KN8P27duvXcc73k8wCVR0qlEnfu3EGVKlUgk8l0uu60tDS4uLjg1q1bvNRWxnis9YfHWn94rPWHx1p/dHWshRB49OgRatWqBSOj4nv5SH4JrDwyMjJC7dq1y3Qb7GukPzzW+sNjrT881vrDY60/ujjWtra2JapXqYfBExEREWnCBIiIiIgMDhMgPZPL5QgPD4dcLpc6lEqPx1p/eKz1h8daf3is9UeKY81O0ERERGRw2AJEREREBocJEBERERkcJkBERERkcJgAERERkcFhAlQGFi1aBFdXV5ibm8PPzw/Hjh0rtv769evh4eEBc3NzeHp6Ytu2bXqKtOLT5lgvW7YM7du3h52dHezs7BAUFPTcvw09oe3/dYE1a9ZAJpMhNDS0bAOsRLQ91g8fPsTIkSNRs2ZNyOVyNGzYkN8jJaTtsY6MjESjRo1gYWEBFxcXjBs3DllZWXqKtuLav38/evbsiVq1akEmk+GPP/547jJ79+5Fq1atIJfLUb9+faxcuVK3QZXo5lhUYmvWrBFmZmZixYoV4sKFC2LYsGGiatWqIjExUWP9Q4cOCWNjY/HFF1+IixcvimnTpglTU1Nx7tw5PUde8Wh7rN98802xaNEicerUKXHp0iUxZMgQYWtrK/799189R17xaHusC8TFxQlnZ2fRvn178corr+gn2ApO22OdnZ0tfHx8RI8ePcTBgwdFXFyc2Lt3rzh9+rSeI694tD3Wq1evFnK5XKxevVrExcWJnTt3ipo1a4px48bpOfKKZ9u2bWLq1Kli48aNAoDYtGlTsfWvX78uLC0tRVhYmLh48aL49ttvhbGxsdixY4fOYmICpGO+vr5i5MiRqtcKhULUqlVLREREaKzfp08fERISolbm5+cnhg8fXqZxVgbaHutn5eXliSpVqoiffvqprEKsNEpzrPPy8kRAQIBYvny5GDx4MBOgEtL2WC9evFi4ubmJnJwcfYVYaWh7rEeOHCk6d+6sVhYWFibatm1bpnFWNiVJgD766CPRtGlTtbK+ffuK4OBgncXBS2A6lJOTg+joaAQFBanKjIyMEBQUhCNHjmhc5siRI2r1ASA4OLjI+pSvNMf6WZmZmcjNzUW1atXKKsxKobTH+tNPP4WDgwPeeecdfYRZKZTmWG/evBn+/v4YOXIkHB0d0axZM8yePRsKhUJfYVdIpTnWAQEBiI6OVl0mu379OrZt24YePXroJWZDoo9zI2+GqkMpKSlQKBRwdHRUK3d0dERMTIzGZRISEjTWT0hIKLM4K4PSHOtnffzxx6hVq1ahDxmpK82xPnjwIH744QecPn1aDxFWHqU51tevX8fu3bsxYMAAbNu2DdeuXcMHH3yA3NxchIeH6yPsCqk0x/rNN99ESkoK2rVrByEE8vLyMGLECEyZMkUfIRuUos6NaWlpePz4MSwsLF54G2wBIoM0Z84crFmzBps2bYK5ubnU4VQqjx49wsCBA7Fs2TLY29tLHU6lp1Qq4eDggKVLl8Lb2xt9+/bF1KlTsWTJEqlDq3T27t2L2bNn47vvvsPJkyexceNG/PXXX/jss8+kDo1KgS1AOmRvbw9jY2MkJiaqlScmJsLJyUnjMk5OTlrVp3ylOdYF5s2bhzlz5uB///sfmjdvXpZhVgraHuvY2FjEx8ejZ8+eqjKlUgkAMDExweXLl+Hu7l62QVdQpfm/rlmzJkxNTWFsbKwqa9y4MRISEpCTkwMzM7MyjbmiKs2xnj59OgYOHIh3330XAODp6YmMjAy89957mDp1KoyM2KagK0WdG21sbHTS+gOwBUinzMzM4O3tjaioKFWZUqlEVFQU/P39NS7j7++vVh8Adu3aVWR9yleaYw0AX3zxBT777DPs2LEDPj4++gi1wtP2WHt4eODcuXM4ffq06tGrVy906tQJp0+fhouLiz7Dr1BK83/dtm1bXLt2TZVkAsCVK1dQs2ZNJj/FKM2xzszMLJTkFCSegrfV1Cm9nBt11p2ahBD5wyrlcrlYuXKluHjxonjvvfdE1apVRUJCghBCiIEDB4pJkyap6h86dEiYmJiIefPmiUuXLonw8HAOgy8hbY/1nDlzhJmZmdiwYYO4e/eu6vHo0SOpdqHC0PZYP4ujwEpO22N98+ZNUaVKFTFq1Chx+fJlsXXrVuHg4CA+//xzqXahwtD2WIeHh4sqVaqI3377TVy/fl38/fffwt3dXfTp00eqXagwHj16JE6dOiVOnTolAIgFCxaIU6dOiRs3bgghhJg0aZIYOHCgqn7BMPiJEyeKS5cuiUWLFnEYfEXw7bffijp16ggzMzPh6+srjh49qnovMDBQDB48WK3+unXrRMOGDYWZmZlo2rSp+Ouvv/QcccWlzbGuW7euAFDoER4erv/AKyBt/6+fxgRIO9oe68OHDws/Pz8hl8uFm5ubmDVrlsjLy9Nz1BWTNsc6NzdXfPLJJ8Ld3V2Ym5sLFxcX8cEHH4gHDx7oP/AKZs+ePRq/fwuO7+DBg0VgYGChZVq2bCnMzMyEm5ub+PHHH3Uak0wIttsRERGRYWEfICIiIjI4TICIiIjI4DABIiIiIoPDBIiIiIgMDhMgIiIiMjhMgIiIiMjgMAEiIiIig8MEiIhIh+Lj4yGTyXD69GmpQyEqd/bv34+ePXuiVq1akMlk+OOPP7RehxAC8+bNQ8OGDSGXy+Hs7IxZs2ZpvR4mQET0QpKTk/H++++jTp06kMvlcHJyQnBwMA4dOqSqU9ovutIYMmQIZDIZ5syZo1b+xx9/QCaT6SUGItIsIyMDLVq0wKJFi0q9jjFjxmD58uWYN28eYmJisHnzZvj6+mq9Ht4NnoheSO/evZGTk4OffvoJbm5uSExMRFRUFO7duydZTObm5pg7dy6GDx8OOzs7yeLQJd7ZnSqD7t27o3v37kW+n52djalTp+K3337Dw4cP0axZM8ydOxcdO3YEAFy6dAmLFy/G+fPn0ahRIwBAvXr1ShULW4CIqNQePnyIAwcOYO7cuejUqRPq1q0LX19fTJ48Gb169QIAuLq6AgBeffVVyGQy1WsA+PPPP9GqVSuYm5vDzc0NM2fORF5enup9mUyGxYsXo3v37rCwsICbmxs2bNjw3LiCgoLg5OSEiIiIIut88sknaNmypVpZZGSkWnxDhgxBaGgoZs+eDUdHR1StWhWffvop8vLyMHHiRFSrVg21a9fGjz/+WGj9MTExCAgIgLm5OZo1a4Z9+/apvX/+/Hl0794d1tbWcHR0xMCBA5GSkqJ6v2PHjhg1ahTGjh0Le3t7BAcHP3e/iSq6UaNG4ciRI1izZg3Onj2LN954A926dcPVq1cBAFu2bIGbmxu2bt2KevXqwdXVFe+++y7u37+v9baYABFRqVlbW8Pa2hp//PEHsrOzNdY5fvw4AODHH3/E3bt3Va8PHDiAQYMGYcyYMbh48SK+//57rFy5stC1/OnTp6N37944c+YMBgwYgH79+uHSpUvFxmVsbIzZs2fj22+/xb///vtC+7h7927cuXMH+/fvx4IFCxAeHo6XX34ZdnZ2+OeffzBixAgMHz680HYmTpyI8ePH49SpU/D390fPnj1VrWIPHz5E586d4eXlhRMnTmDHjh1ITExEnz591Nbx008/wczMDIcOHcKSJUteaD+IyrubN2/ixx9/xPr169G+fXu4u7tjwoQJaNeunepHxvXr13Hjxg2sX78eP//8M1auXIno6Gi8/vrr2m9Qp7dWJSKDs2HDBmFnZyfMzc1FQECAmDx5sjhz5oxaHQBi06ZNamVdunQRs2fPVitbtWqVqFmzptpyI0aMUKvj5+cn3n///SLjefrO823atBFvv/22EEKITZs2iae/8sLDw0WLFi3Ulv3qq69E3bp11dZVt25doVAoVGWNGjUS7du3V73Oy8sTVlZW4rfffhNCCBEXFycAiDlz5qjq5Obmitq1a4u5c+cKIYT47LPPRNeuXdW2fevWLQFAXL58WQiRfydyLy+vIveTqKJ79nth69atAoCwsrJSe5iYmIg+ffoIIYQYNmyY2udECCGio6MFABETE6PV9tkHiIheSO/evRESEoIDBw7g6NGj2L59O7744gssX74cQ4YMKXK5M2fO4NChQ2otPgqFAllZWcjMzISlpSUAwN/fX205f3//Eo+wmjt3Ljp37owJEyZovV8FmjZtCiOjJ43ljo6OaNasmeq1sbExqlevjqSkpEJxFjAxMYGPj4+q5erMmTPYs2cPrK2tC20vNjYWDRs2BAB4e3uXOm6iiiY9PR3GxsaIjo6GsbGx2nsFn5WaNWvCxMRE9RkBgMaNGwPIb0Eq6BdUEkyAiOiFmZub46WXXsJLL72E6dOn491330V4eHixCVB6ejpmzpyJ1157TeP6dKFDhw4IDg7G5MmTC8ViZGSE/B+hT+Tm5hZah6mpqdprmUymsUypVJY4rvT0dPTs2RNz584t9F7NmjVVz62srEq8TqKKzsvLCwqFAklJSWjfvr3GOm3btkVeXh5iY2Ph7u4OALhy5QoAoG7dulptjwkQEelckyZN1Ia9m5qaQqFQqNVp1aoVLl++jPr16xe7rqNHj2LQoEFqr728vEocy5w5c9CyZctCvwxr1KiBhIQECCFUw+N1OXfP0aNH0aFDBwBAXl4eoqOjMWrUKAD5+/7777/D1dUVJib8GibDkZ6ejmvXrqlex8XF4fTp06hWrRoaNmyIAQMGYNCgQZg/fz68vLyQnJyMqKgoNG/eHCEhIQgKCkKrVq3w9ttvIzIyEkqlEiNHjsRLL72k1ipUEuwETUSldu/ePXTu3Bm//PILzp49i7i4OKxfvx5ffPEFXnnlFVU9V1dXREVFISEhAQ8ePAAAzJgxAz///DNmzpyJCxcu4NKlS1izZg2mTZumto3169djxYoVuHLlCsLDw3Hs2DFVIlESnp6eGDBgAL755hu18o4dOyI5ORlffPEFYmNjsWjRImzfvv0Fjoa6RYsWYdOmTYiJicHIkSPx4MEDvP322wCAkSNH4v79++jfvz+OHz+O2NhY7Ny5E0OHDi2UKBJVJidOnICXl5fqR0xYWBi8vLwwY8YMAPmDJQYNGoTx48ejUaNGCA0NxfHjx1GnTh0A+S23W7Zsgb29PTp06ICQkBA0btwYa9as0T4YHfRjIiIDlZWVJSZNmiRatWolbG1thaWlpWjUqJGYNm2ayMzMVNXbvHmzqF+/vjAxMVHrZLxjxw4REBAgLCwshI2NjfD19RVLly5VvQ9ALFq0SLz00ktCLpcLV1dXsXbt2mJjeroTdIG4uDhhZmYmnv3KW7x4sXBxcRFWVlZi0KBBYtasWYU6QT+7rsDAQDFmzBi1srp164qvvvpKtS0A4tdffxW+vr7CzMxMNGnSROzevVttmStXrohXX31VVK1aVVhYWAgPDw8xduxYoVQqi9wOEemOTIhnLoITEZUTMpkMmzZtQmhoqNShEFElw0tgREREZHCYABEREZHB4fADIiq3eIWeiMoKW4CIiIjI4DABIiIiIoPDBIiIiIgMDhMgIiIiMjhMgIiIiMjgMAEiIiIig8MEiIiIiAwOEyAiIiIyOEyAiIiIyOD8H3GcuqvWC1K5AAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Plotting running average reward for FrozenLake-v1\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABxCElEQVR4nO3deVhU1RsH8O+wDTuiCAiiCKi4gyDuQkqSmWWZWyZKv0xLKyVNTYXMFDVSSk1LU1s0l9KyMsxQck8D930BF5TNBQRkmzm/P2hGRkAZlrnAfD/PMw8zZ849970XZu7LuefcKxNCCBARERHpEQOpAyAiIiLSNSZAREREpHeYABEREZHeYQJEREREeocJEBEREekdJkBERESkd5gAERERkd5hAkRERER6hwkQERER6R0mQERVSCaT4cMPP5Q6DNJjsbGxkMlkiI2NlToUyaxduxYymQyJiYlSh0I1GBMgqvFUX2aqh5GREZydnTF69GgkJSVJHV6Nc/bsWchkMpiamuLevXtSh1OjjB49WuNvSS6Xo0WLFggLC0Nubq7U4Uni9OnTePXVV+Hs7Ay5XA4nJyeMGDECp0+fljo0DQEBARq/u7Ie/AeEystI6gCIyuujjz5Cs2bNkJubi0OHDmHt2rXYt28fTp06BVNTU6nDAwA8ePAARkbSfqy+//57ODo64u7du/jxxx/x+uuvSxpPTSOXy7Fq1SoAQEZGBn755RfMmTMHly9fxrp16ySOTre2bNmC4cOHo379+vjf//6HZs2aITExEV9//TV+/PFHbNiwAS+++KLUYQIAZsyYofG3fOTIEXz++ef44IMP0KpVK3V5+/bt0aZNGwwbNgxyuVyKUKm2EEQ13Jo1awQAceTIEY3yqVOnCgBi48aNEkVW8yiVSuHq6ipCQ0PFiy++KAICAiSJIzs7W5L1PsmoUaOEhYWFRplSqRRdunQRMplMJCcnSxRZ+SmVSpGTk1Pm+7t37xYAxO7dux/bzqVLl4S5ubnw9PQUqampGu+lpaUJT09PYWFhIS5fvlwVYZdbVlZWuept3ry5XNtJVBaeAqNaq2fPngCAy5cvq8sCAgIQEBBQou7o0aPh6uqqfp2YmAiZTIbIyEh89dVXcHd3h1wuR6dOnXDkyJESy1paWiIpKQkDBw6EpaUlGjZsiMmTJ0OhUGjUfbQL/sMPP4RMJsOlS5cwevRo1KtXDzY2NggJCUFOTo7Gsg8ePMA777wDOzs7WFlZ4fnnn0dSUpJW3fr79+9HYmIihg0bhmHDhmHPnj24ceOG+v3nnnsObm5upS7btWtX+Pr6apR9//338PHxgZmZGerXr49hw4bh+vXrGnUCAgLQtm1bxMXFoVevXjA3N8cHH3wAAPjll1/Qv39/ODk5QS6Xw93dHXPmzCmx3wBg2bJlcHNzg5mZGfz8/LB3795Sf595eXkIDw+Hh4cH5HI5XFxc8P777yMvL69c++hRMpkMPXr0gBACV65c0Xjvjz/+QM+ePWFhYQErKyv0799f49TQtm3bIJPJcOLECXXZTz/9BJlMhpdeekmjrVatWmHo0KHq12vWrEHv3r1hb28PuVyO1q1bY/ny5SXic3V1xXPPPYcdO3bA19cXZmZm+PLLLwEAN27cwMCBA2FhYQF7e3tMmjSp3Pvhk08+QU5ODr766is0bNhQ4z07Ozt8+eWXyM7OxsKFCwEAP/74I2QyGf7+++8SbX355ZeQyWQ4deqUuuzcuXN4+eWXUb9+fZiamsLX1xfbtm3TWE51evvvv//GW2+9BXt7ezRu3Lhc8T9OaWOAVPsxNjZWvR/btWunHiu1ZcsWtGvXDqampvDx8cHRo0dLtFuebaLagwkQ1VqqLzdbW9sKt7F+/Xp88sknGDt2LD7++GMkJibipZdeQkFBgUY9hUKBoKAgNGjQAJGRkfD398enn36Kr776qlzrGTJkCO7fv4+IiAgMGTIEa9euxezZszXqjB49GkuWLMGzzz6LBQsWwMzMDP3799dqe9atWwd3d3d06tQJAwYMgLm5OX744Qf1+0OHDkVCQkKJJO/q1as4dOgQhg0bpi6bO3cugoOD0bx5cyxatAgTJ05ETEwMevXqVWJs0e3bt9GvXz94eXkhKioKTz31FICiA5GlpSVCQ0Px2WefwcfHB2FhYZg2bZrG8suXL8eECRPQuHFjLFy4ED179sTAgQM1kjcAUCqVeP755xEZGYkBAwZgyZIlGDhwIBYvXqyRXGirtL+l7777Dv3794elpSUWLFiAWbNm4cyZM+jRo4e6fo8ePSCTybBnzx71cnv37oWBgQH27dunLktLS8O5c+fQq1cvjW1u2rQpPvjgA3z66adwcXHBW2+9hWXLlpWI7/z58xg+fDiefvppfPbZZ/Dy8sKDBw/Qp08f7NixAxMmTMCMGTOwd+9evP/+++Xa5l9//RWurq7qfyQe1atXL7i6uuL3338HAPW+2LRpU4m6GzduRJs2bdC2bVsAReOKunTpgrNnz2LatGn49NNPYWFhgYEDB2Lr1q0lln/rrbdw5syZUv82qtKlS5fwyiuvYMCAAYiIiMDdu3cxYMAArFu3DpMmTcKrr76K2bNn4/LlyxgyZAiUSqV6WW23iWoBqbugiJ5EdQrsr7/+EmlpaeL69evixx9/FA0bNhRyuVxcv35dXdff31/4+/uXaGPUqFGiadOm6tcJCQkCgGjQoIG4c+eOuvyXX34RAMSvv/6qsSwA8dFHH2m06e3tLXx8fDTKAIjw8HD16/DwcAFAvPbaaxr1XnzxRdGgQQP167i4OAFATJw4UaPe6NGjS7RZlvz8fNGgQQMxY8YMddkrr7wiOnTooH6dkZEh5HK5eO+99zSWXbhwoZDJZOLq1atCCCESExOFoaGhmDt3rka9kydPCiMjI41yf39/AUCsWLGiREylnaoZO3asMDc3F7m5uUIIIfLy8kSDBg1Ep06dREFBgbre2rVrBQCN3+d3330nDAwMxN69ezXaXLFihQAg9u/fX9buEUI8PAWWlpYm0tLSxKVLl0RkZKSQyWSibdu2QqlUCiGEuH//vqhXr54YM2aMxvLJycnCxsZGo7xNmzZiyJAh6tcdO3YUgwcPFgDE2bNnhRBCbNmyRQAQx48ff+y+CQoKEm5ubhplTZs2FQBEdHS0RnlUVJQAIDZt2qQuy87OFh4eHk88NXTv3j0BQLzwwgtl1hFCiOeff14AEJmZmUIIIYYPHy7s7e1FYWGhus6tW7eEgYGBxuejT58+ol27durfsRBFp+66desmmjdvri5TfbZ79Oih0WZ5PO4UmKrdhIQEdZlqPx44cEBdtmPHDgFAmJmZqf/2hRDiyy+/LNF2ebeJag/2AFGtERgYiIYNG8LFxQUvv/wyLCwssG3btkp1mQ8dOlTjv37Vf8OPngoBgHHjxmm87tmzZ6n1SlPasrdv30ZmZiYAIDo6GkDRf8LFvf322+VqHyg6XXP79m0MHz5cXTZ8+HAcP35cfdrG2toa/fr1w6ZNmyCEUNfbuHEjunTpgiZNmgAoOh2gVCoxZMgQpKenqx+Ojo5o3rw5du/erbFuuVyOkJCQEjGZmZmpn9+/fx/p6eno2bMncnJycO7cOQDAv//+i9u3b2PMmDEaA8hHjBhRondv8+bNaNWqFTw9PTXi6t27NwCUiKs02dnZaNiwIRo2bAgPDw9MnjwZ3bt3xy+//AKZTAYA2LlzJ+7du4fhw4drrMfQ0BCdO3fWWE/Pnj2xd+9e9TYeP34cb7zxBuzs7NTle/fuRb169dQ9JI/um4yMDKSnp8Pf3x9XrlxBRkaGRszNmjVDUFCQRtn27dvRqFEjvPzyy+oyc3NzvPHGG0/cB/fv3wcAWFlZPbae6n3V3+nQoUORmpqqMcX+xx9/hFKpVPfA3blzB7t27VL3eqr23e3btxEUFISLFy+WmL05ZswYGBoaPjHuymrdujW6du2qft25c2cAQO/evdV/+8XLVZ/vimwT1XycBUa1xrJly9CiRQtkZGRg9erV2LNnT6VneRT/0gMengK5e/euRrmpqWmJcRK2trYl6lVkPdbW1rh69SoMDAzQrFkzjXoeHh7lah8oGq/TrFkzyOVyXLp0CQDg7u4Oc3NzrFu3DvPmzQNQdBD7+eefcfDgQXTr1g2XL19GXFwcoqKi1G1dvHgRQgg0b9681HUZGxtrvHZ2doaJiUmJeqdPn8bMmTOxa9cu9UFURXWQv3r1aqnbamRkpDFuSxXX2bNnS/wuVFJTU0stL87U1BS//vorgKIxNAsXLkRqaqpGQnLx4kUAUCdWj7K2tlY/79mzJ1asWIFLly7h8uXLkMlk6Nq1qzoxGjNmDPbu3Yvu3bvDwODh/5z79+9HeHg4Dh48WGI8WEZGBmxsbNSvH/27AIr2m4eHhzppU2nZsuUT94EqsVElQmV5NFF65plnYGNjg40bN6JPnz4AipJnLy8vtGjRAkDRaSYhBGbNmoVZs2aV2m5qaiqcnZ0fu33V4dHPoWofu7i4lFqu+nxXZJuo5mMCRLWGn5+fepDuwIED0aNHD7zyyis4f/48LC0tARQNaC3es6FS2qBbAGX+1/loG5X977S866mozMxM/Prrr8jNzS01aVm/fj3mzp0LmUymHhu0adMmdOvWDZs2bYKBgQEGDx6srq9UKiGTyfDHH3+UGrtqf6sUTx5U7t27B39/f1hbW+Ojjz6Cu7s7TE1NER8fj6lTp2qMrygvpVKJdu3aYdGiRaW+/+iBrDSGhoYIDAxUvw4KCoKnpyfGjh2rHtCqiu27776Do6NjiTaK91T16NEDALBnzx5cuXIFHTt2hIWFBXr27InPP/8cWVlZOHr0KObOnate5vLly+jTpw88PT2xaNEiuLi4wMTEBNu3b8fixYtL7JvS9m9l2NjYoFGjRhqDt0tz4sQJODs7qxM+uVyuHvPyxRdfICUlBfv371cn18DDfTd58uQSvVYqjya7Vb19ZSnrc/ikz2dFtolqPiZAVCsZGhoiIiICTz31FJYuXaoeOGlra1vqaSlVL0NN1bRpUyiVSiQkJGgkMKqenCfZsmULcnNzsXz5ctjZ2Wm8d/78ecycORP79+9Hjx49YGFhgeeeew6bN2/GokWLsHHjRvTs2RNOTk7qZdzd3SGEQLNmzdT/2WsrNjYWt2/fxpYtWzQG/yYkJGjUa9q0qXpbVYOnAaCwsBCJiYlo3769RlzHjx9Hnz59SvR8VFSjRo0wadIkzJ49G4cOHUKXLl3g7u4OALC3t9dIlkrTpEkTNGnSBHv37sWVK1fUp1F79eqF0NBQbN68GQqFQmMf/Prrr8jLy8O2bds0eiXKcwpPpWnTpjh16hSEEBr74vz58+Va/rnnnsPKlSuxb98+dRJX3N69e5GYmIixY8dqlA8dOhTffPMNYmJicPbsWQghNAagq2YZGhsbP3Hf1RZ1cZuIs8CoFgsICICfnx+ioqLUV/F1d3fHuXPnkJaWpq53/Phx7N+/X6owy0X1X+UXX3yhUb5kyZJyLf/999/Dzc0N48aNw8svv6zxmDx5MiwtLTUu8jd06FDcvHkTq1atwvHjx0vMoHrppZdgaGiI2bNnl+ilEkLg9u3bT4xJ9V918eXz8/NLbKOvry8aNGiAlStXorCwUF2+bt26EqcYhwwZgqSkJKxcubLE+h48eIDs7OwnxlWat99+G+bm5pg/fz6Aot+HtbU15s2bV2JGIACNvy+g6DTYrl27cPjwYXUC5OXlBSsrK8yfPx9mZmbw8fFR1y9t32RkZGDNmjXljvnZZ5/FzZs38eOPP6rLVNPay2PKlCkwMzPD2LFjS/w+79y5g3HjxsHc3BxTpkzReC8wMBD169fHxo0bsXHjRvj5+WmcwrK3t0dAQAC+/PJL3Lp1q8R6H913tUFd3CZiDxDVclOmTMHgwYOxdu1ajBs3Dq+99hoWLVqEoKAg/O9//0NqaipWrFiBNm3alBiDUpP4+Phg0KBBiIqKwu3bt9GlSxf8/fffuHDhAgA8trfj5s2b2L17N955551S35fL5QgKCsLmzZvx+eefw9jYGM8++yysrKwwefJkGBoaYtCgQRrLuLu74+OPP8b06dORmJiIgQMHwsrKCgkJCdi6dSveeOMNTJ48+bHb1K1bN9ja2mLUqFF45513IJPJ8N1335VIqExMTPDhhx/i7bffRu/evTFkyBAkJiZi7dq1cHd319j2kSNHYtOmTRg3bhx2796N7t27Q6FQ4Ny5c9i0aZP6WjnaatCgAUJCQvDFF1/g7NmzaNWqFZYvX46RI0eiY8eOGDZsGBo2bIhr167h999/R/fu3bF06VL18j179sS6devU1xQCipKcbt26YceOHQgICNAYI9W3b1+YmJhgwIABGDt2LLKysrBy5UrY29uXeoAtzZgxY7B06VIEBwcjLi4OjRo1wnfffQdzc/NyLd+8eXN88803GDFiBNq1a1fiStDp6en44Ycf1L1hKsbGxnjppZewYcMGZGdnIzIyskTby5YtQ48ePdCuXTuMGTMGbm5uSElJwcGDB3Hjxg0cP368XDHWJHVxm/Se7ieeEWmnrCtBCyGEQqEQ7u7uwt3dXT2N9vvvvxdubm7CxMREeHl5iR07dpQ5Df6TTz4p0SYemXZe2tWDhXg4xf1xy6rqpKWllbpNxafpZmdni/Hjx4v69esLS0tLMXDgQHH+/HkBQMyfP7/M/fPpp58KACImJqbMOqop5b/88ou6bMSIEQKACAwMLHO5n376SfTo0UNYWFgICwsL4enpKcaPHy/Onz+vruPv7y/atGlT6vL79+8XXbp0EWZmZsLJyUm8//776qnHj05f/vzzz0XTpk2FXC4Xfn5+Yv/+/cLHx0c888wzGvXy8/PFggULRJs2bYRcLhe2trbCx8dHzJ49W2RkZJS5LUKU/bsUQojLly8LQ0NDMWrUKHXZ7t27RVBQkLCxsRGmpqbC3d1djB49Wvz7778ay54+fVoAEK1atdIo//jjjwUAMWvWrBLr27Ztm2jfvr0wNTUVrq6uYsGCBWL16tWlTt/u379/qTFfvXpVPP/888Lc3FzY2dmJd999V0RHR2t1heQTJ06I4cOHi0aNGgljY2Ph6Ogohg8fLk6ePFnmMjt37hQAhEwm07gMRXGXL18WwcHBwtHRURgbGwtnZ2fx3HPPiR9//FFd53Gf7SepyDT40vYjADF+/HiNsrK+H8qzTVR7yISoolGYRFTljh07Bm9vb3z//fcYMWKE1OHolFKpRMOGDfHSSy+VesqLiKgyOAaIqIZ48OBBibKoqCgYGBhoDKCti3Jzc0ucGvv2229x586dUm9tQkRUWRwDRFRDLFy4EHFxcXjqqadgZGSEP/74A3/88QfeeOONck3vrs0OHTqESZMmYfDgwWjQoAHi4+Px9ddfo23bthrT84mIqgpPgRHVEDt37sTs2bNx5swZZGVloUmTJhg5ciRmzJihcd2ZuigxMRHvvPMODh8+jDt37qB+/fp49tlnMX/+fNjb20sdHhHVQUyAiIiISO9wDBARERHpHSZAREREpHfq9sCCClIqlbh58yasrKyq7HL7REREVL2EELh//z6cnJw0bj5cGiZApbh582adn3VDRERUV12/fh2NGzd+bB0mQKWwsrICULQDVXdBJiIiopotMzMTLi4u6uP44zABKoXqtJe1tTUTICIiolqmPMNXOAiaiIiI9A4TICIiItI7TICIiIhI7zABIiIiIr3DBIiIiIj0DhMgIiIi0js1IgFatmwZXF1dYWpqis6dO+Pw4cNl1l25ciV69uwJW1tb2NraIjAw8LH1x40bB5lMhqioqGqInIiIiGojyROgjRs3IjQ0FOHh4YiPj0eHDh0QFBSE1NTUUuvHxsZi+PDh2L17Nw4ePAgXFxf07dsXSUlJJepu3boVhw4dgpOTU3VvBhEREdUikidAixYtwpgxYxASEoLWrVtjxYoVMDc3x+rVq0utv27dOrz11lvw8vKCp6cnVq1aBaVSiZiYGI16SUlJePvtt7Fu3ToYGxvrYlOIiIiolpA0AcrPz0dcXBwCAwPVZQYGBggMDMTBgwfL1UZOTg4KCgpQv359dZlSqcTIkSMxZcoUtGnTpsrjJiIiotpN0lthpKenQ6FQwMHBQaPcwcEB586dK1cbU6dOhZOTk0YStWDBAhgZGeGdd94pVxt5eXnIy8tTv87MzCzXckRERFQ71ep7gc2fPx8bNmxAbGwsTE1NAQBxcXH47LPPEB8fX657gQBAREQEZs+eXZ2hEhERUQ0i6SkwOzs7GBoaIiUlRaM8JSUFjo6Oj102MjIS8+fPx59//on27dury/fu3YvU1FQ0adIERkZGMDIywtWrV/Hee+/B1dW11LamT5+OjIwM9eP69euV3rba5ua9BxBCSB0GUaUplAIZDwqkDoOIajhJe4BMTEzg4+ODmJgYDBw4EADUA5onTJhQ5nILFy7E3LlzsWPHDvj6+mq8N3LkSI3TYQAQFBSEkSNHIiQkpNT25HI55HJ55TamljqffB9BUXvUr5e90hH92zeSMCKiilMoBdw/2A4AiBrqhYHezhJHREQ1leSnwEJDQzFq1Cj4+vrCz88PUVFRyM7OVicrwcHBcHZ2RkREBICi8T1hYWFYv349XF1dkZycDACwtLSEpaUlGjRogAYNGmisw9jYGI6OjmjZsqVuN66Gyy9UaiQ/ADB+fTz6tX0WBgblO32oC6qeqZx8BXaeSXnsQU0IgcBFf+NyWjZm9m+F13u66SrMapFXqEDLmdEwMzbE6dlBOv29uE77Xf18dDdX7L+Ujh0Te9Wov43irt/JQc+Fu9WvJ248VukEyGfOTtzOzgcAnPkoCOYmVfeVmVeowIkbGfBtalvu0/VEVHUkT4CGDh2KtLQ0hIWFITk5GV5eXoiOjlYPjL527RoMDB6eqVu+fDny8/Px8ssva7QTHh6ODz/8UJeh13otZv5RarnbB9txYFpvONUzK1c7QghcvZ2DoKg9yCtUAgAS5/evkhhVCUBxEzceK7X9A5fS8cqqf9SvP/79bIkEaOORa7iTXYA3A9xLXZ8QAs2mb1e/Toh4VuPgVKBQovmMP9CzuR2++1/nCm1TeSXde4Du83cBAB4UKOD2X8/G3vefgkt9c9zKeIAFf5zD/EHtYWpsqF7ubnY+bC1MKrXuf67c1ni99kAigKK/japOBLS1Zn8COrjUQ8cmtuoypVJoJD8qc347g1nPta7QelLv56qTHwBoHbYDQNX8bSuUosTf9cW5/WBsaIBChRLZ+QrsvZiGCeuP4tcJPdCusY3W7d/PLYCNmTGu3clB0wYWlY5Zl7bE30BWXiGCu7pizf4E/HU2BV+N9EWb8B3qOpfm9kN2vgI2Zg8vc1KgUOLN7+PwwbOt0KS+OTxmaH7HDfdzwZwX2sLIsPTRH98cSMS87WdRoFBCKSqX9CqUApM2HsOYnm5a//7K69CV22jawByNbMr3XU2aZIIDP0rIzMyEjY0NMjIyYG1tLXU4WjlzMxMfbD2J3p726NWiIRytTdEl4uE1kob7NcH0Zz1xKTULL31x4LFtXZn35J6gRxMGlePhfdVfTA/yFcjJL8T55Pt4ZdU/iBrqBQdrU7RvbAMLeelfLnmFCvx1JhXj18eXue43erlhSlBLrDt0FR/+eqbMegsGtcPUn05qlH37mh96tWhYYp3+C2ORnJmrUX5uzjOQyYAjCXfx6tf/aLy36z1/1LcwQT3z8iccuQUKyI0Mnvhff/EemEeZGBkg/79kszRd3Rrghze6PDEWpVIg9kIqurg1UH/Rl/U7LU51sH7Ujbs56LGgKBG58HE/mBhV3TDDjAcF6DD7T42ywx/0gb216WP31aNJ7JN8suMclu2+XOb7W97qppF8VURZ8V6c2w/NZ5T8x2TT2K7wa1a/lCXK3/6Yns0wo3/ZyeDKPVew+K8LOPlhEAwf87kXQuCrPVfwTFtHrRKrxPRsFCqV8LC3emy9U0kZeG7JPgCAd5N6OHrt3mPr9/a0h4O1KX44fK3csTSyMcW3r/mhuYMVHuQrYGZiiO8OJmLWL6dL1DU1NoC9lSmu3cnBpbn9NJKnAoUS/1y5g8zcAng3qYdGNmYoUCjxzg9H8cepZHW9kO6uCB9Q+iVZsvMKMf+Pc/ju0FUAwM/ju8PLpR5iz6eiaQMLFCqUMDEy0NjXaffz0GnuX+rXxZNyIQQOXr6NFo5WsLOs/NCOvRfT0MjG9Im/t0ep/ga/fc0PwasPw9HaFIc+6FPpeJ5Em+M3E6BS1NYE6NFTAOXl29QWG8d2VY+dKM2jB5Fx38Uh+nRymfXL6+SHfWFl+vA/uLxCBX45ehPv/3Si0m2X1+rRvnht7b+VakMmA05+GATLYgmd78c7kZ5V1INw4eN++PZgIj7+/az6/TZO1jh98+ElF4p/if1yLAnvbjhWqZiKm9y3Bcb5u5f4z7f4gTJxfv/HJhIVoUqi8wuVeHfDUbwV4AEDA2DC+qNISM/Gb2/3wIMCBWZsPYn1Y7rAxMgAt7Py0cyu6Ms+v7Doy//A5XS8svKfJ6ytSPiA1phdLCG2NjVC/KynNbb9bnY+vOfsBAB8OdIH+YVKvP3DURz+oA/85sWUaPNRqt/V/dwC/Hk6Be9tPq5+b01IJ/Rq3lCdRCiUAgYyYM/FdIxafRj+LRri7wtppbZrbmKInHzFY9f5OE/6/R2d9XSpvYO3s/Lg8/HDA2pZPR/Fx1iptHCwxNwX26GTa+kJmhAC247fVP89R0/siZYOVpDJZPj2YCLCfjmNMx8F4UpaNlo1sn7s91BNUPz3oM3npbREXJvlj4U9jXrmJo/9ByXsudZYvT8BN+4+0Cg/8WFfWBf7nlX9TW49moQpP55Ah8Y2mPtiO7RqZI37uQX4POYiVu5NKDOW397ugbbOmr1ax67fg6FMhnaNbZBboIDnrOgSy3k6WiF6Yq9yb3NFMAGqpNqSAJ1Pvo+luy/h1+M3K9VO8Q/m4z6Q7/T2QHA3V/gW+6KsCgM6OOGTl9uX+oGprdwbWuByWrbUYZRQ/Mu732d7cfbW4695tXtyAL45kKg+BVYRM55thbnbzz65YhVJnN8fD/IVaBUWXaI8t0CBjAcF6FyOJKe4hlZypN1/eK2wv6cEYNHOC/jlWOU+e9pInN8fSqXAPwl30MWtvsbBtKwDTlkCWzlg1nOt4P9JbJl1ivcACyHg/0ksrt3JKbP+o70jquXKOlgbyABlLTz6fDbMCy94OZf6N/Ykxb9rtf2dAcCgjo3xU/wNrZZRKZ60t/vwzyfUfrKXvJ3x6ZAOuJCSpTGW9Kc3u2LQ8sdfyHjFqx0R0NJe49R9VWECVEm1IQFqG74DWXmFlW5nzgttMLKrq0ZZRXsBVgX7wsAAle5NKY3qw1vaf6DFtXSwwo5JvSr05VLcX6G9ELhoz5Mr6sD5j5+B3MgQt7PycOpmJkatLvvmv+U1sktTdZf74zza61DVPURVrfh/uo/GeizsaXh9tFOr9mzNjRE/62nIZLJq2XZVUuNWgV6PuS+2xSt+TaotNqAovpM3MjBg6b4n1n10rNKl1CwELvq7ymNaG9IJo9cceWI9T0crjPN3Vw+EL97r9zhfj/LFjtPJ2PRv2YnG0VlPY8IP8dh/6XaZdcqi+kzV9M+SLnjYW+KvUP8qbZMJUCXV9ASoUKEsMbjvUXaWJurTL2c/egamxgY4nHAHu86l4ss9V9T1yupWL34evjye1ItkaCDDsle8Me77ssf0lOX07KASY4Vu3M1BQyu5eiDpkuHeGNCh5E1vn/1sL87cykT0xJ6ws5TDzlKOc8mZmPv7Wey9mF6ifvFTco8emHZPDoBrA3PIZDIIIZB4OwdPRcZqvT3aaN/YBtsm9NAoy3hQgI1HrsHeylT95f7+j8ex6d8bWBnsizHfVk0COrqbKz58XnPcws9HkzBx47Eyl7GzlGOcvxs+2XFePSC+Kq0Z3Qk9mtvB2NAAz0Ttwbnk++r3lo/oiH7tHl7CoaKJhcqjn430rLwq7f1c93pndPewA1DyMzOyS1P0a+eIrm4Nnjgeq7tHgxIH4vABrTG6myuu3s6BS31zzPz5JH44XL3XN+vgUg+/jO8OoGhcS/EBy5URNzMQhUqBiO1nMS7AHZ6Omt/JQghM3nwCr3R2wdoDV/Fad1d4lzFG6252PoZ9dQgfvdAGQ786VGod1e+9QKFE1F8XHjseTGXui20xY+spjbI9U56Cs61ZiX/Y+rdrhHkvtSsxnu1RK171wb2cfEzbcrLMOtVx2ro6PK4XuKomzKgwAaqkmpwAlWeA6l+h/vCwt6yydR68fBvDV5b8sujX1hFLhnuXOqMi/tpd1Dc3gaGBDE71zNTjIYQQUCgFjAwNsGz3JXyy43yp63Spb4a97/eusm0oS16hAl/+fQXnk+9jzsC2qF+B2VPl7VJ+qmVDrAnxA1C0H3ILlDAzKeoC3nr0BiZtPK5R/90+zTHp6RZax5OamYs9F9MxefNxvBXgjmfbNXpsMvtmgDuWxz78kn+ufSMsfaVjmfXP3MyEvXVRMvnoGKLiOs39S+O0EQDMe7EdPtj68Au9q1sD9Ghuh9HdXLH2QGKZfw/AwwHPjypUKMuc1ZOTX6ievaWN0sY4AGX/135kRiCsTI2wJT4JX+65jKu3NU8VtW5kjTPFTjc+Ooi8+OBxoOLjTICif3hUf1cq5fneaONkjTd6uZV77FlpB9/XujfD6v2ljx05N+cZpN3PK/c4RW0Hr2urtF7iy/OeLXMAePgvp/DNwdJ7TZ90EH/c77D45/yvMyl4/dt/YW8lx+EZRdezKz4btLhTs4vGHOYVKnDs2j3sv5SOz3ddUr9/ZEYgdpxOxsyfT5VYtjhbc2O0drLWSKLXvd4Z/1y5jeYOVnjK0x6mRgbqz9iTeuGLOzfnGRgZyGBkaFDqPjAykOHSvGfL1VZ5MQGqpJqcAD36R/TjuK5o19gGJoYG2HE6BZ6OVnC1q/opr/8m3sHLK4rO6z46oK4ycgsU2HE6Wf2le/7jZ6BQCkmnWVdE8d6I4ge3QoUStzJyYWZiWK4ZGYUKJb45eBVudhZ4ytO+SmNsNv13PPppLz47JSe/EDLIShw8Hyc5IxeTNh7DmpBOpZ7PV/29Lhnujb5tHCA3qvpz/k/y5vdxGjNyVFQHrW8OJMKnqW2pCc+jin/5D/drgh8OX8Nfob1KnSGz/p9ruJSahbAB5Z+GfzsrD7bmJhqzL4d+eRD/JNwp1/LzXmyHVzo3KfW9uKt3MWh5yZmfR2YE4sSNe+jT6uE9Gcs6YKvG7cwZ2BYjuzR9bN3i1o/pjG7uRb1dxZOxo7Oexrnk+xi+8pBGL4Gur+F1JS0LTvXMHjsmpawk8tD0PnC0KZmYF5eSmVvmuLPy9IDcy8nH/dxChP1yCrvPp5U5SH3Tv9cRtfMC/gz1V0/I2HTkeqmTSkobs1VexX/nR2c9jZ1nU/D+jydw4sO+aP/fP4OPJuKFCiWe+jQWz3dwwrLdl9HQSo79U3tX6UxRgAlQpdWmBKiquw+p4i6m3Id7Q8sae6FAfTby63+w92L6Y6cj11RCCARExmK4XxOM8394/arSEo/yfh8cv34PLyzbr55y/aiE9GyN07vxs54us3d0zf4EjVl35YnpQb4CCiE0Zk3WFhX9Dn70qvvaLl8VMnMLYGpkWOVJR03CBKiSamoC9Oh5dSY/RPot9X4uPo+5iO8PXcOKVzvimba6v41NfqGyzIuq1tXvKCFEhU7Prd6XgI9+e5gsVvdpPn3EBKiSamoCVPw0Cz84RFRTvLrqH+y79HBSwU9vdkWHxvUqfIqlLsstUOBCyn20c7bhd3g10Ob4Xfv6H/VY8Rkv/OAQUU3x/eudkZKZi7O3MhHQsmrHrtU1psaGaN+4ntRhEJgAERFRFXCwNoVDKbP0iGoq9k/WQvI6PICNiIhIF3gkrSXe3XBU/Xz9mOq9CzkREVFdxwSolih+zyGfpuW/KzQRERGVxASoFlDWxjsGEhER1WBMgGqBPsVuKNi6Uc2Zlk9ERFRbMQGqBRLSs9XPt7/bU8JIiIiI6gYmQERERKR3mADVIkFtHJ5ciYiIiJ6ICVANt/6fa+rno7q6ShcIERFRHcIEqIb7YOtJ9fNuHnYSRkJERFR3MAEiIiIivcMEiIiIiPQOE6Baws7SROoQiIiI6gwmQDVYfqFS/fyVzk0ljISIiKhuYQJUg129/fACiG/39pAwEiIiorqFCVANdjMjV/3c2JC/KiIioqrCo2oNNmr1YalDICIiqpOYABEREZHeYQJEREREeocJEBEREemdGpEALVu2DK6urjA1NUXnzp1x+HDZY19WrlyJnj17wtbWFra2tggMDNSoX1BQgKlTp6Jdu3awsLCAk5MTgoODcfPmTV1sSpU5lZShfv5mgLuEkRAREdU9kidAGzduRGhoKMLDwxEfH48OHTogKCgIqamppdaPjY3F8OHDsXv3bhw8eBAuLi7o27cvkpKSAAA5OTmIj4/HrFmzEB8fjy1btuD8+fN4/vnndblZlbYg+pz6+WCfxhJGQkREVPfIhBBCygA6d+6MTp06YenSpQAApVIJFxcXvP3225g2bdoTl1coFLC1tcXSpUsRHBxcap0jR47Az88PV69eRZMmTZ7YZmZmJmxsbJCRkQFra2vtNqiKdIuIUU+DT5zfX5IYiIiIahNtjt+S9gDl5+cjLi4OgYGB6jIDAwMEBgbi4MGD5WojJycHBQUFqF+/fpl1MjIyIJPJUK9evcqGrDPFrwFEREREVctIypWnp6dDoVDAwcFBo9zBwQHnzp0rYylNU6dOhZOTk0YSVVxubi6mTp2K4cOHl5kN5uXlIS8vT/06MzOznFtAREREtZHkY4AqY/78+diwYQO2bt0KU1PTEu8XFBRgyJAhEEJg+fLlZbYTEREBGxsb9cPFxaU6w36i1Ez2/hAREVUnSRMgOzs7GBoaIiUlRaM8JSUFjo6Oj102MjIS8+fPx59//on27duXeF+V/Fy9ehU7d+587LnA6dOnIyMjQ/24fv16xTaoily7k6N+fvLDvhJGQkREVDdJmgCZmJjAx8cHMTEx6jKlUomYmBh07dq1zOUWLlyIOXPmIDo6Gr6+viXeVyU/Fy9exF9//YUGDRo8Ng65XA5ra2uNh5TW/XNN/dzK1FjCSIiIiOomSccAAUBoaChGjRoFX19f+Pn5ISoqCtnZ2QgJCQEABAcHw9nZGREREQCABQsWICwsDOvXr4erqyuSk5MBAJaWlrC0tERBQQFefvllxMfH47fffoNCoVDXqV+/PkxMTKTZUC1sPZokdQhERER1muQJ0NChQ5GWloawsDAkJyfDy8sL0dHR6oHR165dg4HBw46q5cuXIz8/Hy+//LJGO+Hh4fjwww+RlJSEbdu2AQC8vLw06uzevRsBAQHVuj1ERERU80l+HaCaSOrrALlO+139nNcAIiIiKp9acx0gerxPXi45uJuIiIgqjwlQDVbPvOaPVyIiIqqNmADVMMXPSLo3tJAwEiIiorqLCVANk52vUD+3ty55cUciIiKqPCZANczFlPvq5xYmhhJGQkREVHcxAaphjl2/p34uk8mkC4SIiKgOYwJUwyzddUnqEIiIiOo8JkA1zO3sfKlDICIiqvOYABEREZHeYQJUwzhYywEAL3g5SRwJERFR3cUEqIZJycwDAJhzBhgREVG1YQJUQyVn5EodAhERUZ3FBKiGGtrJReoQiIiI6iwmQDWULe8DRkREVG2YANUg93ML1M/lxhwDREREVF2YANUg1+88UD/v0NhGwkiIiIjqNiZANciucynq57wNBhERUfVhAlSDMOkhIiLSDSZANcjhhDtSh0BERKQXmADVIKn386QOgYiISC8wAapBChRKAICtubHEkRAREdVtTIBqkEupWQCAuzkFT6hJRERElcEEqAYyNeavhYiIqDrxSFsDffBsK6lDICIiqtOYANVAJ25kSB0CERFRncYEqAZ6rn0jqUMgIiKq05gA1UD1eCNUIiKiasUEqIbILHYjVDtLJkBERETViQlQDZFfqFQ/d7IxkzASIiKiuo8JUA1x9XaO+rmBAe8JRkREVJ2YANUQx6/fkzoEIiIivcEEqIZwsDaVOgQiIiK9USMSoGXLlsHV1RWmpqbo3LkzDh8+XGbdlStXomfPnrC1tYWtrS0CAwNL1BdCICwsDI0aNYKZmRkCAwNx8eLF6t6MSrmSliV1CERERHpD8gRo48aNCA0NRXh4OOLj49GhQwcEBQUhNTW11PqxsbEYPnw4du/ejYMHD8LFxQV9+/ZFUlKSus7ChQvx+eefY8WKFfjnn39gYWGBoKAg5Obm6mqztHaMp8CIiIh0RiaEEFIG0LlzZ3Tq1AlLly4FACiVSri4uODtt9/GtGnTnri8QqGAra0tli5diuDgYAgh4OTkhPfeew+TJ08GAGRkZMDBwQFr167FsGHDnthmZmYmbGxskJGRAWtr68ptYDn1+2wvzt7KBAAkzu+vk3USERHVJdocvyXtAcrPz0dcXBwCAwPVZQYGBggMDMTBgwfL1UZOTg4KCgpQv359AEBCQgKSk5M12rSxsUHnzp3LbDMvLw+ZmZkaD11zb2gBAPByqafzdRMREekbSROg9PR0KBQKODg4aJQ7ODggOTm5XG1MnToVTk5O6oRHtZw2bUZERMDGxkb9cHFx0XZTKi0ztxAA0N2jgc7XTUREpG+MylNp27Zt5W7w+eefr3Aw2po/fz42bNiA2NhYmJpWfBbV9OnTERoaqn6dmZmp8yTI0VoOALidla/T9RIREemjciVAAwcO1Hgtk8lQfOiQTPbwwn0KhaLcK7ezs4OhoSFSUlI0ylNSUuDo6PjYZSMjIzF//nz89ddfaN++vbpctVxKSgoaNXp4U9GUlBR4eXmV2pZcLodcLi933NUh6d4DAEAzOwtJ4yAiItIH5ToFplQq1Y8///wTXl5e+OOPP3Dv3j3cu3cP27dvR8eOHREdHa3Vyk1MTODj44OYmBiNdcXExKBr165lLrdw4ULMmTMH0dHR8PX11XivWbNmcHR01GgzMzMT//zzz2PblNr+S7cBAP8k3JE4EiIiorqvXD1AxU2cOBErVqxAjx491GVBQUEwNzfHG2+8gbNnz2rVXmhoKEaNGgVfX1/4+fkhKioK2dnZCAkJAQAEBwfD2dkZERERAIAFCxYgLCwM69evh6urq3pcj6WlJSwtLSGTyTBx4kR8/PHHaN68OZo1a4ZZs2bBycmpRE9WTSKTAUIAPk1tpQ6FiIioztM6Abp8+TLq1atXotzGxgaJiYlaBzB06FCkpaUhLCwMycnJ8PLyQnR0tHoQ87Vr12Bg8LCjavny5cjPz8fLL7+s0U54eDg+/PBDAMD777+P7OxsvPHGG7h37x569OiB6OjoSo0Tqm4WJkbIyiuEh72l1KEQERHVeVpfB6hXr14wNTXFd999p05SUlJSEBwcjNzcXPz999/VEqguSXEdoN6RsbiSno0Vr/rgmbaPH/9EREREJVXrdYC+/vpr3Lp1C02aNIGHhwc8PDzQpEkTJCUl4euvv65w0Pour1AJALCzNJE4EiIiorpP61NgzZs3x4kTJ7Bz506cO3cOANCqVSsEBgZqzAYj7ahmgeUrlBJHQkREVPdplQAVFBTAzMwMx44dQ9++fdG3b9/qiktvWcmNpQ6BiIioztPqFJixsTGaNGmi1bV+6MnuZD+8+KG1mdadckRERKQlrccAzZgxAx988AHu3OH1aqpKofLhaa8m9c0ljISIiEg/aN3dsHTpUly6dAlOTk5o2rQpLCw0r1wcHx9fZcHpi6z/7gNmZmzIcVREREQ6oHUCVJMvJlhbqQY+PyjgqUUiIiJd0DoBCg8Pr4449FpuQVEC5FzPTOJIiIiI9IPWY4Co6t3LKRoELTfmr4OIiEgXtO4BUigUWLx4MTZt2oRr164hPz9f430OjtZeoaLoYtypmXkSR0JERKQftO5ymD17NhYtWoShQ4ciIyMDoaGheOmll2BgYKC+Fxdp5+j1uwCArLxCiSMhIiLSD1onQOvWrcPKlSvx3nvvwcjICMOHD8eqVasQFhaGQ4cOVUeMdd6y3ZelDoGIiEivaJ0AJScno127dgAAS0tLZGRkAACee+45/P7771UbHREREVE10DoBaty4MW7dugUAcHd3x59//gkAOHLkCORyedVGR0RERFQNtE6AXnzxRcTExAAA3n77bcyaNQvNmzdHcHAwXnvttSoPkIiIiKiqaT0LbP78+ernQ4cORdOmTXHgwAE0b94cAwYMqNLgiIiIiKpDpe+82aVLF3Tp0qUqYtFbA72c8POxmwhsZS91KERERHpB6wSoSZMmCAgIgL+/PwICAuDu7l4dcemVfZduAwDvA0ZERKQjWo8BmjdvHkxNTbFgwQI0b94cLi4uePXVV7Fy5UpcvHixOmKs89Kzii6AeOLGPWkDISIi0hNa9wC9+uqrePXVVwEAt27dwt9//43ffvsNb731FpRKJRQK3tBTW3087RFzLhVje7E3jYiISBcqNAYoJycH+/btQ2xsLHbv3o2jR4+ibdu2CAgIqOLw9EPshTQAQG4hk0ciIiJd0DoB6tatG44ePYpWrVohICAA06ZNQ69evWBra1sd8ekFhbLoXmDnk+9LHAkREZF+0HoM0Llz52BhYQFPT094enqiVatWTH4qyd6q6AKSga0cJI6EiIhIP2idAN2+fRu7du1Cly5dsGPHDnTv3h3Ozs545ZVXsHLlyuqIsc7LK1QCACxNK31VAiIiIioHmRBCVHRhIQTi4uKwdOlSrFu3rs4Mgs7MzISNjQ0yMjJgbW1d7etznVZ0D7WZ/Vvh9Z5u1b4+IiKiukib47fWXQ7x8fGIjY1FbGws9u3bh/v376Ndu3Z4++234e/vX+GgCWjjZCN1CERERHpB6wTIz88P3t7e8Pf3x5gxY9CrVy/Y2PDAXRVszIylDoGIiEgvaJ0A3blzRyenhfRRZm6B1CEQERHpBa0HQVtbW+PevXtYtWoVpk+fjjt37gAoOjWWlJRU5QHWdcWHYLnZWUgYCRERkf7QugfoxIkT6NOnD+rVq4fExESMGTMG9evXx5YtW3Dt2jV8++231RFnnaWaAQYA5nLOAiMiItIFrXuAQkNDERISgosXL8LU1FRd/uyzz2LPnj1VGpw+eJD/cNacqZHWvw4iIiKqAK2PuEeOHMHYsWNLlDs7OyM5OVnrAJYtWwZXV1eYmpqic+fOOHz4cJl1T58+jUGDBsHV1RUymQxRUVEl6igUCsyaNQvNmjWDmZkZ3N3dMWfOHFRitn+1KlA87AEyMmQCREREpAtaH3HlcjkyMzNLlF+4cAENGzbUqq2NGzciNDQU4eHhiI+PR4cOHRAUFITU1NRS6+fk5MDNzQ3z58+Ho6NjqXUWLFiA5cuXY+nSpTh79iwWLFiAhQsXYsmSJVrFpiv5/yVAcvb+EBER6YzWR93nn38eH330EQoKimYsyWQyXLt2DVOnTsWgQYO0amvRokUYM2YMQkJC0Lp1a6xYsQLm5uZYvXp1qfU7deqETz75BMOGDYNcLi+1zoEDB/DCCy+gf//+cHV1xcsvv4y+ffs+tmdJSoWKop4pE/b+EBER6YzWR91PP/0UWVlZsLe3x4MHD+Dv7w8PDw9YWlpi7ty55W4nPz8fcXFxCAwMfBiMgQECAwNx8OBBbcNS69atG2JiYnDhwgUAwPHjx7Fv3z7069evzGXy8vKQmZmp8dAV1SkwY/YAERER6YzW045sbGywc+dO7Nu3DydOnEBWVhY6duyokciUR3p6OhQKBRwcNG8A6uDggHPnzmkbltq0adOQmZkJT09PGBoaQqFQYO7cuRgxYkSZy0RERGD27NkVXmdl3M8rBAAUFJsNRkRERNWrwvOue/TogR49eqhfx8fHIywsDL/99luVBFZRmzZtwrp167B+/Xq0adMGx44dw8SJE+Hk5IRRo0aVusz06dMRGhqqfp2ZmQkXFxedxKsam61KhIiIiKj6aZUA7dixAzt37oSJiQlef/11uLm54dy5c5g2bRp+/fVXBAUFlbstOzs7GBoaIiUlRaM8JSWlzAHO5TFlyhRMmzYNw4YNAwC0a9cOV69eRURERJkJkFwuL3NMUXXL/6/nx70hL4JIRESkK+UeePL111+jX79+WLt2LRYsWIAuXbrg+++/R9euXeHo6IhTp05h+/bt5V6xiYkJfHx8EBMToy5TKpWIiYlB165dtduKYnJycmBgoLlZhoaGUCpr5ikm1SwwEyNDiSMhIiLSH+XuAfrss8+wYMECTJkyBT/99BMGDx6ML774AidPnkTjxo0rtPLQ0FCMGjUKvr6+8PPzQ1RUFLKzsxESEgIACA4OhrOzMyIiIgAUDZw+c+aM+nlSUhKOHTsGS0tLeHh4AAAGDBiAuXPnokmTJmjTpg2OHj2KRYsW4bXXXqtQjNUt7X4eAMCEg6CJiIh0ptwJ0OXLlzF48GAAwEsvvQQjIyN88sknFU5+AGDo0KFIS0tDWFgYkpOT4eXlhejoaPXA6GvXrmn05ty8eRPe3t7q15GRkYiMjIS/vz9iY2MBAEuWLMGsWbPw1ltvITU1FU5OThg7dizCwsIqHGd1Ul3/53JqlsSREBER6Q+ZKOclkg0MDJCcnAx7e3sAgJWVFY4fPw43N7dqDVAKmZmZsLGxQUZGBqytrat1Xd8fuoqZP59CCwdL/DnJv1rXRUREVJdpc/zWahD0qlWrYGlpCQAoLCzE2rVrYWdnp1HnnXfe0TJc/fb9oasAgAsp7AEiIiLSlXL3AKnuv/XYxmQyXLlypUoCk5Iue4Dm/HYGX+9LQCMbUxyc3qda10VERFSXVUsPUGJiYmXjolLYmhsDAAJaancfNSIiIqo4Tj2S2JW0bACAnNPgiYiIdIYJkMQsTYs64dKz8iSOhIiISH8wAZKY8r8hWI1tzSWOhIiISH8wAZLYpn9vAACu382ROBIiIiL9wQRIYqp7gZ29lSlxJERERPqjQgnQ5cuXMXPmTAwfPhypqakAgD/++AOnT5+u0uD0gWr2V0j3ZhJHQkREpD+0ToD+/vtvtGvXDv/88w+2bNmCrKyiC/gdP34c4eHhVR6gvpDzXmBEREQ6o/VRd9q0afj444+xc+dOmJiYqMt79+6NQ4cOVWlw+uBUUtGpL8MnXGSSiIiIqo7WCdDJkyfx4osvlii3t7dHenp6lQSlT1TT349dvydtIERERHpE6wSoXr16uHXrVonyo0ePwtnZuUqC0kdd3RtIHQIREZHe0DoBGjZsGKZOnYrk5GTIZDIolUrs378fkydPRnBwcHXEWKd52BfdXNbW3OQJNYmIiKiqaJ0AzZs3D56ennBxcUFWVhZat26NXr16oVu3bpg5c2Z1xFinXUotGkQuN+YgaCIiIl0p981QVUxMTLBy5UrMmjULp06dQlZWFry9vdG8efPqiK/Os7OUIz0rj4OgiYiIdEjrBEilSZMmaNKkSVXGopdUg6B5CoyIiEh3tE6AQkNDSy2XyWQwNTWFh4cHXnjhBdSvX7/SwdV1hQql+jk7gIiIiHRH6wTo6NGjiI+Ph0KhQMuWLQEAFy5cgKGhITw9PfHFF1/gvffew759+9C6desqD7guySt8mADZWcoljISIiEi/aD3y9oUXXkBgYCBu3ryJuLg4xMXF4caNG3j66acxfPhwJCUloVevXpg0aVJ1xFun5BdLgEx4JWgiIiKdkQkhhDYLODs7Y+fOnSV6d06fPo2+ffsiKSkJ8fHx6Nu3b629MGJmZiZsbGyQkZEBa2vralvP+eT7CIraAwBInN+/2tZDRESkD7Q5fmvd7ZCRkaG+AWpxaWlpyMwsuq1DvXr1kJ+fr23TesfM2FDqEIiIiPRShU6Bvfbaa9i6dStu3LiBGzduYOvWrfjf//6HgQMHAgAOHz6MFi1aVHWsdU6BsugUmJVphSfjERERUQVofeT98ssvMWnSJAwbNgyFhYVFjRgZYdSoUVi8eDEAwNPTE6tWraraSOugQkXR2UcTQ47/ISIi0iWtEyBLS0usXLkSixcvxpUrVwAAbm5usLS0VNfx8vKqsgDrsoL/psEbGXIOPBERkS5V+NyLpaUl2rdvX5Wx6J2cfAUAICUzT+JIiIiI9EuFEqB///0XmzZtwrVr10oMdt6yZUuVBKYPzidnSh0CERGRXtJ68MmGDRvQrVs3nD17Flu3bkVBQQFOnz6NXbt2wcbGpjpirLNc7SwAAIYGPAVGRESkSxW6G/zixYvx66+/wsTEBJ999hnOnTuHIUOG8N5gWlJdCLGtU/Vda4iIiIhK0joBunz5Mvr3L7pon4mJCbKzsyGTyTBp0iR89dVXVR5gXZaZWwCAV4EmIiLSNa2PvLa2trh//z6AoqtCnzp1CgBw79495OTkVG10dVx2XtEg6Kz/fhIREZFuaJ0A9erVCzt37gQADB48GO+++y7GjBmD4cOHo0+fPloHsGzZMri6usLU1BSdO3fG4cOHy6x7+vRpDBo0CK6urpDJZIiKiiq1XlJSEl599VU0aNAAZmZmaNeuHf7991+tY6tuxv9Nf+cQICIiIt3SehbY0qVLkZubCwCYMWMGjI2NceDAAQwaNAgzZ87Uqq2NGzciNDQUK1asQOfOnREVFYWgoCCcP38e9vb2Jern5OTAzc0NgwcPLvNmq3fv3kX37t3x1FNP4Y8//kDDhg1x8eJF2Nraarup1U51N/imDcwljoSIiEi/aJUAFRYW4rfffkNQUBAAwMDAANOmTavwyhctWoQxY8YgJCQEALBixQr8/vvvWL16dantdurUCZ06dQKAMte7YMECuLi4YM2aNeqyZs2aVTjG6pSSWZRIyo14TzAiIiJd0uoUmJGREcaNG6fuAaqM/Px8xMXFITAw8GEwBgYIDAzEwYMHK9zutm3b4Ovri8GDB8Pe3h7e3t5YuXLlY5fJy8tDZmamxkMXlEV3wsD9/wZDExERkW5oPQbIz88Px44dq/SK09PToVAo4ODgoFHu4OCA5OTkCrd75coVLF++HM2bN8eOHTvw5ptv4p133sE333xT5jIRERGwsbFRP1xcXCq8fm2o7gFmzHuBERER6ZTWY4DeeusthIaG4vr16/Dx8YGFhYXG+1LfHkOpVMLX1xfz5s0DAHh7e+PUqVNYsWIFRo0aVeoy06dPR2hoqPp1ZmamTpKgwv/uBu9gbVrt6yIiIqKHtE6Ahg0bBgB455131GUymQxCCMhkMigU5ZvSbWdnB0NDQ6SkpGiUp6SkwNHRUduw1Bo1aoTWrVtrlLVq1Qo//fRTmcvI5XLI5fIKr7OiVHeDN+I0MCIiIp3SOgFKSEiokhWbmJjAx8cHMTExGDhwIICi3puYmBhMmDChwu12794d58+f1yi7cOECmjZtWplwq0WBKgHiKTAiIiKd0joBqspEIjQ0FKNGjYKvry/8/PwQFRWF7Oxs9ayw4OBgODs7IyIiAkDRwOkzZ86onyclJeHYsWOwtLSEh4cHAGDSpEno1q0b5s2bhyFDhuDw4cP46quvauRVqs/eKhpsfTuLd4MnIiLSpQp1PXz33Xfo3r07nJyccPXqVQBAVFQUfvnlF63aGTp0KCIjIxEWFgYvLy8cO3YM0dHR6oHR165dw61bt9T1b968CW9vb3h7e+PWrVuIjIyEt7c3Xn/9dXWdTp06YevWrfjhhx/Qtm1bzJkzB1FRURgxYkRFNrVaHbxyGwBw4PJtiSMhIiLSLzIhhNBmgeXLlyMsLAwTJ07E3LlzcerUKbi5uWHt2rX45ptvsHv37uqKVWcyMzNhY2ODjIwMWFtX341Kh355EP8k3MFYfzdM79eq2tZDRESkD7Q5fmvdA7RkyRKsXLkSM2bMgKHhwwv4+fr64uTJk9pHq8fs/5v95chZYERERDqldQKUkJAAb2/vEuVyuRzZ2dlVEpS++Oe/U2A5+bwZKhERkS5pnQA1a9as1AshRkdHo1UrnsbRRur9osHP8VfvShwJERGRftF6FlhoaCjGjx+P3NxcCCFw+PBh/PDDD4iIiMCqVauqI8Y6y95KjtT7eejTyuHJlYmIiKjKaJ0Avf766zAzM8PMmTORk5ODV155BU5OTvjss8/UF0mk8nG1s0Dq/TzYmBlLHQoREZFe0ToBAoARI0ZgxIgRyMnJQVZWFuzt7as6Lr1wMeU+AMCQV4ImIiLSKa3HAH388cfqq0Gbm5sz+amEuzlFd4E/eo1jgIiIiHRJ6wRo8+bN8PDwQLdu3fDFF18gPT29OuLSK51c60sdAhERkV7ROgE6fvw4Tpw4gYCAAERGRsLJyQn9+/fH+vXrkZOTUx0x1llNG5gDAOqZcwwQERGRLlXoVhht2rTBvHnzcOXKFezevRuurq6YOHFipe7iro/S/5sGb2LEm6ESERHpUqWPvBYWFjAzM4OJiQkKCgqqIia9kf3fBRAzHnC/ERER6VKFEqCEhATMnTsXbdq0ga+vL44ePYrZs2cjOTm5quPTCw2t5FKHQEREpFe0ngbfpUsXHDlyBO3bt0dISAiGDx8OZ2fn6oitzjMzNsSDAgUsTCp0NQIiIiKqIK2PvH369MHq1avRunVrjXKlUont27fjueeeq7Lg6rpCpRIAYGTI6wARERHpktYJ0Ny5czVeX7p0CatXr8batWuRlpbGcUBaKFAIALwQIhERka5VaAzQgwcP8O2336JXr15o2bIlDhw4gLCwMNy4caOq46uz8guV6udK5WMqEhERUZXTqgfoyJEjWLVqFTZs2AB3d3eMGDECBw4cwBdffFHilBg9Xl6hQv2c1wEiIiLSrXInQO3bt0dmZiZeeeUVHDhwAG3atAEATJs2rdqCq8tUp78AwNiQ1wEiIiLSpXIfec+fP49evXrhqaeeYm9PFVCdAjM0kHEMEBERkY6VOwG6cuUKWrZsiTfffBONGzfG5MmTcfToUchkPHhXhCoBMmHvDxERkc6V++jr7OyMGTNm4NKlS/juu++QnJyM7t27o7CwEGvXrsWFCxeqM846J/e/MUAPChRPqElERERVrULdD71798b333+PW7duYenSpdi1axc8PT3Rvn37qo6vzlIoi8YA8T5gREREulepo6+NjQ3eeust/Pvvv4iPj0dAQEAVhVX3FSiKToHZWZhIHAkREZH+qbLuBy8vL3z++edV1Vydp0qAMnMLJY6EiIhI//D8i0RizqYCALLymAARERHpGhMgidzJzpc6BCIiIr3FBEgiXd0bAADqcwwQERGRzjEBksiXf18BwJ4gIiIiKWh9N/iyBjrLZDKYmprCw8MDvXr1gqGhYaWDq8sa25rhzK1MqcMgIiLSS1onQIsXL0ZaWhpycnJga2sLALh79y7Mzc1haWmJ1NRUuLm5Yffu3XBxcanygOuKLm4N8OeZFAzo4CR1KERERHpH61Ng8+bNQ6dOnXDx4kXcvn0bt2/fxoULF9C5c2d89tlnuHbtGhwdHTFp0qTqiLfOyOOtMIiIiCSj9dF35syZWLx4Mdzd3dVlHh4eiIyMxPTp09G4cWMsXLgQ+/fvL3eby5Ytg6urK0xNTdG5c2ccPny4zLqnT5/GoEGD4OrqCplMhqioqMe2PX/+fMhkMkycOLHc8ejCpdQsAICxIe+lRkREpGtaJ0C3bt1CYWHJa9cUFhYiOTkZAODk5IT79++Xq72NGzciNDQU4eHhiI+PR4cOHRAUFITU1NRS6+fk5MDNzQ3z58+Ho6PjY9s+cuQIvvzyyxp5i45GNqYAgBt3H0gcCRERkf7ROgF66qmnMHbsWBw9elRddvToUbz55pvo3bs3AODkyZNo1qxZudpbtGgRxowZg5CQELRu3RorVqyAubk5Vq9eXWr9Tp064ZNPPsGwYcMgl8vLbDcrKwsjRozAypUr1WOVapLC/+4F1tLRSuJIiIiI9I/WCdDXX3+N+vXrw8fHB3K5HHK5HL6+vqhfvz6+/vprAIClpSU+/fTTJ7aVn5+PuLg4BAYGPgzIwACBgYE4ePCgtqFpGD9+PPr376/Rdlny8vKQmZmp8ahuCmXRGCAjA54CIyIi0jWtZ4E5Ojpi586dOHfuHC5cuAAAaNmyJVq2bKmu89RTT5WrrfT0dCgUCjg4OGiUOzg44Ny5c9qGprZhwwbEx8fjyJEj5aofERGB2bNnV3h9FaHqATJkAkRERKRzWidAKp6envD09KzKWKrE9evX8e6772Lnzp0wNTUt1zLTp09HaGio+nVmZma1T+E/nVTUy5SckVut6yEiIqKStE6AFAoF1q5di5iYGKSmpkL536kclV27dpW7LTs7OxgaGiIlJUWjPCUl5YkDnMsSFxeH1NRUdOzYUSPmPXv2YOnSpcjLyytxkUbVqTxdOpx4BwCw5WgSFg310um6iYiI9J3WCdC7776LtWvXon///mjbti1ksoqfwjExMYGPjw9iYmIwcOBAAIBSqURMTAwmTJhQoTb79OmDkydPapSFhITA09MTU6dOrTFXqO7hYYd9l9Ixzt/9yZWJiIioSmmdAG3YsAGbNm3Cs88+WyUBhIaGYtSoUfD19YWfnx+ioqKQnZ2NkJAQAEBwcDCcnZ0REREBoGjg9JkzZ9TPk5KScOzYMVhaWsLDwwNWVlZo27atxjosLCzQoEGDEuVSamBZdBNUO0veDJWIiEjXtE6ATExM4OHhUWUBDB06FGlpaQgLC0NycjK8vLwQHR2tHhh97do1GBg8nKx28+ZNeHt7q19HRkYiMjIS/v7+iI2NrbK4qluhomgQtDGvBE1ERKRzMiGE0GaBTz/9FFeuXMHSpUsrdfqrJsvMzISNjQ0yMjJgbW1dLesYsGQfTiZlYM4LbTCyq2u1rIOIiEifaHP81roHaN++fdi9ezf++OMPtGnTBsbGxhrvb9myRdsm9dLJpAwAwMZ/rzMBIiIi0jGtE6B69erhxRdfrI5Y9FJvT4cnVyIiIqIqpXUCtGbNmuqIQ+/4NrXFv1fvonUj3gqDiIhI1zgCVyIF/10J2siAvwIiIiJdK1cPUMeOHRETEwNbW1t4e3s/dvBzfHx8lQVXlx2/fg8AUPjIhSSJiIio+pUrAXrhhRfUV0pWXbCQqkbi7RypQyAiItI75UqAwsPDS31Olefb1FbqEIiIiPROhW+Gmp+fX+q9wJo0aVLpoPSBjZkxMh4UwMSIY4CIiIh0TesE6MKFC/jf//6HAwcOaJQLISCTyaBQKKosuLos40GBxk8iIiLSHa0ToJCQEBgZGeG3335Do0aN6uzVoHWlgYVu70JPREREFUiAjh07hri4OHh6elZHPHrDzNgQDwoUsDKt8FlIIiIiqiCtB6C0bt0a6enp1RGLXlFNfzcyZA8aERGRrmmdAC1YsADvv/8+YmNjcfv2bWRmZmo86MmEEChQ8EKIREREUtH6/EtgYCAAoE+fPhrlHARdfor/rgINAMbsASIiItI5rROg3bt3V0cceqWwWAJkZMgeICIiIl3TOgHy9/evjjj0Sr7i4bWTjAzYA0RERKRrFZqCdO/ePRw+fLjUCyEGBwdXSWB1WUHhw31mwh4gIiIindM6Afr1118xYsQIZGVlwdraWuM6QDKZjAlQOah6gIwMZDBgDxAREZHOad398N577+G1115DVlYW7t27h7t376ofd+7cqY4Y65yCwqIxQMXHAhEREZHuaJ0AJSUl4Z133oG5uXl1xKMXbtzlHeCJiIikpHUCFBQUhH///bc6YtEbpiaGUodARESk17QeA9S/f39MmTIFZ86cQbt27WBsbKzx/vPPP19lwdVVhv+Nm3KyMZU4EiIiIv2kdQI0ZswYAMBHH31U4j1eCLF8VGN/eA0gIiIiaWidAD067Z20V1hsFhgRERHpHrsgJKBQ9wAxASIiIpKC1j1ApZ36Ki4sLKzCwegL9Skw3giViIhIElonQFu3btV4XVBQgISEBBgZGcHd3Z0JUDnczckHACgFrwNEREQkBa0ToKNHj5Yoy8zMxOjRo/Hiiy9WSVB1nalx0TT463d4PSAiIiIpVMk5GGtra8yePRuzZs2qiubqvIL/BkG3dbaROBIiIiL9VGWDUDIyMpCRkVFVzdVpqgTIxIhjgIiIiKSg9Smwzz//XOO1EAK3bt3Cd999h379+lVZYHWZ6l5gxrwOEBERkSS0PgIvXrxY4/H5558jNjYWo0aNwpdfflmhIJYtWwZXV1eYmpqic+fOOHz4cJl1T58+jUGDBsHV1RUymQxRUVEl6kRERKBTp06wsrKCvb09Bg4ciPPnz1cotupw494DALwOEBERkVS07gFKSEgo870HDx5oHcDGjRsRGhqKFStWoHPnzoiKikJQUBDOnz8Pe3v7EvVzcnLg5uaGwYMHY9KkSaW2+ffff2P8+PHo1KkTCgsL8cEHH6Bv3744c+YMLCwstI6xqtmYFd0+JPF2tsSREBER6acqOQeTl5eHRYsWoVmzZlovu2jRIowZMwYhISFo3bo1VqxYAXNzc6xevbrU+p06dcInn3yCYcOGQS6Xl1onOjoao0ePRps2bdChQwesXbsW165dQ1xcnNbxVQfVlaDbOHEQNBERkRTKnQDl5eVh+vTp8PX1Rbdu3fDzzz8DAFavXo1mzZph8eLFZfbIlCU/Px9xcXEIDAx8GJCBAQIDA3Hw4EGt2noc1eDs+vXrl/p+Xl4eMjMzNR7VKb+wKAGScxA0ERGRJMp9BA4LC8Py5cvh6uqKxMREDB48GG+88QaioqKwaNEiJCYmYurUqVqtPD09HQqFAg4ODhrlDg4OSE5O1qqtsiiVSkycOBHdu3dH27ZtS60TEREBGxsb9cPFxaVK1l2WfM4CIyIiklS5xwBt3rwZ3377LZ5//nmcOnUK7du3R2FhIY4fPw6ZrOYO5h0/fjxOnTqFffv2lVln+vTpCA0NVb/OzMys1iToYkoWAMCQg6CJiIgkUe4E6MaNG/Dx8QEAtG3bFnK5HJMmTapU8mNnZwdDQ0OkpKRolKekpMDR0bHC7apMmDABv/32G/bs2YPGjRuXWU8ul5c5nqg6ONUzAwCkZObqbJ1ERET0ULnPwSgUCpiYmKhfGxkZwdLSslIrNzExgY+PD2JiYtRlSqUSMTEx6Nq1a4XbFUJgwoQJ2Lp1K3bt2lWhwdnVqVBZdArMo2Hl9h8RERFVTLl7gIQQGD16tLqnJDc3F+PGjSsxrXzLli1aBRAaGopRo0bB19cXfn5+iIqKQnZ2NkJCQgAAwcHBcHZ2RkREBICigdNnzpxRP09KSsKxY8dgaWkJDw8PAEWnvdavX49ffvkFVlZW6vFENjY2MDMz0yq+6qC6EjQvhEhERCSNcidAo0aN0nj96quvVkkAQ4cORVpaGsLCwpCcnAwvLy9ER0erB0Zfu3YNBgYPE4WbN2/C29tb/ToyMhKRkZHw9/dHbGwsAGD58uUAgICAAI11rVmzBqNHj66SuCujQFF0JWgjJkBERESSkAkhhNRB1DSZmZmwsbFBRkYGrK2tq7z9gE92I/F2Dsb2csP0Z1tVeftERET6SJvjN7sgJJB4OwcAEH/trsSREBER6ScmQBJo6WAFAHi2XSOJIyEiItJPTIAkYG9dNJBcdU8wIiIi0i0mQBJQ3QqDV4ImIiKSBo/AEjh67R4AwMiAu5+IiEgKPAJLoIVj0QUQVfcEIyIiIt1iAiSBwv+uA1Tf3OQJNYmIiKg6MAGSgOpK0EaGvBkqERGRFJgASeByWjYAwJgJEBERkSSYAEkoIT1H6hCIiIj0EhMgCakuiEhERES6xQRIAvUtigY/y425+4mIiKTAI7AECv8bBG1owDFAREREUmACJAGFsmgavBETICIiIkkwAZJA4X8JEHuAiIiIpMEESAIPe4C4+4mIiKTAI7COCSHYA0RERCQxJkA69l/uA4AJEBERkVSYAOlYQbEboMqNuPuJiIikwCOwjhW/A7yxIXc/ERGRFHgE1rGCwuIJEE+BERERSYEJkI6peoCMDWWQyZgAERERSYEJkI49yFcA4ABoIiIiKTEB0jGlKJoGllugfEJNIiIiqi5MgHSsQFGUANlZyiWOhIiISH8xAdKxwv8SIBMOgCYiIpIMEyAdUw2CNuIUeCIiIsnwKKxjheoEiD1AREREUmECpGN3svOlDoGIiEjvMQHSMZP/bn9x9XaOxJEQERHpLyZAOqa6E3z7xjYSR0JERKS/akQCtGzZMri6usLU1BSdO3fG4cOHy6x7+vRpDBo0CK6urpDJZIiKiqp0m7qkmgVmbFAjdj0REZFekvwovHHjRoSGhiI8PBzx8fHo0KEDgoKCkJqaWmr9nJwcuLm5Yf78+XB0dKySNnXp3oOiMUCnbmZIHAkREZH+kjwBWrRoEcaMGYOQkBC0bt0aK1asgLm5OVavXl1q/U6dOuGTTz7BsGHDIJeXfjFBbdvUpV+P3wQA5Px3SwwiIiLSPUkToPz8fMTFxSEwMFBdZmBggMDAQBw8eFBnbebl5SEzM1PjUV0CWzkAABrbmlXbOoiIiOjxJE2A0tPToVAo4ODgoFHu4OCA5ORknbUZEREBGxsb9cPFxaVC6y4Po/9ugtrBpV61rYOIiIgeT/JTYDXB9OnTkZGRoX5cv3692taluhK0Ca8ETUREJBkjKVduZ2cHQ0NDpKSkaJSnpKSUOcC5OtqUy+VljieqapdTswEwASIiIpKSpEdhExMT+Pj4ICYmRl2mVCoRExODrl271pg2q5KDjSkAIPF2tsSREBER6S9Je4AAIDQ0FKNGjYKvry/8/PwQFRWF7OxshISEAACCg4Ph7OyMiIgIAEWDnM+cOaN+npSUhGPHjsHS0hIeHh7lalNKqnuBtXayljgSIiIi/SV5AjR06FCkpaUhLCwMycnJ8PLyQnR0tHoQ87Vr12BQ7KKBN2/ehLe3t/p1ZGQkIiMj4e/vj9jY2HK1KSXVlaCNeQqMiIhIMjIhhJA6iJomMzMTNjY2yMjIgLV11fbUzP71NNbsT8RbAe54/xnPKm2biIhIn2lz/GY3hI6t2Z8IANh+8pa0gRAREekxJkA6Zm9VNNvMmRdCJCIikgwTIB3r1aIhAKCHR0OJIyEiItJfTIB0TDULzNhQJnEkRERE+osJkI4V/DcLzNCACRAREZFUmADpmKoHyIjT4ImIiCTDo7CO3crIBQAYsweIiIhIMkyAdKxAUXQKTHVTVCIiItI9JkA6ZmtuDIA3QyUiIpISj8I6VvhfD5CNmbHEkRAREekvJkA6ls9B0ERERJLjUVjHCpWqBIiDoImIiKTCBEjHTiVlAuAYICIiIinxKKxjLvWL7gHGWWBERETSYQIkEWtTDoImIiKSChMgHVPNAuO9wIiIiKTDBEjHVBdCNDLgriciIpIKj8I6ppoFxh4gIiIi6TAB0rH7uYUAeB0gIiIiKfEorGMKZdEpMN4LlYiISDpMgHRMlfjIjQylDYSIiEiPMQHSIYVS4L8OIJgYcdcTERFJhUdhHSoodvFD3gqDiIhIOkyAdKhQ1f0D3gqDiIhISjwK61BhsR4gYyZAREREkuFRWIfyCh8mQIacBkZERCQZJkA6VPwUGBEREUmHCZAOqU6BWZhwCjwREZGUmADpkGoWGK8CTUREJC0eiXWogHeCJyIiqhGYAOlQIe8ET0REVCPUiCPxsmXL4OrqClNTU3Tu3BmHDx9+bP3NmzfD09MTpqamaNeuHbZv367xflZWFiZMmIDGjRvDzMwMrVu3xooVK6pzE8rlfl4BAM0LIhIREZHuSZ4Abdy4EaGhoQgPD0d8fDw6dOiAoKAgpKamllr/wIEDGD58OP73v//h6NGjGDhwIAYOHIhTp06p64SGhiI6Ohrff/89zp49i4kTJ2LChAnYtm2brjarVIayolNft7PzJY2DiIhI30meAC1atAhjxoxBSEiIuqfG3Nwcq1evLrX+Z599hmeeeQZTpkxBq1atMGfOHHTs2BFLly5V1zlw4ABGjRqFgIAAuLq64o033kCHDh2e2LNU3VTT4Fs4WEoaBxERkb6TNAHKz89HXFwcAgMD1WUGBgYIDAzEwYMHS13m4MGDGvUBICgoSKN+t27dsG3bNiQlJUEIgd27d+PChQvo27dv9WxIOalngXEMEBERkaSMpFx5eno6FAoFHBwcNModHBxw7ty5UpdJTk4utX5ycrL69ZIlS/DGG2+gcePGMDIygoGBAVauXIlevXqV2mZeXh7y8vLUrzMzMyu6SY+lGgRtzDvBExERSapOHomXLFmCQ4cOYdu2bYiLi8Onn36K8ePH46+//iq1fkREBGxsbNQPFxeXaomrUFnUA2TM22AQERFJStIeIDs7OxgaGiIlJUWjPCUlBY6OjqUu4+jo+Nj6Dx48wAcffICtW7eif//+AID27dvj2LFjiIyMLHH6DACmT5+O0NBQ9evMzMxqSYJU1wEy4nWAiIiIJCVpD5CJiQl8fHwQExOjLlMqlYiJiUHXrl1LXaZr164a9QFg586d6voFBQUoKCiAwSPjbAwNDaFUlj79XC6Xw9raWuNRHWQywMzYEKbGvBUGERGRlCTtAQKKpqyPGjUKvr6+8PPzQ1RUFLKzsxESEgIACA4OhrOzMyIiIgAA7777Lvz9/fHpp5+if//+2LBhA/7991989dVXAABra2v4+/tjypQpMDMzQ9OmTfH333/j22+/xaJFiyTbTgB4rr0TnmvvJGkMREREVAMSoKFDhyItLQ1hYWFITk6Gl5cXoqOj1QOdr127ptGb061bN6xfvx4zZ87EBx98gObNm+Pnn39G27Zt1XU2bNiA6dOnY8SIEbhz5w6aNm2KuXPnYty4cTrfPiIiIqp5ZEIIIXUQNU1mZiZsbGyQkZFRbafDiIiIqGppc/yuk7PAiIiIiB6HCRARERHpHSZAREREpHeYABEREZHeYQJEREREeocJEBEREekdJkBERESkd5gAERERkd5hAkRERER6hwkQERER6R0mQERERKR3mAARERGR3pH8bvA1ker+sJmZmRJHQkREROWlOm6X5z7vTIBKcf/+fQCAi4uLxJEQERGRtu7fvw8bG5vH1pGJ8qRJekapVOLmzZuwsrKCTCar0rYzMzPh4uKC69evw9raukrbJk3c17rDfa073Ne6w32tO1W1r4UQuH//PpycnGBg8PhRPuwBKoWBgQEaN25creuwtrbmB0pHuK91h/tad7ivdYf7WneqYl8/qedHhYOgiYiISO8wASIiIiK9wwRIx+RyOcLDwyGXy6UOpc7jvtYd7mvd4b7WHe5r3ZFiX3MQNBEREekd9gARERGR3mECRERERHqHCRARERHpHSZAREREpHeYAFWDZcuWwdXVFaampujcuTMOHz782PqbN2+Gp6cnTE1N0a5dO2zfvl1HkdZ+2uzrlStXomfPnrC1tYWtrS0CAwOf+Luhh7T9u1bZsGEDZDIZBg4cWL0B1iHa7ut79+5h/PjxaNSoEeRyOVq0aMHvkXLSdl9HRUWhZcuWMDMzg4uLCyZNmoTc3FwdRVt77dmzBwMGDICTkxNkMhl+/vnnJy4TGxuLjh07Qi6Xw8PDA2vXrq3aoARVqQ0bNggTExOxevVqcfr0aTFmzBhRr149kZKSUmr9/fv3C0NDQ7Fw4UJx5swZMXPmTGFsbCxOnjyp48hrH2339SuvvCKWLVsmjh49Ks6ePStGjx4tbGxsxI0bN3Qcee2j7b5WSUhIEM7OzqJnz57ihRde0E2wtZy2+zovL0/4+vqKZ599Vuzbt08kJCSI2NhYcezYMR1HXvtou6/XrVsn5HK5WLdunUhISBA7duwQjRo1EpMmTdJx5LXP9u3bxYwZM8SWLVsEALF169bH1r9y5YowNzcXoaGh4syZM2LJkiXC0NBQREdHV1lMTICqmJ+fnxg/frz6tUKhEE5OTiIiIqLU+kOGDBH9+/fXKOvcubMYO3ZstcZZF2i7rx9VWFgorKysxDfffFNdIdYZFdnXhYWFolu3bmLVqlVi1KhRTIDKSdt9vXz5cuHm5iby8/N1FWKdoe2+Hj9+vOjdu7dGWWhoqOjevXu1xlnXlCcBev/990WbNm00yoYOHSqCgoKqLA6eAqtC+fn5iIuLQ2BgoLrMwMAAgYGBOHjwYKnLHDx4UKM+AAQFBZVZn4pUZF8/KicnBwUFBahfv351hVknVHRff/TRR7C3t8f//vc/XYRZJ1RkX2/btg1du3bF+PHj4eDggLZt22LevHlQKBS6CrtWqsi+7tatG+Li4tSnya5cuYLt27fj2Wef1UnM+kQXx0beDLUKpaenQ6FQwMHBQaPcwcEB586dK3WZ5OTkUusnJydXW5x1QUX29aOmTp0KJyenEh8y0lSRfb1v3z58/fXXOHbsmA4irDsqsq+vXLmCXbt2YcSIEdi+fTsuXbqEt956CwUFBQgPD9dF2LVSRfb1K6+8gvT0dPTo0QNCCBQWFmLcuHH44IMPdBGyXinr2JiZmYkHDx7AzMys0utgDxDppfnz52PDhg3YunUrTE1NpQ6nTrl//z5GjhyJlStXws7OTupw6jylUgl7e3t89dVX8PHxwdChQzFjxgysWLFC6tDqnNjYWMybNw9ffPEF4uPjsWXLFvz++++YM2eO1KFRBbAHqArZ2dnB0NAQKSkpGuUpKSlwdHQsdRlHR0et6lORiuxrlcjISMyfPx9//fUX2rdvX51h1gna7uvLly8jMTERAwYMUJcplUoAgJGREc6fPw93d/fqDbqWqsjfdaNGjWBsbAxDQ0N1WatWrZCcnIz8/HyYmJhUa8y1VUX29axZszBy5Ei8/vrrAIB27dohOzsbb7zxBmbMmAEDA/YpVJWyjo3W1tZV0vsDsAeoSpmYmMDHxwcxMTHqMqVSiZiYGHTt2rXUZbp27apRHwB27txZZn0qUpF9DQALFy7EnDlzEB0dDV9fX12EWutpu689PT1x8uRJHDt2TP14/vnn8dRTT+HYsWNwcXHRZfi1SkX+rrt3745Lly6pk0wAuHDhAho1asTk5zEqsq9zcnJKJDmqxFPwtppVSifHxiobTk1CiKJplXK5XKxdu1acOXNGvPHGG6JevXoiOTlZCCHEyJEjxbRp09T19+/fL4yMjERkZKQ4e/asCA8P5zT4ctJ2X8+fP1+YmJiIH3/8Udy6dUv9uH//vlSbUGtou68fxVlg5aftvr527ZqwsrISEyZMEOfPnxe//fabsLe3Fx9//LFUm1BraLuvw8PDhZWVlfjhhx/ElStXxJ9//inc3d3FkCFDpNqEWuP+/fvi6NGj4ujRowKAWLRokTh69Ki4evWqEEKIadOmiZEjR6rrq6bBT5kyRZw9e1YsW7aM0+BrgyVLlogmTZoIExMT4efnJw4dOqR+z9/fX4waNUqj/qZNm0SLFi2EiYmJaNOmjfj99991HHHtpc2+btq0qQBQ4hEeHq77wGshbf+ui2MCpB1t9/WBAwdE586dhVwuF25ubmLu3LmisLBQx1HXTtrs64KCAvHhhx8Kd3d3YWpqKlxcXMRbb70l7t69q/vAa5ndu3eX+v2r2r+jRo0S/v7+JZbx8vISJiYmws3NTaxZs6ZKY5IJwX47IiIi0i8cA0RERER6hwkQERER6R0mQERERKR3mAARERGR3mECRERERHqHCRARERHpHSZAREREpHeYABERVaHExETIZDIcO3ZM6lCIapw9e/ZgwIABcHJygkwmw88//6x1G0IIREZGokWLFpDL5XB2dsbcuXO1bocJEBFVSlpaGt588000adIEcrkcjo6OCAoKwv79+9V1KvpFVxGjR4+GTCbD/PnzNcp//vlnyGQyncRARKXLzs5Ghw4dsGzZsgq38e6772LVqlWIjIzEuXPnsG3bNvj5+WndDu8GT0SVMmjQIOTn5+Obb76Bm5sbUlJSEBMTg9u3b0sWk6mpKRYsWICxY8fC1tZWsjiqEu/sTnVBv3790K9fvzLfz8vLw4wZM/DDDz/g3r17aNu2LRYsWICAgAAAwNmzZ7F8+XKcOnUKLVu2BAA0a9asQrGwB4iIKuzevXvYu3cvFixYgKeeegpNmzaFn58fpk+fjueffx4A4OrqCgB48cUXIZPJ1K8B4JdffkHHjh1hamoKNzc3zJ49G4WFher3ZTIZli9fjn79+sHMzAxubm748ccfnxhXYGAgHB0dERERUWadDz/8EF5eXhplUVFRGvGNHj0aAwcOxLx58+Dg4IB69erho48+QmFhIaZMmYL69eujcePGWLNmTYn2z507h27dusHU1BRt27bF33//rfH+qVOn0K9fP1haWsLBwQEjR45Eenq6+v2AgABMmDABEydOhJ2dHYKCgp643US13YQJE3Dw4EFs2LABJ06cwODBg/HMM8/g4sWLAIBff/0Vbm5u+O2339CsWTO4urri9ddfx507d7ReFxMgIqowS0tLWFpa4ueff0ZeXl6pdY4cOQIAWLNmDW7duqV+vXfvXgQHB+Pdd9/FmTNn8OWXX2Lt2rUlzuXPmjULgwYNwvHjxzFixAgMGzYMZ8+efWxchoaGmDdvHpYsWYIbN25Uaht37dqFmzdvYs+ePVi0aBHCw8Px3HPPwdbWFv/88w/GjRuHsWPHlljPlClT8N577+Ho0aPo2rUrBgwYoO4Vu3fvHnr37g1vb2/8+++/iI6ORkpKCoYMGaLRxjfffAMTExPs378fK1asqNR2ENV0165dw5o1a7B582b07NkT7u7umDx5Mnr06KH+J+PKlSu4evUqNm/ejG+//RZr165FXFwcXn75Ze1XWKW3ViUivfPjjz8KW1tbYWpqKrp16yamT58ujh8/rlEHgNi6datGWZ8+fcS8efM0yr777jvRqFEjjeXGjRunUadz587izTffLDOe4nee79Kli3jttdeEEEJs3bpVFP/KCw8PFx06dNBYdvHixaJp06YabTVt2lQoFAp1WcuWLUXPnj3VrwsLC4WFhYX44YcfhBBCJCQkCABi/vz56joFBQWicePGYsGCBUIIIebMmSP69u2rse7r168LAOL8+fNCiKI7kXt7e5e5nUS13aPfC7/99psAICwsLDQeRkZGYsiQIUIIIcaMGaPxORFCiLi4OAFAnDt3Tqv1cwwQEVXKoEGD0L9/f+zduxeHDh3CH3/8gYULF2LVqlUYPXp0mcsdP34c+/fv1+jxUSgUyM3NRU5ODszNzQEAXbt21Viua9eu5Z5htWDBAvTu3RuTJ0/WertU2rRpAwODh53lDg4OaNu2rfq1oaEhGjRogNTU1BJxqhgZGcHX11fdc3X8+HHs3r0blpaWJdZ3+fJltGjRAgDg4+NT4biJapusrCwYGhoiLi4OhoaGGu+pPiuNGjWCkZGR+jMCAK1atQJQ1IOkGhdUHkyAiKjSTE1N8fTTT+Ppp5/GrFmz8PrrryM8PPyxCVBWVhZmz56Nl156qdT2qkKvXr0QFBSE6dOnl4jFwMAARf+EPlRQUFCiDWNjY43XMpms1DKlUlnuuLKysjBgwAAsWLCgxHuNGjVSP7ewsCh3m0S1nbe3NxQKBVJTU9GzZ89S63Tv3h2FhYW4fPky3N3dAQAXLlwAADRt2lSr9TEBIqIq17p1a41p78bGxlAoFBp1OnbsiPPnz8PDw+OxbR06dAjBwcEar729vcsdy/z58+Hl5VXiP8OGDRsiOTkZQgj19PiqvHbPoUOH0KtXLwBAYWEh4uLiMGHCBABF2/7TTz/B1dUVRkb8Gib9kZWVhUuXLqlfJyQk4NixY6hfvz5atGiBESNGIDg4GJ9++im8vb2RlpaGmJgYtG/fHv3790dgYCA6duyI1157DVFRUVAqlRg/fjyefvppjV6h8uAgaCKqsNu3b6N37974/vvvceLECSQkJGDz5s1YuHAhXnjhBXU9V1dXxMTEIDk5GXfv3gUAhIWF4dtvv8Xs2bNx+vRpnD17Fhs2bMDMmTM11rF582asXr0aFy5cQHh4OA4fPqxOJMqjXbt2GDFiBD7//HON8oCAAKSlpWHhwoW4fPkyli1bhj/++KMSe0PTsmXLsHXrVpw7dw7jx4/H3bt38dprrwEAxo8fjzt37mD48OE4cuQILl++jB07diAkJKREokhUl/z777/w9vZW/xMTGhoKb29vhIWFASiaLBEcHIz33nsPLVu2xMCBA3HkyBE0adIEQFHP7a+//go7Ozv06tUL/fv3R6tWrbBhwwbtg6mCcUxEpKdyc3PFtGnTRMeOHYWNjY0wNzcXLVu2FDNnzhQ5OTnqetu2bRMeHh7CyMhIY5BxdHS06NatmzAzMxPW1tbCz89PfPXVV+r3AYhly5aJp59+WsjlcuHq6io2btz42JiKD4JWSUhIECYmJuLRr7zly5cLFxcXYWFhIYKDg8XcuXNLDIJ+tC1/f3/x7rvvapQ1bdpULF68WL0uAGL9+vXCz89PmJiYiNatW4tdu3ZpLHPhwgXx4osvinr16gkzMzPh6ekpJk6cKJRKZZnrIaKqIxPikZPgREQ1hEwmw9atWzFw4ECpQyGiOoanwIiIiEjvMAEiIiIivcPpB0RUY/EMPRFVF/YAERERkd5hAkRERER6hwkQERER6R0mQERERKR3mAARERGR3mECRERERHqHCRARERHpHSZAREREpHeYABEREZHe+T9YAVI7oZND8wAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Plotting running average reward for BanditTwoArmedHighLowFixed-v0\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACcH0lEQVR4nO3deVwU5R8H8M8CsggICigIoggeeOCF4q2oJN6h5pV3ZlqaJWppJqblmalllqWYZv3SNC2PRA3vE8UTD1QUb0BQQFG5dn5/4C47uzO7M7uzB/B9v168hNnZ2WdXmPnO83yf7yNjGIYBIYQQQkgZYmPpBhBCCCGEmBsFQIQQQggpcygAIoQQQkiZQwEQIYQQQsocCoAIIYQQUuZQAEQIIYSQMocCIEIIIYSUORQAEUIIIaTMoQCIEEIIIWUOBUCESEgmk+GLL76wdDNIGXbw4EHIZDIcPHjQ0k2xmHXr1kEmkyE5OdnSTSFWjAIgYvWUJzPll52dHXx8fDBq1Cg8ePDA0s2zOlevXoVMJoODgwMyMzMt3RyrMmrUKNbvklwuR506dRAVFYVXr15ZunkWcfnyZQwbNgw+Pj6Qy+Xw9vbG0KFDcfnyZUs3jSU0NJT1f8f3RTcgRCg7SzeAEKHmzp2LmjVr4tWrVzh58iTWrVuHo0ePIiEhAQ4ODpZuHgDg5cuXsLOz7J/Vb7/9Bi8vLzx9+hRbtmzBu+++a9H2WBu5XI41a9YAALKysvDPP//gyy+/RFJSEn7//XcLt868tm7diiFDhsDNzQ1jxoxBzZo1kZycjOjoaGzZsgUbN25E3759Ld1MAMDMmTNZv8unT5/Gd999h88++wz16tVTbW/UqBEaNGiAwYMHQy6XW6KppKRgCLFyv/zyCwOAOX36NGv7p59+ygBgNm3aZKGWWR+FQsH4+fkxkZGRTN++fZnQ0FCLtCMnJ8cir6vPyJEjGScnJ9Y2hULBtGrVipHJZExKSoqFWiacQqFgXrx4wfv4gQMHGADMgQMHdB7n5s2bjKOjIxMYGMikpaWxHnv8+DETGBjIODk5MUlJSVI0W7Dnz58L2m/z5s2C3ichfGgIjJRY7du3BwAkJSWptoWGhiI0NFRr31GjRsHPz0/1c3JyMmQyGZYsWYKff/4ZAQEBkMvlaNGiBU6fPq31XGdnZzx48AARERFwdnZG5cqVMXXqVBQWFrL21eyC/+KLLyCTyXDz5k2MGjUKFStWhKurK0aPHo0XL16wnvvy5UtMmjQJHh4eqFChAvr06YMHDx6I6tY/duwYkpOTMXjwYAwePBiHDx/G/fv3VY/36tUL/v7+nM9t3bo1mjdvztr222+/ITg4GOXLl4ebmxsGDx6Me/fusfYJDQ1Fw4YNER8fjw4dOsDR0RGfffYZAOCff/5Bz5494e3tDblcjoCAAHz55ZdanxsArFy5Ev7+/ihfvjxCQkJw5MgRzv/P3NxczJ49G7Vq1YJcLoevry8++eQT5ObmCvqMNMlkMrRr1w4Mw+DWrVusx3bv3o327dvDyckJFSpUQM+ePVlDQ9u3b4dMJsPFixdV2/766y/IZDL069ePdax69eph0KBBqp9/+eUXdO7cGVWqVIFcLkf9+vXx448/arXPz88PvXr1wp49e9C8eXOUL18eP/30EwDg/v37iIiIgJOTE6pUqYLJkycL/hy+/vprvHjxAj///DMqV67MeszDwwM//fQTcnJysHjxYgDAli1bIJPJcOjQIa1j/fTTT5DJZEhISFBtu3btGt566y24ubnBwcEBzZs3x/bt21nPUw5vHzp0CB988AGqVKmCatWqCWq/Llw5QMrP8eDBg6rPMSgoSJUrtXXrVgQFBcHBwQHBwcE4d+6c1nGFvCdSclAAREos5cmtUqVKBh/jf//7H77++muMGzcOX331FZKTk9GvXz/k5+ez9issLER4eDjc3d2xZMkSdOzYEd988w1+/vlnQa8zcOBAPHv2DAsWLMDAgQOxbt06zJkzh7XPqFGjsGLFCvTo0QOLFi1C+fLl0bNnT1Hv5/fff0dAQABatGiB3r17w9HREX/88Yfq8UGDBuH27dtaQd6dO3dw8uRJDB48WLVt3rx5GDFiBGrXro2lS5fi448/RmxsLDp06KCVW5SRkYHu3bujSZMmWL58OTp16gSg6ELk7OyMyMhIfPvttwgODkZUVBSmT5/Oev6PP/6IiRMnolq1ali8eDHat2+PiIgIVvAGAAqFAn369MGSJUvQu3dvrFixAhEREVi2bBkruBCL63dpw4YN6NmzJ5ydnbFo0SLMmjULV65cQbt27VT7t2vXDjKZDIcPH1Y978iRI7CxscHRo0dV2x4/foxr166hQ4cOrPdco0YNfPbZZ/jmm2/g6+uLDz74ACtXrtRqX2JiIoYMGYI33ngD3377LZo0aYKXL1+iS5cu2LNnDyZOnIiZM2fiyJEj+OSTTwS95x07dsDPz091I6GpQ4cO8PPzw65duwBA9Vn8+eefWvtu2rQJDRo0QMOGDQEU5RW1atUKV69exfTp0/HNN9/AyckJERER2LZtm9bzP/jgA1y5coXzd0NKN2/exNtvv43evXtjwYIFePr0KXr37o3ff/8dkydPxrBhwzBnzhwkJSVh4MCBUCgUqueKfU+kBLB0FxQh+iiHwP777z/m8ePHzL1795gtW7YwlStXZuRyOXPv3j3Vvh07dmQ6duyodYyRI0cyNWrUUP18+/ZtBgDj7u7OPHnyRLX9n3/+YQAwO3bsYD0XADN37lzWMZs2bcoEBweztgFgZs+erfp59uzZDADmnXfeYe3Xt29fxt3dXfVzfHw8A4D5+OOPWfuNGjVK65h88vLyGHd3d2bmzJmqbW+//TbTuHFj1c9ZWVmMXC5npkyZwnru4sWLGZlMxty5c4dhGIZJTk5mbG1tmXnz5rH2u3TpEmNnZ8fa3rFjRwYAs2rVKq02cQ3VjBs3jnF0dGRevXrFMAzD5ObmMu7u7kyLFi2Y/Px81X7r1q1jALD+Pzds2MDY2NgwR44cYR1z1apVDADm2LFjfB8PwzDFQ2CPHz9mHj9+zNy8eZNZsmQJI5PJmIYNGzIKhYJhGIZ59uwZU7FiRWbs2LGs56ekpDCurq6s7Q0aNGAGDhyo+rlZs2bMgAEDGADM1atXGYZhmK1btzIAmAsXLuj8bMLDwxl/f3/Wtho1ajAAmJiYGNb25cuXMwCYP//8U7UtJyeHqVWrlt6hoczMTAYA8+abb/LuwzAM06dPHwYAk52dzTAMwwwZMoSpUqUKU1BQoNrn0aNHjI2NDevvo0uXLkxQUJDq/5hhiobu2rRpw9SuXVu1Tfm33a5dO9YxhdA1BKY87u3bt1XblJ/j8ePHVdv27NnDAGDKly+v+t1nGIb56aeftI4t9D2RkoN6gEiJERYWhsqVK8PX1xdvvfUWnJycsH37dqO6zAcNGsS661feDWsOhQDA+PHjWT+3b9+ecz8uXM/NyMhAdnY2ACAmJgZA0Z2wug8//FDQ8YGi4ZqMjAwMGTJEtW3IkCG4cOGCatjGxcUF3bt3x59//gmGYVT7bdq0Ca1atUL16tUBFA0HKBQKDBw4EOnp6aovLy8v1K5dGwcOHGC9tlwux+jRo7XaVL58edX3z549Q3p6Otq3b48XL17g2rVrAIAzZ84gIyMDY8eOZSWQDx06VKt3b/PmzahXrx4CAwNZ7ercuTMAaLWLS05ODipXrozKlSujVq1amDp1Ktq2bYt//vkHMpkMALBv3z5kZmZiyJAhrNextbVFy5YtWa/Tvn17HDlyRPUeL1y4gPfeew8eHh6q7UeOHEHFihVVPSSan01WVhbS09PRsWNH3Lp1C1lZWaw216xZE+Hh4axt//77L6pWrYq33npLtc3R0RHvvfee3s/g2bNnAIAKFSro3E/5uPL3dNCgQUhLS2NNsd+yZQsUCoWqB+7JkyfYv3+/qtdT+dllZGQgPDwcN27c0Jq9OXbsWNja2uptt7Hq16+P1q1bq35u2bIlAKBz586q33317cq/b0PeE7F+NAuMlBgrV65EnTp1kJWVhbVr1+Lw4cNGz/JQP+kBxUMgT58+ZW13cHDQypOoVKmS1n6GvI6Liwvu3LkDGxsb1KxZk7VfrVq1BB0fKMrXqVmzJuRyOW7evAkACAgIgKOjI37//XfMnz8fQNFF7O+//8aJEyfQpk0bJCUlIT4+HsuXL1cd68aNG2AYBrVr1+Z8rXLlyrF+9vHxgb29vdZ+ly9fxueff479+/erLqJKyov8nTt3ON+rnZ0dK29L2a6rV69q/V8opaWlcW5X5+DggB07dgAoyqFZvHgx0tLSWAHJjRs3AEAVWGlycXFRfd++fXusWrUKN2/eRFJSEmQyGVq3bq0KjMaOHYsjR46gbdu2sLEpvuc8duwYZs+ejRMnTmjlg2VlZcHV1VX1s+bvBVD0udWqVUsVtCnVrVtX72egDGyUgRAfzUCpW7ducHV1xaZNm9ClSxcARcFzkyZNUKdOHQBFw0wMw2DWrFmYNWsW53HT0tLg4+Oj8/2ZgubfofIz9vX15dyu/Ps25D0R60cBECkxQkJCVEm6ERERaNeuHd5++20kJibC2dkZQFFCq3rPhhJX0i0A3rtOzWMYe3cq9HUMlZ2djR07duDVq1ecQcv//vc/zJs3DzKZTJUb9Oeff6JNmzb4888/YWNjgwEDBqj2VygUkMlk2L17N2fblZ+3knrwoJSZmYmOHTvCxcUFc+fORUBAABwcHHD27Fl8+umnrPwKoRQKBYKCgrB06VLOxzUvZFxsbW0RFham+jk8PByBgYEYN26cKqFV2bYNGzbAy8tL6xjqPVXt2rUDABw+fBi3bt1Cs2bN4OTkhPbt2+O7777D8+fPce7cOcybN0/1nKSkJHTp0gWBgYFYunQpfH19YW9vj3///RfLli3T+my4Pl9juLq6omrVqqzkbS4XL16Ej4+PKuCTy+WqnJcffvgBqampOHbsmCq4Boo/u6lTp2r1WilpBrtSvz8+fH+H+v4+DXlPxPpRAERKJFtbWyxYsACdOnXC999/r0qcrFSpEuewlLKXwVrVqFEDCoUCt2/fZgUwyp4cfbZu3YpXr17hxx9/hIeHB+uxxMREfP755zh27BjatWsHJycn9OrVC5s3b8bSpUuxadMmtG/fHt7e3qrnBAQEgGEY1KxZU3VnL9bBgweRkZGBrVu3spJ/b9++zdqvRo0aqveqTJ4GgIKCAiQnJ6NRo0asdl24cAFdunTR6vkwVNWqVTF58mTMmTMHJ0+eRKtWrRAQEAAAqFKlCitY4lK9enVUr14dR44cwa1bt1TDqB06dEBkZCQ2b96MwsJC1mewY8cO5ObmYvv27axeCSFDeEo1atRAQkICGIZhfRaJiYmCnt+rVy+sXr0aR48eVQVx6o4cOYLk5GSMGzeOtX3QoEFYv349YmNjcfXqVTAMw0pAV84yLFeunN7PrqQoje+J0CwwUoKFhoYiJCQEy5cvV1XxDQgIwLVr1/D48WPVfhcuXMCxY8cs1UxBlHeVP/zwA2v7ihUrBD3/t99+g7+/P8aPH4+33nqL9TV16lQ4OzuzivwNGjQIDx8+xJo1a3DhwgWtGVT9+vWDra0t5syZo9VLxTAMMjIy9LZJeVet/vy8vDyt99i8eXO4u7tj9erVKCgoUG3//ffftYYYBw4ciAcPHmD16tVar/fy5Uvk5OTobReXDz/8EI6Ojli4cCGAov8PFxcXzJ8/X2tGIADW7xdQNAy2f/9+xMXFqQKgJk2aoEKFCli4cCHKly+P4OBg1f5cn01WVhZ++eUXwW3u0aMHHj58iC1btqi2Kae1CzFt2jSUL18e48aN0/r/fPLkCcaPHw9HR0dMmzaN9VhYWBjc3NywadMmbNq0CSEhIawhrCpVqiA0NBQ//fQTHj16pPW6mp9dSVAa3xOhHiBSwk2bNg0DBgzAunXrMH78eLzzzjtYunQpwsPDMWbMGKSlpWHVqlVo0KCBVg6KNQkODkb//v2xfPlyZGRkoFWrVjh06BCuX78OADp7Ox4+fIgDBw5g0qRJnI/L5XKEh4dj8+bN+O6771CuXDn06NEDFSpUwNSpU2Fra4v+/fuznhMQEICvvvoKM2bMQHJyMiIiIlChQgXcvn0b27Ztw3vvvYepU6fqfE9t2rRBpUqVMHLkSEyaNAkymQwbNmzQCqjs7e3xxRdf4MMPP0Tnzp0xcOBAJCcnY926dQgICGC99+HDh+PPP//E+PHjceDAAbRt2xaFhYW4du0a/vzzT1WtHLHc3d0xevRo/PDDD7h69Srq1auHH3/8EcOHD0ezZs0wePBgVK5cGXfv3sWuXbvQtm1bfP/996rnt2/fHr///ruqphBQFOS0adMGe/bsQWhoKCtHqmvXrrC3t0fv3r0xbtw4PH/+HKtXr0aVKlU4L7Bcxo4di++//x4jRoxAfHw8qlatig0bNsDR0VHQ82vXro3169dj6NChCAoK0qoEnZ6ejj/++EPVG6ZUrlw59OvXDxs3bkROTg6WLFmideyVK1eiXbt2CAoKwtixY+Hv74/U1FScOHEC9+/fx4ULFwS10ZqUxvdU5pl/4hkh4vBVgmYYhiksLGQCAgKYgIAA1TTa3377jfH392fs7e2ZJk2aMHv27OGdBv/1119rHRMa0865qgczTPEUd13PVe7z+PFjzvekPk03JyeHmTBhAuPm5sY4OzszERERTGJiIgOAWbhwIe/n88033zAAmNjYWN59lFPK//nnH9W2oUOHMgCYsLAw3uf99ddfTLt27RgnJyfGycmJCQwMZCZMmMAkJiaq9unYsSPToEEDzucfO3aMadWqFVO+fHnG29ub+eSTT1RTjzWnL3/33XdMjRo1GLlczoSEhDDHjh1jgoODmW7durH2y8vLYxYtWsQ0aNCAkcvlTKVKlZjg4GBmzpw5TFZWFu97YRj+/0uGYZikpCTG1taWGTlypGrbgQMHmPDwcMbV1ZVxcHBgAgICmFGjRjFnzpxhPffy5csMAKZevXqs7V999RUDgJk1a5bW623fvp1p1KgR4+DgwPj5+TGLFi1i1q5dyzl9u2fPnpxtvnPnDtOnTx/G0dGR8fDwYD766CMmJiZGVIXkixcvMkOGDGGqVq3KlCtXjvHy8mKGDBnCXLp0ifc5+/btYwAwMpmMVYZCXVJSEjNixAjGy8uLKVeuHOPj48P06tWL2bJli2ofXX/b+hgyDZ7rcwTATJgwgbWN7/wg5D2RkkPGMBJlYRJCJHf+/Hk0bdoUv/32G4YOHWrp5piVQqFA5cqV0a9fP84hL0IIMQblABFiJV6+fKm1bfny5bCxsWEl0JZGr1690hoa+/XXX/HkyRPOpU0IIcRYlANEiJVYvHgx4uPj0alTJ9jZ2WH37t3YvXs33nvvPUHTu0uykydPYvLkyRgwYADc3d1x9uxZREdHo2HDhqzp+YQQIhUaAiPESuzbtw9z5szBlStX8Pz5c1SvXh3Dhw/HzJkzWXVnSqPk5GRMmjQJcXFxePLkCdzc3NCjRw8sXLgQVapUsXTzCCGlEAVAhBBCCClzKAeIEEIIIWUOBUCEEEIIKXNKd2KBgRQKBR4+fIgKFSpIVm6fEEIIIabFMAyePXsGb29v1uLDXCgA4vDw4cNSP+uGEEIIKa3u3buHatWq6dyHAiAOFSpUAFD0ASpXQSaEEEKIdcvOzoavr6/qOq4LBUAclMNeLi4uFAARQgghJYyQ9BVKgiaEEEJImUMBECGEEELKHAqACCGEEFLmWDwAWrlyJfz8/ODg4ICWLVsiLi6Od9/Q0FDIZDKtr549e6r2ef78OSZOnIhq1aqhfPnyqF+/PlatWmWOt0IIIYSQEsKiAdCmTZsQGRmJ2bNn4+zZs2jcuDHCw8ORlpbGuf/WrVvx6NEj1VdCQgJsbW1ZiyVGRkYiJiYGv/32G65evYqPP/4YEydOxPbt2831tgghhBBi5SwaAC1duhRjx47F6NGjVT01jo6OWLt2Lef+bm5u8PLyUn3t27cPjo6OrADo+PHjGDlyJEJDQ+Hn54f33nsPjRs31tmzRAghhJCyxWIBUF5eHuLj4xEWFlbcGBsbhIWF4cSJE4KOER0djcGDB8PJyUm1rU2bNti+fTsePHgAhmFw4MABXL9+HV27duU9Tm5uLrKzs1lfhBBCCCm9LBYApaeno7CwEJ6enqztnp6eSElJ0fv8uLg4JCQk4N1332VtX7FiBerXr49q1arB3t4e3bp1w8qVK9GhQwfeYy1YsACurq6qL6oCTQghhJRuFk+CNlR0dDSCgoIQEhLC2r5ixQqcPHkS27dvR3x8PL755htMmDAB//33H++xZsyYgaysLNXXvXv3TN18QgghhFiQxSpBe3h4wNbWFqmpqaztqamp8PLy0vncnJwcbNy4EXPnzmVtf/nyJT777DNs27ZNNTOsUaNGOH/+PJYsWcIablMnl8shl8uNeDeEEEIIKUks1gNkb2+P4OBgxMbGqrYpFArExsaidevWOp+7efNm5ObmYtiwYazt+fn5yM/P11oB1tbWFgqFQrrGE0IIIaREs+haYJGRkRg5ciSaN2+OkJAQLF++HDk5ORg9ejQAYMSIEfDx8cGCBQtYz4uOjkZERATc3d1Z211cXNCxY0dMmzYN5cuXR40aNXDo0CH8+uuvWLp0qdneFyGEEEKsm0UDoEGDBuHx48eIiopCSkoKmjRpgpiYGFVi9N27d7V6cxITE3H06FHs3buX85gbN27EjBkzMHToUDx58gQ1atTAvHnzMH78eJO/H1MrVDAoUCggt7O1dFMIIYSQEk3GMAxj6UZYm+zsbLi6uiIrK8uqVoMf/PMJ3E7PwaFpneBQjoIgQgghRJ2Y63eJnQVWFp289QSp2bmIv/PU0k0hhBBCSjQKgAghhBBS5lAAVALJLN0AQgghpISjAKgEkskoBCKEEEKMQQFQCaGeq07xDyGEEGIcCoBKiOyXBarvFTRxjxBCCDEKBUBW7nluARQKBo+fv1JtK1RQAEQIIYQYw6KFEIluDzNfos3C/QCATe+1Um0vKKQAiBBCCDEG9QBZsX/OP1R9P3v7ZdX3/4u7a4nmlBlZL/Nx5MZj6mmzcgNWHcd7v56xdDMIISUU9QBZMRu1ZOdrKc9U32e9yLdAa8qOxnOKlln5vGc9vNve38KtIVzOJD/B6eSigqAPM1/Cu2J5C7eIEFLSUA+QFePrf3hVUGjWdpRVX+26aukmEB4nb2Wovs8rUFiwJaQsiklIwe+n7li6GcRI1ANkxRbFXOPcXp7WASNlXL5aHtyzVwU69iREeuN/iwcAtAnwQE0PJwu3hhiKeoCsGN9s94dZL83bEEKsTNbL4mHgxNRnOvYkpnL8ZjoelfFz0ZOcXEs3gRiBeoCsmL2dDWf3/r0nZfukY0pPcvIs3QQiwLrjyarvg3xcLdeQMurojXQMiz4FAEhe2NPCrbEkqkpbklEPkBUzNLchv1BBidIGephJwWVJY0tnMbPbdyXF0k0gxGh06iiFui47jMZz9yIt+5X+nQkp4QopB9rs1p8ouwnA6ssSpWTRObYkowCoFLqdngMAOJCYZuGWlDxU+6fkof8z83r8rGznvbzML56F++eZexZsCTEWBUCl2C/Hki3dhBJH/eRGSgZaG8+8cnLZs+7K2nRw9dSEV3S+KNEoACrF1IsnClGoYLB033UcT0o3UYusX6LaZ1bdzdGCLSFCUQ+Qef13NZX188xtCRZqiWXIZMWJz05ymkdUklEARFS2nr2P72Jv4O3VpyzdFIup7ems+r6CA53cSoI3Vx6zdBPKlOtlvOyAeg5QPiWglWgUAJUyxlTFvf+UZkB9tbO4+vPlh9kme50bqc9QPyoGy/+7brLXKI2SHj/HkRuPLd2MMu3PM/dNevz8QgXGrDuNFbE3TPo6htp69oHq+6cvistmDP75BPym76IeyRKEAqBSJvOF4XVs7GyopsWVR6YLetQNj47Di7xCLP/vBt5/XVWW6Nflm0MYHh1n6WYQDZ2WHMSLPGkqcv928g5ir6Xhm33XUWCFPSz3nr5Qfe+jtgbdyVtPAACtFsSavU3EMBQAlTKHrht+d2ynVlDlAdXDAQA8e2WaekopaiUKdidQTRVSst1Oz0G/H45Lcqw5O66ovo+7/USSY0rJXu08ydXbU9ZnyZUkFACVMtO2XDT4ueo9QGPWnZaiOSXeubuZlm4CISWC2EkXljqmsX46fEv1vfqadKTkoQCIqNiqBUDWeOIxB81hwJI6yyPp8XMcuMauA+U3fRf8pu+iqbvEIH/F30f/H6Xp5dGlWfWKqu+zTdQDKxVlziVDpRhKJAqAiIox+UOlRYFGl7ZmzZOSoss3hzB63WmcvftU67F315+xQItMK/05DTuY2pTNFxB/R/v3SWrl7W1V3zuUs9Wxp+UpZ4FR/FMyUQBEVL7bf9PSTbA4zanvI9aaJ+FWyoqyCrUg7sK9TK3Hj94smXWeaMqxeOnPcyUJ4s05syn9WfGNWPkSEgBpFuOkHqGSgQIgK6Y+w4CYR69GVS3yup8Ykbul6YXaEFdqdunpGcl+yT8cMm5DPC7dzzJja6zflYfZaP7Vfwj6Yo/Rx9p+4YH+nSDNhT9Rrc6QQznrvkTlvc4B0owPd158ZIHWELGs+7erjKO7CNM6fy8TxzV6Q/6IK/lr+3y29ZLqe+VsvtLwu3Qj7TnvY/F3nqL390fN2Brr1+O7IwC0L86GiLstbOhL6llb1j4ElldQdLOh2QP0+d9lqzp2SUUBkBWjelqmFbHyGN5ecwoL/r2qf+cSZPuFh6rvL9zLRH6hArdeL5Bbkhmy5pdUtWlM5Wbac6w6lISXedqJ6dbU9vg7wgIbqdfSk9tZ9yUqI6douE5z6nuWjt5KYj2s+7erjCs08q7d1800Q2iFCgbJBl5QX+YV4qGV1RhSn9Za2tx98gK1Z+5Gl28OWbopRhP757DywE3Uj9qDGIF1lhiGwe30HFYOlS4Zz3Mx6+8EJDwwfOgtbOkhLNx9DSv2s6sebzh5B/Wj9mCzlaw2fj2Vv/dNna3ExVTdnOSSHk9qrf3dAQCpanW9SMlBAZAVUw5b9Ajy0nrsCscyDVc1qhgrTJQzOumPcwhdchBb4sWXxO/w9QG0Wbgftx4LO6Ea496TFxiz7jT+d+oubmoMn5T19YzKgq/3JAIAPv1LWH7Vp39dRKclB9FP4FTvmdsSsOHkHfRaYfzQ2yWNIGrW6yEUY+p6WYKtzPgA6L0O/qrvlQFg/J2nVllgsE9jb7O91sPMlxiz7rTWsD0xHAVAVkx5I+pavpzWYzEJ2kl2Px5MYv1sbDVnvryRXZeKXnvp3kTRx1SexDp/cwjzdl3Rs7dxui0/jNhrafhs2yWELT3Eqn+T+aJsd1E/yrKuXjgh7E08HKJc4+o8x8w5LlIum3Lkhmkvauaq7J4owY2FjVoQtTn+Po7ceIz+Px5Hi3n/GX1sofjOfZq918q9KjnZm7hFRQF67LU0vL2m7C5WLTWrCIBWrlwJPz8/ODg4oGXLloiL4596HBoaCplMpvXVs2dP1T5cj8tkMnz99dfmeDuSUeY8cHYrc9xpqed+KH0Xe8Pg7tmrj3SfzB5mGdftu/rIbaOer0+ORl6FelE1Q/JJSoLTycJyNXLzS96U8oocNwJCGJKPIWQNqrtPiteEMiTJ/PdTd0Q/x1Cm6DVYNqix1jYpFkrVnLZv7rXfTiRloOmX+zjPp5ptU/a6qy+PYQoMw5g8SC6LLB4Abdq0CZGRkZg9ezbOnj2Lxo0bIzw8HGlpaZz7b926FY8ePVJ9JSQkwNbWFgMGDFDto/74o0ePsHbtWshkMvTv399cb0sSyl6K4BqVtB4TOtS+dN91jFlv2LIWlx+WrmnF6r0+QvM8SpoBq04I2k/qZFVzUF+E0tQ26cm9ua/Rlpoz/hV03OxX+Xh79Ukcv5mOmdvMN1PIFD2etatU0NqmOQxviIqO/IHuwFUncDNNWC/T3YwXmLH1oujh9uHRp5D5Ih+T/jin9dgrjRuHFa9rp5m6TtLBRMPXeCT8LB4ALV26FGPHjsXo0aNRv359rFq1Co6Ojli7di3n/m5ubvDy8lJ97du3D46OjqwASP1xLy8v/PPPP+jUqRP8/f05j2ntqrs54YehzTD3zQaqbS84Zo3wSXig/6TEdcdb0vIP9Jm3q3i2F1cAYK6p4tYwJd1GglwNc/tql+lm62kWWdRXP+lpjmEBRdg3h3A8KUNrGENXza87GcbP4JtngpmOpvod0qzGri4u+QnClh7We4xX+YXo8PUB/BF3D92/PWLw6yuXulCasY37nGjqHmUpfgeINosGQHl5eYiPj0dYWJhqm42NDcLCwnDihLA72ejoaAwePBhOTk6cj6empmLXrl0YM2aMJG2W0v2nL/DnmXt6K9zayIAeQVVZC+/9LPHMpR0Xtbt7pWbpC/+h68V3UZonNkD4TBelFbE34Dd9F+exdEnOMF9PBh9rLzDH5dZj010E/jnP/v1vXM1V5/55BlalTuNJ5NWVo9Px64OiX0fqvzWuz0PqGV9KUvTOBs6KUX2fK/LvU922c+whPb6bSa4A6OStDINfV5M5zpzJ6TnYfOaeWat+W5pFz4Lp6ekoLCyEp6cna7unpydSUvRPXY2Li0NCQgLeffdd3n3Wr1+PChUqoF+/frz75ObmIjs7m/VlDqFfH8QnWy5iDUcujPpJwMO5aCqoKQOIL3eavhaONf1dKRO51eWIrLvyzb7rAID6UTF69mSbu+My5/ZrKcb93v1zXli1XgDYf417iLms0syR+WjjeZ37W3t9Gqn/1i5wVNk2VSeirh4gIbjWhVMoGINywYSWUOBq8uCfT0o21P7sFfvclF+oQJbEw5qhSw5i2paLWHmg7CyJZN1/xXpER0cjKCgIISEhvPusXbsWQ4cOhYODA+8+CxYsgKurq+rL19fXFM3VovxDX7pPezbV12ozrJSzl0xZFfVJjrCFUI35g84tsJ68E64V0Ss5GjaTQ+wJ+wDPeH635eK66jXpu2irE9trVdptPccOHp/rWT9LV56KNTDHumn6PiNDCZmFp6uXgmu6/Jj1p9F4zl7RNxl8f6vqGIbhbY+xtdyULmoEoLVn7kbjuXvxR9xdSY6vbunrG7uywKIBkIeHB2xtbZGamsranpqaCi8v7do36nJycrBx40adQ1tHjhxBYmKizh4iAJgxYwaysrJUX/fumbf4mPrQlpL6lHab113N1nDXaWjXPwAcuGY9iXyXOeoolSXG3mUrnUl+gntPdA/pnbyVYfHil37TdxlUt4qPFaRx6WTMsA+X/s2qaW3LfCHspkmMB5kvBa04r6u3NoVj1qsykPl+/034Td8Fv+m7OIcdE1PET+N/+iKftximVLlBlx5kcm6fobbsjTHUA7hW/m6SHLMksOgV1d7eHsHBwYiNjVVtUygUiI2NRevWrXU+d/PmzcjNzcWwYcN494mOjkZwcDAaN9aerqlOLpfDxcWF9WVObQLcdT6u7Gl2dzZ9rQl9uHpOhDp8XTsA8pu+y5jm8NJ3UX7EMYXfRCkNVqlxtYpGHyMx5RneWnUC7Rcf4N3nTPITDP75JNos3G/06xlr6uYLkh3rDMfSENY0s1DqHiCuWmSmKKXw5vfHBO2nq+CirmBbfZHSthq/kwWFCoQv159grelEUgb2XE7lfEyqQNnUa6It3nNN9f3JW8at53Yz7Rk6LTnIOyS/53IKOi85aFQFdalYvEshMjISq1evxvr163H16lW8//77yMnJwejRowEAI0aMwIwZM7SeFx0djYiICLi7cwcP2dnZ2Lx5s97eH2ugbzaF7PXjruX5AyB9F3ypGNPtbWdrvgjj7TUnBe/bwLso4D3F84d/lyNpOYMjz6Ak4bqgiSWkTMKvJ0xb6+aN+p76dzKQrmFhrkU/r+noPfjz9D38fU54jpaxpO7hXHtMO0+xQMHg9MwwtK/tIdnrcOXviCUmEN2tlguoWTdMqAn/O4uUbO6gS6oAKLROZWkOxOOnQ4ZNqmEYRuvvpMd3R3E7PYd3SH7chnjcSs/BuA3xBr2mlCweAA0aNAhLlixBVFQUmjRpgvPnzyMmJkaVGH337l08esROWE1MTMTRo0d1Dn9t3LgRDMNgyJAhJm2/FI4KLFLm78E90+3i/Uydd+HWItOMCwTeeyJ8yEV5klpzlPsk8MH/tP9QheZMWStGgnklQqZBcxWTkxLfrCopNPtyH347eQfPcwtwQ6PCMdcyMwU8a89kPM/FJ39dxMebzpugldw0l34xhQ61K6NyBTk2jGlp8tfSpOu3V8xv9vu/n1V9b0yOopjZYWIwDIMrD7Ox3sQ3Eob6eNN5NPtyH2sSgdD8QmtY7NfiARAATJw4EXfu3EFubi5OnTqFli2L/6AOHjyIdevWsfavW7cuGIbBG2+8wXvM9957Dy9evICrq+7prNaI7w7GhmeM5mMRya/GMqbiKdddszVQnqSa+FbkfJzr5KaeY6FvyrQ6qf/on73Kx74rqaJP3lLcmZpiFhDDMDhwLQ1pz4qHKAsKFYi9qj3EMCSkOh6YuDji538noPOSg3hj2WHWiuixHLPorvFUTtecwWMOmnW9dNUZMpSrBRPBdc2IvWNgmQlTDOkZGwD9dzUNPb4zbnKEKSnLR/ygnrOqdl5Qr75vjawiACLsKcyafzRVXYtmsDnZc48Da16IxravKVm7NJfRMDTN4XluAe9ihuasO6FQMNhwkn03dev1yvYvRZwA1WdKtPATnjS4/rh0d3Kv8gsR9MVejP31DL7cKW5dNSmSMzUPUahgjM6D2Xr2AUavO41OarVvfj5yC2PWn9Ha10YGpD83fU+cspdJfUo011DNEZ6eXLGfNdfFXWwJDM0aNN4V+WfBlkR/6UhoN3SoXWwZDCGM/Sv7Yjt3yQxrVsGhODBu/uV/vJW7xRTzNRUKgKzERxvP49LrqY6afzTKBDg7nt4XzTseKe84NUv+n7urf4YGl406pmt2XXbIbAmkK/bfVK20reTwenZdiogFQi/ez1R9L2YhxEUx1/TvJECXbw6yir39dlLcdFixn7Z6j4ySeiG8/EIFAj77F/6f8S8Jkfbsld4LuTI4VeZjvMgrwG883f+aK6gD4pP0n+cWCK6yqy8G8a3E3csi5Fe7oU/xxIvHHMHV6HXilrPRnL59Otmwv1tr9cWOK7zrtRlazkJznS8lY/L9jJk0AphvEVspqS9anFeo4K3cLfVMRUNQAGRFlIsrir1j9NE48YbUlG4ao+bJW33MHADSBC60mpPLfyJIepzDedI3hT851nhSduWrr/PjLLfjfH5+oQLT/7rI6nkwZOqssZIMqIq8+6P2qOHuCEBcj8LPh5MQMi8W3++/odGG4jwT9XWT+E76IfNi0WnJQZ2vpV4DhmEYNJi9h3fRXc3aKEDx7KeXPHeXmm1rOHuP4ErLf5/Xnc+09wrfTCD9n7X6Wl1cuVW0FpS2JXu569UYev7jGwJTLvBqSF5Vx8UHDWpLScbX02+NKACyIsoOHrEVPjW7EqWcMqneM/OUI/H3QKKwisL6ysKba3X2jnW1Z1Moc1fVexR6BGnXobr/9AVqz9yNjafZQZQxib5eLuyhiWcmHDOv7uaoKqkg5uOe/29Rr5XmBWe7WkCwW214SLPXUJ2YZUAKFIzoXCVlEMt3sTqo9vuanC4uiNQ3Q4n/NfUHL/efFt3pH7r+2OI1k0qKVYeSOLcbmpqmeSOptOdyCp69ykfY0kOij1kSFx0uSygAsiLKtag0L7Dq1LvKlTQjbkMSlaeF1wUA+LqxTwJuasM7XLVFPv3rEh4JGDp6oedEIDP4tCUO1wWVa+y/VyNvrW2Rm6SrIwMUDSHtmdyBtW35fzd49jbOgamhcJLbqXoXpAg3+daCkmo08xMDFuNV9vDwJWgfu1kciJsjufTYzXTBC5HG3X6CkWvj0IenFo7QYWIpppKXZOuPJxv0PL6P16GcjSpANSdzlTaRali+JKIAyIokvp5qq6sUuZA7YkOSAJUn13a12DU91h5LVn3PN2a7ZI/xpdPNtTg513BE5ot8rbtuzc8BKFqJWkqL+jfSqscTfVS73oomQ9aEq/m6hIIy4dvY3AQAGNSCe8kYqZLatxlQN0f5O8rXo+hS3g7rjydjS/x9syRh8vVScDmmpxyG0OrdC/6V9oJmTQUehdA3VMnn1mPuHrywep4WWSD0s23CqjwbM7N058WHrFUHDKXv/G3phbD5UABkRYTUVRHyh1jRgCRA5WFtZDJsn9hWtV19rRm+wIwrQVZTOSsps8x3YVywm33R4Cs5IKXaVZwBAJM61xL1vLkiZ3xx+fmwYYXP1CkX6dVk6LlOisqw11/fRPD1otja2GD29suSVoXmU1CowJEbwmp8AezkUc7j8dQZ0vTXWd1LfhQUKjBt8wWdM6nUcS0i+vu75q/9Y2q6AgFLrGMo9Hcnv8Dw4GLi/87p30kCn2tMPFGy9JqEFABZERuZ/khZLiC/x1kuPgdIGRjYyGRoxLNMAl+l2yM30rVmVmmqqqcOiSEXzQW7r2LchjOi7lD5dj0rYP0hqdV/XYF6aKsaop73i1qvnKGup5oucVtfkUWuodQTSRnoteKo0a/9gUaSviYXB+7kdlMQW/hQXzJ9Ck8yuFh/nL6HzfH3MUVgEKh+M+DiYIfRbf3QVqOHVDmEXpLx3fTkFzJa5RbK6ehlPz69M+tnU1/kbQy8ivPV6PF2FV8yQV+w9vsp7lmqlh6upQDIQrgu2jYymd7FRlMFnASzXorvEmVUARD/Plcf8ZfX33DyjlGl/tOevcLPh5NETTn96dAt7LmcisM3hM+Q4esBMtd0U/Vii8r1jKQe/usZVBVRverr3IdrAV4phsUAwE7PGZmrIOZ/HIUOTcGctUfU150SQl8yfedvDkkylPCtyDwz9VyvVcODMbt3A619rug4N5QUfJ/ti7wCrZ73zoFVeI/jrXGzt/PiQ+QWFCK/UCH5Gm2GOHLjMfym70KjL/ZyPs4369IULJ0kTgGQhXBdiGUy4FWe7j8QRwG9O24i6tIo/fE68drQ8XNA9x3vDo2T++i2fqyf31l3GvP/vYZ3f9UueKfP/3juLrioF7MTQ9fMJkD4XZ76cibKwEfI0KcYH3aphXfa6S6Gqb6Ux9OcPPhN34XAWTH495LwizbfjLU7GTm809AB4IqR61R9FdEQAPDj0GZaw0b2djY6e7d05dcBxblS1oprpXOxxF6E7dTvinjiL74aOmJo5uFd+7Kb0ccUQpn7w1crKb+Q0cq/+ryn7hsMdanZuQj+8j/UnrkbLeb9Z/Fhn+HRcZIcx5DZiieS2LOB1xwxfijeGBQAWchTjqnuNjIZNpxM1vm8Lmp3Hnx3LFzds5kvdFfMVc4k4xrvNwXNu0hlF/O5u5mijyWmgquhPQDtFulea63L0oOCjqNc2PbTboGq76UOgOp6VhC0nzJQiPihOF9G3xCSuln/cFepfW9DPOpFxaD5V/9xPs41RHZURK7MsFY1cGNed3QPqqp1Manq6iBokVY+a0Y2N/i55iAm2OdTTscs0cSUZ1oXNvUeIL7+J758MDHaqK3OPrVrHUHlPPT13P40PFjvMaZv1Z1sfPlhllbRRV83R73HVVoUc021iHTmi3wkCyy8qSmgMndwbqn0YiETNjRdS2Hf/PwRxz/j2RwoALIQrrswhuEvpqbUpV7x6tdcQRQfMftykXIWxJQ36kh2LABITjfPdFFdhC6+qgxa1e+qpU63lgkMqLouO4zjSekGr52kD9/4/sNMdi/G13uuqWZACsV3Efet5IjJRpQrMHbNLLFDVMNaVRe1/41Uwxc5/fN1Ly/f/0ta9it0//YwIlayE8jVf5v43t5gnhmBhmroU7S+XgNv7bIf6j7eWJzEyzAMun/LLm3Qtb4novUEtdl6bvqyXxVIWrXY0JpnfLMALTXBSrMMxr0nL/Dm97rz+KTowZQSBUAWUp7j7uavs/c5q9uqC/Qqvru/y1MngusCaGwX9W8a62eJoTkcMlHkrCcu6hcarrtAZb2kVcOC0ax6RVHH1ixOKCVlIrl6rpdmD5Cu4Tapp+Me4FjUc+m+67gtskigGOs06rSsPGD8NFylelWF9X6ZivL/57ieKe1KYtaRA4DTRpRi0Dcl/3rqcyiYorXP/KbvUm1Xr0vmz9ML4fS6croUPUFA8Tls1bBgRDTx5p0hpz5sdT31uVaeokwmY900cuGa3NHa3131/YV7mZIOWxn6Nxxap7iIq/pIQKGCkXyWmpBAXvO8NWPrJVzQc/366ZBlh7w0UQBkIUJremhSLxhYyDMttirHBfzqo2zeX2r19b34Lv6zBS7Kl5yew3qdlKxXqBcVw9pHXw+FkD++aWpF8tRn9igUDJLTc1QruKdmv0LfZtUEtV2Jb1V4KSiTRX/ScTHSVQBw50XhOVrBNSoB0J2wuYsjUfe72Bt6l6ywFq382QFEu9ralb7FMHY4UnkxfXvNKUH7N6teSdTxMziqsQt1Kz0H38XyJ0BrToBQLnOjPoVZzhOIKD82fbN6sl7mC5rooGyKr5sjlg9uiq3vt+F9bSUpE4y/G9KU9bOUAZDAagZapnULVH2v3gMa+vUBNJ27T7JJDICwXiXNgLokrltGAZCFHBS4hIQWtZPUg0zu7kSu6ZzTtlxEzRn/Ys9l7SRg9RL+xnZRhi45yLrDN6SYnZDYcItaDZOQmsV3azP/voRQtYv3tnMPUFGj2KA++qpWG0o9sMtWW7BWcybEcY1EQXVicrRWj2iOLyMaYtnAJrz76JvxwbeSs7XQTPjnuykQSkj5p1SNv5Et41urvufrleUjL2feU7BmErj68jaaNyb9fjyu9Xy+v02huXs9vj2CTksO6i3e5+fO7mlq6OOKy3PCBb0GF103AVwqV2D3ZD1T60HXF4jp88rA3hpnuZ1qjcKw+sW9WtmvCvAirxBnJFzwtlBPBMQVEJqy19hUKACyEENreqifo9Yd005Ci9STXzNuQ7zWNqkTn+erlf7XvKHu1kB7jS1NZ0WuON++dnFNEs2kuvP3MkXfGSqXEglvoLvrXCyu4BMQN2tPzDpvbk72GN6qhmqxV0O8sYx7JWepGNt1P6A5O/eEq0aSmKJ9+nqA0p690rrQqBceFZtkL3UCvKa3gnX3fqoPGWk25f7Tl1q9Ch7O3L+r6rMKm8zdC7/pu1jbgKIbgAeZL5H9qkBvzpxmAAIAdnqW+NHV+zSitbhaW5qaqvUKR/UWPgOMC9+iq5rC6mkHbQemhmL9OyHo38xH67HJf543ql3qMvXkjG4RWETT2lEAZCFSVBo+ezeTlWvg7eqASV1qA+CfMcDlqMB8BaG4aswoXbifqff5z3nylfIKFJzd5/qSCjvUETcs4lOxaBiQb6kHQ+25zJ3gLuYaKOVCt0KYOsFy/i5h62Tx6VS3CvaprafGVZBNs2ifLjIZcGF2VzTmGQYNmRfLCtpkMnaPgNhlCWxNHABxLemiTr03YvDPJ7Uen6+xjhnf8LWjffHvpfLiOUxjGFC990jfecCQRGHNtqoLrSuuB0hTFZfigOyVwACGj9Dp41x/65UryNGxTmXO/wcpV2HXd4Mupl6bNaMAyEKkuvN7e80pVd7OuI4Bqu17Pu6AS190FXQMUxbn0rwjfiSk54vn3Ffn890I/uo/rVW8dbW/nK1MdGLmqLY1Xz9X2j+PZ2rDXuqJ2WJ+E4y9YIbUFJd0q2neriuISXhkcD0lTetPGJ5cr1Rb4LR/IWQyGVzLl8NPw/inT0f+WTzLbGrXuqoEYADIyRXWAxTzcXvcXtADtgas2yeGvgKFOy7orvv0q8D/H7md9sVas6aSelCjb9kT9YBKqDQJAwBNi2MSVd8b22v5yV/CFvk1ZbV2fcrr+fzFLPGilF+oQJsAd9a2v95vI/o4UqIAyEKUCapiOXLcFdT2LFpTSr0wnZ2tDSo4CBv6UF8hW2qXDVjfSd/d36caJ5BLD/hP8rqmNffnSY72cy+q8SFFcTd16tWOW6nNMuFbVZ2Lvphs/5SOup9vZAC1+shtjP/tLH48eNOo45hL/aq6p1Hz8RK4HED3hl6sIczEFN0Bh5K7kxwymQwuAv9GDaUvedeQHD0uXLO07mnMZlT/u9bXwSO0lIM6fcM2a0cZXuPp0PXiavPNaxh3EyHUkxzz1GTjMmb9aZ2PJ/EsHKtLavYr+FYqrp/k7epg8HVQKhQAlTBc4+DKk4nQ4lybTt9ljZer16RxlmuvlcQ1LXu4wPWrDLlT4Eq0VA/uTmkspeCuI4fm/dAA3sf4ptYqT75XHpnuDuzPM8W5SjKZDP8by85T4VvfrJKehW79KzvrfNzQdYM06Zvuai2UNwem4qGRq3LuXqag/D4xQa8YmiUn9N3JS4Xrb0mzvo560KNZDFNo/aSOIoez1XUOlCanT9mL2qexN+8+cTO7GP06Uix7cvVRNmump9Bj6qsNZsiMM5lMxpr9XM7IZHIpWL4FZZSU47XKuhJC75o+/esSq0qv+i9l09dDM+pDNOqzxJT0LQ2hpGtts1Ft/Di3cy3HoD58pGnd8WTM+jsB3ZZrJ+zqChj4co2UNLtr9RFzwtJcXDFAI3Dhm4lm7JpZ1SoKr2BrrqrgpjQkRFyhQbGea/xe3sl4gVYLYvU+T4qLG2d7NH6nDS26J5Y9x41ZcgZ/D5BmfC+0mZpL6Ii1sF8QPupSG9e/6m7Q82u4F//9zOnTAC38uHswqlTQ3YO46+IjvYnEUgTJ3b89ggn/O6taf0+qOmI5BlbUV1+3cd3oEEnaYgwKgCxkc7x0JcCLV3I3/ljKvJf2avVUuPKVpPgzms0zm4Jr2Q59w1EbTt7hLGimK2lYczozAPRtWjy7Ql9viybNWS+6aJ7cNC9UPx/mLhi2X61w4bJBjUW0rsicN7UXsuTTeA73YoklSUsjc570Uebfcc1a0kXqgpZKmr9XHk7SFCbUR8h0fvW3fFljPTihgVrHOpUxS89Cv7oMDqmOyW/Ugb2dDSt5XWi5B/Uh5EpO9vhmQBOD2jHhf2cxdfMFnTPXpMxpUuYT6bvpk1Iljdmney+nsG78rWHdPQqALGRAsHQzjJRDQlLkrPRqVBUAO3eGa2aLFOdvvh4rru50Q0vR6wqAuBbsVC+sJzagTM0WfsJy0hiaqFieHWzV4BnOTHpcnADet2k13F7QQ0QLTTOLTGylbXMyJJdEDOVsTrE9uqYamtIMJFr6SxcA1q7CP5woFzBhQH3m0IV7mazHhJ5PZDIZxuhZ6FcpRE+V7QpqBVTDlhb3Ho/vyD9s/kzjHOvrVh7dG3KX9uhaX/+Qm65Fg6WU/TqFQNdCqNPC6wo+np+7/p7kfya0Y/08Z8cVwcc3FwqALKS8fdFHr2+tGzE+/Uv3on5CKHtAAtROdgkaScY9G1VF79eBEp9OSw6yKkyLwZXnpFksUChdf6hcgd2fZ4q7pX0qiVsXSsgUf6UaGoXeNC+IQutsmPoCL8RUESdOpfxCBf6I41/Yc3JYHUSPbI6LAmcyWoIxBfGETlDgomvWo2b+kdieKV0MrV6vpKvCudihuii1XqBYnmHh5YOb6DwG3xCTemCkSTPQlclk+JFnxqC+GkyAYSuqKzWu5ip438Uxibib8QKXdExKmdBJ+BJFfZvqf2/VBQRJlkYBkIUoz2H6inuZm/KCqr5W2fcH2DN+egVVxVvB1VBFx8n1dnoO+v6gXUlWiFCOREdD75ScdZzMegRpB3Hxd4qDNkd77ucemBrKuV3MDCt9vSYnbpluZh7fek6GUp/ZwefwtE6sn6P+ScAMHatwfxRWG13qecLFoRwmdOK/I1da8XrpAjH1r4xlaK/kuI7+gvfd+WE7rW26ii1uUlu3q7FvRVR1NW5xV3XfDOQfchWS0Ko5eUGd2FIc6rHLmPVnOPfRF6Dy9dgq/4y58pr0aa42q0nfhASAP5VAfdIHH0+RaxZ2+PoA5/YqFeS4ECXuRqM05AcCFABZzH+vV33X7Ao2RjWRPRa66KrDoWCKAiXN9XKkoj7Mo8S1ZpUQuuotvdlEu5qqEHY8d458s8q4NBe5CKYuPw0PRiXHctgwRlhSYb+mhr1vPr5ujlgyoDEW9Q/i3UfzblCzYrcu08ID9e6j/D9R/90RWgH6nbbChlSk0jZAWGHGrR+0Ua2Kri5XR2/oBrVFi7dJXGNF17plmvke6pIeP0fkpvNa2+PvFAdEtzj+5nXRrP10i2NathPHjFYhlAsET+oivEdkevdAeDjLsWRAcZAopMJ7udd1oO49ecHKIWw1X38ivVS15NKe5YquFr+WYxWCkogCIAvZdEb3BWBq1zpo4O2iNT1aF0OKhwFAv9dl1dVPYrqSGpXl8MUs4aAkpEtefZG9Jzl5GLbmlN7Pi4+umRS+btoBo4OAZE6+KaC1dORIaOqlZwhRjPAGXjg76w1W4rouQdUqSvbaSm8FV8OgFrpnXK0b3QKA7rIFhuKqrK4+i++j1xXSuXzSrS5CarphcpjuZWSkIjQPSxlwaJam2CxweFSKavNCOcv5L6CjfonDVo56Q+prGR5MfKz1uC71NOo7df7mkNY+hua7KVeYjxBxozC+YwBOz+wCP7XEXlcBaxDa2dgg7dkrtF98AM2+3KfarjnLiutcpT6jSkm9Z0bKBVxLKwqArFQDb1fsmtQebQTeLQLieiDU1Xydj9KtYfFFubyOk4fyxKo5dVsIzWnD+izdl2jUUh26hqXKcRTF0Txpvt1S+6LOt2DsCR2LmGqSOndHzPGk7CkUYu7rmWfKC7muHAtDcd0Nq38mugobOpSzxZ/jWuOjMO4gqWdQVZ0FNcUSEmR/q5a/ci7qDdYael/vSeR4BlBgREV3Y6fl6/p8+db92n6+OCjKkXgJkend9fca6lNNwNCuOs2/QSHT2HdceIgrGjPiuGYIcvWCcg2FqlfJN3amobJH6v7TF+j8zUFW76JQmkPf1oYCICv1z3nxFVoNGbMGimdgqP/9chVEBIp6mRp6F3XLG1KngutORhd91V310XUXzPWYg0ZJ/6/ebKjVM8BXXVhf3o6yJo2pp2brwzeEZwqDmvuqimYqP29TzADfd0X30hyG/K7umtQOo9r44cuIhjg0LdTAlmkT0jOhPjxbztYGw/Us5hl99DZqzdxtUHte5Rci+6X5pkcr/Xe1uKSD+qKsUQKmuetLmtY1k8uarDl6WysPqEChHcj25MhX5KLek6ZZbFKs9349g1f5hWi36ABuPc7BrL8TAIgLrPgSoU1xE2QICoCslCFJZob2AHHVEeIbXz4X9YZRU3ib8CwyySeRo7aPIaZ2ZQcxfMFiOTv2+7axkWklrfK9f32JnBVfDzE28OaevfHbGOHDncbg+7/VlcNjqEVvNVLdGStf9+4TYUU0ufANu3rpSQit5yV+tmUDb1d80acB3JzsJZ2soBlke2v0nnD9HbdWWzqFy5c7+acY6+u9+vnwLaQ9E7BGnwmpV4znK5CqztyLApuTZmznaG8rOICPSy6+CTOkB0h9VvKZO09VQY9SQaEC89QWMOa7UdbHz93yNYAACoCs1gMd0yMb+nCfzO05FiQUQjnzSb04GV8PM9eih2JUNGFxQV1cNV73W54pspoXJ0D7ZMsXQLTSc5HSV7CyXW3hw53G4Arg1o5qjkEtqktaOTlII4FX6DALV52XP8a2QuNqrljPUz02r1D3setLWG5CjL2TO+DHoc3wec96qm2a+XVNNZKLuYYojcnnUU/M5XLxfiarN8bShLxXcy3xYQ6jf9G97tbKoc0EH0t9XUeOjiRO6rPlejViL++hmW+Wk1vISoDmK6zIVyFbSZlHamkUAFkpXRf+hjw9CIYOgSlzbM7dzTTo+WK82YR//Rx1ynyGDIkCIM1TqsvrBEXN6ehCetH47sb0FaJUru9lqnWghPJw1k5Ed31diFFsgnKPIO4icACw8b1WrJ+5ZvepWz2iOXo1qsqZi9M6wB3/TGyHIJ7aJ/qCK0t94tUqlUf3oKroWr/4c9LsxdIczhE7RPn9/hs6H+f7zJT+u5qGRTHXRL0mMR3N6tBclfGFKBR4w1FRbfKLvqFFO1thv5v6chJtpVqU0EjW0Yoybsv41lrbwhvwX1g+U7ubVGdMYTZNpiiwF+hVgXf4R1PokoNGJXVq2nHhIetnZdKmZq0fIW+bLwHzh4NJnNuVVh8punMyZY0fQykDQSEJuur+vcSfe6M5DVlf7aM36nvi+7ebGbZCusZ/ieZsQ77/VxcT5yLIXjesursjJofVwaxe9VXLzShpDlV82Jl/xhqXJXuv63zc0GEKdYbOWpSiOn1Zs+0sO/+Tay1GPur5hVy5RFw6qM0ebVdLdy+00EE1fTG8mBmzpkQBkBXgqgkzkyfIAcB7gbj8ULvKp66uSClrEAkhpnDX/acvcf+p4VVSNWnOtFDmRWheGDvXraL3WOpd9IbkzVzUsZJ6cA3dXcemogx4b6WLq8cihil7vhQaQQRXxV51ywc1QflytljxtvDhBUOov+xHYbU5h/fUmz73zQaSlkgwROJX3bS2zekjfA05pZoeTmgwe48UTTIrS/bQKhQMvtnHDmjFpB2oJxfvvCCsdpp6QVh9M0S51k/8sLN2vSR9qQvVeZb6MTeLB0ArV66En58fHBwc0LJlS8TF8a9VEhoaCplMpvXVs2dP1n5Xr15Fnz594OrqCicnJ7Ro0QJ37/KX3bdGhhS5yuaYYv6/sa0wsDl32fLzZg6AxJLyRKS5hg/f5xsaqD8AAoryVHo39sbA5uw13YydTix0aZS+EhczLH594eX1dfF00R5m0/U7XVejsJ1Y+XpygDRFNPXB1S+7oSNH1XFzUx92GNHaz6TLmyzu30jvPnI7W61EZLELAwNF1eCFsqZZW7oq3Iulnvu1+C39nz3XEFR7EbmBQT4VVd8v2ctdLkGT+pD4Kz21g7jOb1xLaOjL9aRZYAA2bdqEyMhIzJ49G2fPnkXjxo0RHh6OtDTuhLytW7fi0aNHqq+EhATY2tpiwIABqn2SkpLQrl07BAYG4uDBg7h48SJmzZoFBwdxZcNLonIc47PlbG2wsB/3H15tT/N2Q4otdy/ldUCzKJky10ez0q7Ql5zVqz5WDGmqdbG6niq8u5rLex2KZ5wdvs5fHO6pgXkB+vAt7CiWHccYv67k1twC4xaFFNLdr7yom6KXjW+BTiE3MsbUuRJq2aDGGNi8mqroqT6avbWmKqqoLCqqTIqNEJgjqIt6AUxd+P7PxM5U1SU5ozgIDPTSH+Rz5e2IuRFc9t91bH5dNFbXkinq1CfVcF1D1HFNLONq3/t6AlpLTUrQZNEAaOnSpRg7dixGjx6N+vXrY9WqVXB0dMTatWs593dzc4OXl5fqa9++fXB0dGQFQDNnzkSPHj2wePFiNG3aFAEBAejTpw+qVBF2Z1+S1eWZ6st3Dl59+JbBCXZKYgrEiV1MUXNhR2Pw/WFrBkb67r71dd3y1d4QejfspJaTNGJtnGSz4ITiO9m+HyrsDn1mj6I73q857nZ1Fa8T24OjaZKOSs9Ks3vXx6nPuuAviZeIANh3+uqEJNX3aWz8RV+fvk2rYfFbjQVP5zfk5sOQNeYCZ8Xg+M101dRvKZZ3ENqOWb3q46/3tfMvx+m4ePPNHuWjXsZDV7FIpY6LD2ptCxRZwmHalou4dD9L70wsJfVzXpUKutvINbVeM6ct7rMu6CSwJ93SLBYA5eXlIT4+HmFhYcWNsbFBWFgYTpw4IegY0dHRGDx4MJycin7hFQoFdu3ahTp16iA8PBxVqlRBy5Yt8ffff+s8Tm5uLrKzs1lf5qZ9Ida9P1fdk094VuXmu6gfSHyMdceTBbWPz1/vt9G5zIC6JW/pno6r6X86VgsXy92Ju1tb83rPl3B4ckYXzO5dH/9+1F7n63DN4HmVX4j3f4sX1E7NO8CIlccEPU+smI+53wffBejTboG4/lV3vccd28Ef17/qjjYcyZS6Jn6MbW/celz6TtxA0d+B2AUk1S3ox5/vZcywlZDFZM3NkHezf0oorn/VXVBPh7q315xCYmpRoKD81xiXdOTYaQquoZ1/qWtJIbHVodVvKCpzzL7UxFVlni+I1pUr9jDrpSTJ75p+E1ANuooRf2PmJugT2r59u+AD9unTR9B+6enpKCwshKenJ2u7p6cnrl3TPyUzLi4OCQkJiI6OVm1LS0vD8+fPsXDhQnz11VdYtGgRYmJi0K9fPxw4cAAdO3bkPNaCBQswZ84cQe02FYdyNshSy/nlGkJQx/VHakhtjNpVjMu98HJ1wDtta+LbWN1Tcf+L7MhZFdTDWa417VNp61nx1bDVnY96Q/X95DfqYDxHECLTONVX5blL83J1wGgBi2Zy/b9N+P0srqndCX7/Nv8ispo94HxFA42tpuzEs9K9ruu40EKbfPvp6gEawrHkiLUZElJUJ8lv+i5Jj+tu5pooH4fVxoV7mTigY/0tQ+M5ezsb+Fd2Yv2+C7Hldb2Zyw+F33xun9gW/zt1FxtPs9cJvCAiAOKiaxFVsSOB6sGY1LldSwY0xk6eRaJlAB5J2IOu9PspaW5KjbkRkZKgACgiIoL1s0wmYyVDqf/HFhYaN5YvVHR0NIKCghASUlwYTfE6D+DNN9/E5MmTAQBNmjTB8ePHsWrVKt4AaMaMGYiMjFT9nJ2dDV9fX859TUXzwqdv3LdVgLskM3YqilgFmG99MCElHfjez8yegZi86YLgNgiVvJCdGM83u8FXY0jL2OmZj7JesRZEBIDYa+ycNq5ii0pCT7C68oOE4DsXm3L+i67hDWMLbJZkA5v7IuFBluDFbIGi2UK68nJ+Hh7M+9jHr5d20RXIKRcEBYD4z8N49+PSr2k1neURpNKoWkU0qlZRKwCq6WFclWH1Yf2dH7ZDrxVHVT+LHaLTXNRUSrqqYR+9mS46CDVGa393nLiVwbkA7IjWNfDriTvo1agqZvWqj7wChUl6pwwh6JZOoVCovvbu3YsmTZpg9+7dyMzMRGZmJv799180a9YMMTExgl/Yw8MDtra2SE1NZW1PTU2Fl5fuRMycnBxs3LgRY8aM0TqmnZ0d6tdnryVTr149nbPA5HI5XFxcWF/m8svrFbLVk1+F+OrNhnDS6PExZALS0DWnBO/LN1NDyJ0N37m6b9Nq2DGxndZ7kZp6Aa9hrYp7G8IbeGJq1zro36wa5vcNMvrOREii99m7T3kf06xLZCouaieq5mpJwWLztMTILeWrU+/5uAM+EJgrpc7ezgYL+zdCTz3T34eEFN+U+X/2r859haxErot6sVV3AUM36tafSDbqtY0lZvYZAAxuwX+zqzVJQuQdwjiR53WpZGsspXTkE+5FSd8K5p4hLFSF14HMd0OaYnzHAGyf2FZrn7lvNsTpmWH4/u1m8HRx0LrptCTROUAff/wxvv32W4SHh6uChfDwcCxduhSTJk0SfBx7e3sEBwcjNjZWtU2hUCA2NhatW2snpqnbvHkzcnNzMWzYMK1jtmjRAomJ7Ol/169fR40auhcTNCf18uHKXhWxF14bGxkuz9Wu12EM5ardfAa24P5jEbLmjK47p6Bqrrgwu6veYxhDffilu9qq9zKZDBM718Y3AxtzrvwulmYBPi4ndRRCFDqMKWTBSF1cHMrhl9EtED2yObaoJQVLOQVYk9hZgCVNXa8K+KSb8auQ8xmgUXIh80Ueb9kFY2dtje1QE4FeFQQnv6tL0lO4b9kgcbmAptZYxKyvqq7iFnOe9jovs5Geatx8uMpJCKH5W1GF4zg353XXu0yKPsrh7soV5JjePRA1eNb4EnJetATRt5tJSUmoWLGi1nZXV1ckJyeLOlZkZCRGjhyJ5s2bIyQkBMuXL0dOTg5Gjx4NABgxYgR8fHywYMEC1vOio6MREREBd3ft6Y7Tpk3DoEGD0KFDB3Tq1AkxMTHYsWMHDh48KKptprRCrXS9MjCQYvaD8Dqd3OrxrHKuj5DqwTfSnumM/O1sbdA5sAr2XzPNmkT31dZWM+Uwj+Z/I9cik1L0sTjJje8x68RR9FEmkyF5YU/e4ZH/jW2Jt1cL7zVU58oz3KqvQrS1qehYDpkv8vG/d82zeK2S5lI1eQUK3psPvhwvoRzt7RDzcQeDnivXs1BpxfLWsQ6UkvrEBc2FaTWJvZDb2dpoDceLIWZYVJ3mqgBc1xcpFvjtUq9kzPbiI/oTaNGiBSIjI1lDV6mpqZg2bRorH0eIQYMGYcmSJYiKikKTJk1w/vx5xMTEqBKj7969i0eP2EleiYmJOHr0qNbwl1Lfvn2xatUqLF68GEFBQVizZg3++usvtGvXTuQ7NZ1HmcUXReU09KoVjU8Kc5Yb1+1taGAgt7PFzg/baa2ark7I8FxDAbUhvuhtWM+H+p0yV8FIY3RRm/J5Q6MO0J0M7STmkJraM0/E0px6KrWwep6c29sEeGDf5OILo67hA01ca9W1r+2BLeOln5ZuSsend8a+yR04Z7qZklZ9GxnwNU+xOyF1Vgy94dGnuZ46S3cyTFdt3BDqN2Z1RM5gM7WRrf0Met6p209YP2sGQL++w3+tnt9XeHX7iZ3ELdtibUSfRaOjo/Ho0SNUr14dtWrVQq1atVC9enU8ePCANSNLqIkTJ+LOnTvIzc3FqVOn0LJl8V3VwYMHsW7dOtb+devWBcMweOONN8DnnXfewY0bN/Dy5UucP38eb775puh2mZJ6bQjlYp/Nqus+aQghpM6ELpce6J49oWuoq6GPK5pUq8j7uJAk1/IC7lpHta2J74Y0RTlbGW/3PFcgZm9b/Pozt13S+zpiqFc1/fCPc6zHuILKMQJmk3FRX+5Birs3XXT1MNVWq9rc0l94MMd1F7phTEuTFdkzFUd7O9ZnYC5cn99Ph24ZfLyWEgTiXEbouWjbS5zw/kk37vIfQrXyLw4sNStgA8DUrkVJ4/qKBEptWnhdvQvZ8tG88VL/E/Nzd0QHHRXQhQ6DrxrWjHNmb0ki+ixau3ZtXLx4ETt27MCkSZMwadIk7Ny5E5cuXUKtWtolsYk29VoX6j0j3XQsgMpHbG/CkBD+PBfNAGhEa3belL7y5rrIBQyT7buie+aIskJrn8beuPZld631n1SvxREcqM/uai2wUqxQuvK3uJKKha7SrOmc2tIl9iY+Gesrw/BfZEcsGdAYbzYWviSHvZ2NyZbwsEbGJiJr0qyWrVnCQayOdU2zDIi+CvPNalSU9PWyXuTr30mP4a1qoJW/G+eQ03sdArCofxD2Twk1+nXEEDLTc+nAxvCv7KQzoAHEFTsUuohqt4aWXbNOCqIGivPz81G+fHmcP38eXbt2Rdeupk1cLQuUpeAB/QvRcWnk44o4je5OLhdmd8X5e5loV8sDf/AUGNQ8YQ9q4YtfTxQXvtI3dfHeU+6aNYCwIRt9s8nqqJ1YbW1kqt4zTS4cFx713jGpE/Imdq6Fnw5z34lzzXwSWqJeU/8fj6u+lyZnjJ/63W4Ix2K9tao4G1QyoLlfJWw7Z1x9Jz6L+gfh07+k7d0zhtSLar7U+L3hqzouFFcOmBT0/W7qKgNhCCmGg7+MaMj7mL2dDQa1MH+dKs1hLC79mlVDv2bVMPufBBwWeNxRbf10Pq458600E/WbU65cOVSvXt1stX7KguxXxXcvyllI+qJ5dUKHQlzLl0PHOpVhayPjLfaneUIUuzCm5orr6rjyPzTN68t/EgK084i4qi4D2gmAmqRc6wcAKjjw3+lzFf+TovKv0KKEhlK/qBh7oVXH12snBUtcpHSRemTvZT77vPv+b2elfQGwS0QYSl9sLqQ3WAzNAOj0THF1i0qDq4+E1/xJ46g2rU6zR1vo+nElkejfxJkzZ+Kzzz7Dkyf6o1Oin/ofr39lZ1z6oivWjWoh+PnOBswG+nEYd5G0OkbmNTzTkVws5KTnxzOFUknz0qn+2alXfdaXAGrMUJ4QL/KKP4db6dpTgqUIXkxdSEy99+KrCOFJkfo0U0uQFTP9WKwwC85OUa5FFW7AkLYumiuyx99h15PSF/gL0VmCNZz0xX1CbobEeJ7LHgKTeuixJOBaRZ6PvgK6mgHlN0ZOlbdmos+i33//PW7evAlvb2/UqFFDtQ6X0tmz0t+VlGaalZh19SZwGdW2JvZdSUW4iFW8M3iWnjA2ifp6Gv9diJCTnq7KpoB2D5D6H6qLQznsmtQOdzNecK7vo66OnhwFY73KV8DRHth/LRVR/1w2+ngHE9MQqtE7Z+rkQ/XetboSzoxRD6ykuGDzkTr4EGPTe61xIDENvRtJu8ipvloyl+eE45/zD42aZegi8vzDhW8IrGt9T1RwKGfQkj26pGSzz2clLKdeEmICoKYiy05wpSaILdxrrUQHQJrLYhDj8C0vIZSz3A7/TBQ3xV/XUJUxqrqWx70nRfV2BjX3xaYzxSXq7SRI2tUcilFfQ8nGRoYG3q46h+0OTQvFs1cFoouZiaW8yH8Xe1OS44365bRWLREhi38aw9DZJ/oUqK36rplkL4X5fYNw7GY63mxiuW77yhXkGNhc+qV09OXI2dnaoL+RlX01l3ExBFczZ/aoh7GvL5qv8qVNobjykD15Q+rcK0sJ1lNOQN1ZjRpRXN5uWR3n7mYiQoK/DVPnIJqL6ABo9uzZpmhHmaV5Z28OUo/BK6mf2Ia2qs4KgKQ4Kb3bjn3XITYJl69KqakY2hXfr6kPtqolCoeaaLaOLl1e1wHykHihTvVZcb0k7iEBik7yUlT0Lqs8RC57wYUrUHu3fXHpB6kvnoNbVMe8f6/qfH1r0baWO47dzMCyQY0xb9c13oWgAel6WZQ5j2Lq++hj4iocZlNK3kbJ5eZk/qqoporeAyoXBySNNGoCGZuzMqdPA63cHvXXsybKootcHzNf4ra6qeHsuiYVLZDT4Cy3w7Uvu+HkjC6SHrdQ4BRbYn6f9TDdMh7qQYnUHTS9GhdPxz47i78+nDX4/d1WSJgTjr5Nq+lNFm8goJilEKtHNJfkOOpKSw+Q6ACosLAQS5YsQUhICLy8vODm5sb6IvrxzcIyF1Ml0GoWIHRUG+sXusjntS+7IYhjGibXEFqtKs74ZXQL7PzQeqp8A0WLMZ5OfsJZ/bqGgNwd74rl8e3gJqqfDZ02byyHcraSF1xUL0hJxOPrSX2zifG9abomMRijRxA7H0vqISr1/EK+tdGsidDzr/qq9MYwxTpcvxxLlvyYliD6SjhnzhysWbMGU6ZMweeff46ZM2ciOTkZf//9N6KiokzRxlJHyOKhpqQv2dhQgV4u2D6xrWoapSEXbodytujXzEerKKPQqfvWoO8Px3kfS3osbBmAfVdS9e9UAjX0ccHQltVRTYJSAGWRZwU5HmZpT2Oe8oZx1ZAB4Ozdp/p3MoDm37KuISpDFg1VD9K5Co9aq8fP+Ie/rJ36gt4lmejbu99//x2rV6/GlClTYGdnhyFDhmDNmjWIiorCyZMnTdHGUqdPY+lzH8Qw5eybRtUqqgKgSjyLX+rDdXq0xkDHEO1rC1s/Sn29sssmSlq3BJlMhnl9gwxaZZzwl1Co5GT8MOnZO5lGH4OLcmKEEBfv616Oh0sFuR2qVJDDzcke7hZIKZBCoNpMy5Y13XAhqqtV5TINbyX9hAVrIPpKmJKSgqCgomQqZ2dnZGUV/cL26tULu3ZxryBN2JRdkpYqMGXqInpKhnZ1c/3hW9PJwBgDDJgdZO41iIj16hHEvfyAZpFEoRb2K06MNfQY+khRW0gXGxsZjk3vjJMzuph8jTwpDXp9LnC0t8XWD4oXBO4RVBWuBt48moquStklmejflmrVqqlWaA8ICMDevXsBAKdPn4ZcLv1YY2mk7KXlqhJsDmIWn/xldFFRRl2rB/PRV9iQT1szr7JtLu+0rSm4989D7U42IycPC9RmuZCy6+OwOpzbDZ1xOFjH2oBSmdTF9CuGl7O1MduNnVTmvNkA3w5ugqOfdmblSBpbGoUIJ/o3pm/fvoiNjQUAfPjhh5g1axZq166NESNG4J133pG8gaWRQscsIXMQMhNJqVPdKkhe2FPU8hxKywc3QbcGXtgyvrWo5xmyxpSl9Wykf2HAqN71BR9P/cL07FUB71pjpGzhu8jLJV5fS0ouDqatWl5SOZSzxZtNfEw6E7idgTeT3q9zLgMlLIJqjUT/Zi5cuFD1/aBBg1CjRg0cP34ctWvXRu/evSVtXGmlXIvl7hP+xUNNyVui2QX6VKvkiFXDuZfdKG1WDG6KXRcfSXY8XcNepWQ0kFgZU9Wb4hq+HtXGD+uOJ2ttHxIifQHJkkZMD70+EU0NS7P4471W+OVYsqp4ZWlldGjeqlUrtGrVSoq2lBnrX6+wfvKWZdZTq2KCaZFlnZQnLUB73TN14fUtt8wDKb2kXqNLl/AGXpwBUCt/d7O1wdoMb1UDp25noCdPnpcuNrLi1Ap1hp6Varg74Ys+DQx8dskh+je+evXqGDFiBKKjo5GUlGSKNpV6b9QvqrJrqaEeseuNEfPTtWq6mHV/SOkn1cwnU9UH47oIt6zphq6vz4Pq+JK8y4IvIxpi7+SOBq2V9sdY7k6IBbuvGdusUk10ADR//nw4ODhg0aJFqF27Nnx9fTFs2DCsXr0aN27cMEUbSx3lYpyGjs+awrRw4+uISMnSpQIsTVdeQMID8VOFSemxWSOnLnZKR0mO203EgsrGsrGR4WeOCsWaK5ETYVr6u2Pnh+1wIaora7uupTbEclILzHqXkvOz6N+2YcOG4eeff8b169fx4MEDfP311wCADz74AIGBpiulXpoUvl4NwJLlxP+LLD5p9mnsjQmdalmsLVzEJGqXBGKHHf11LPOxoH8jY5tDSrAWfuyK+xUdjesBWjuqOT4Oq63qmZaC+u97tUrmyTks6xr6uJp0+nwftUVUZ4uY0GHNDOrzfPHiBY4ePYqDBw/iwIEDOHfuHBo2bIjQ0FCJm1c6Kcu1W/JmR334zdkKZ2l0rFtZtSDohdld9ext/ToaMIuOj6EFJgnh0jnQE50DpQt+AGBK1zr49K9LAKCzNk9rf3ecuJUh6WuXdQ28XUxSPDUp7bnqeykWzbUGoq98bdq0wblz51CvXj2EhoZi+vTp6NChAypVqmSK9pVKyqUwpE6cFcvdyR4ZOXnoZ+BMAVNqrnaXa2iNE2tSxUW6E0YJqvZPyqgBwb5wd5IjSM/SFoNa+FIAJLFqlcqbJABKe6a9BEtJJ7oP4tq1a3ByckJgYCACAwNRr149Cn5EKlT2AFl4PvPBaaGI+bg9K9iwFj4VyyN2SkfEfx5m6aZIwpC85bqe3DU4KAma/DOhLQCgsW9FyzaEh42NDGH1PVXL4vCRYhFXwrbncvE6glLW8Vk2qAkAoIVf6bneiw6AMjIysH//frRq1Qp79uxB27Zt4ePjg7fffhurV682RRtLHeUMH0vmAAFFs8ECvVws2gZdAio7w72UdLUaUvNpUIuimii9NIosUvxDGvtWxNlZb+AvkUVGrU1pWeLGWk3pKt3klqbVK+HM52HY9F7J/p1TJzoAkslkaNSoESZNmoQtW7Zg9+7deOONN7B582aMHz/eFG0sdZRDGJYeAiPmcydDfACkLIaYr8ya19hOyjY3J/sStfYVMT+pi1t6OMtL1XVLdA7Q2bNncfDgQRw8eBBHjx7Fs2fPEBQUhA8//BAdO0ozHbO0s5YhMCKtJr4Vcf5eJudj8/sGcW7XRXlxU+/SBoAgH915FYQQAlBZAX1EB0AhISFo2rQpOnbsiLFjx6JDhw5wdaUTshh/xd8HAFy4n2nZhhBJjWhdgzMA6tbAS28yKJcHT19ybqdhA0IIMZ7oAOjJkydwcbHevJGSILegaEhj/7U0C7eESCmvQMG5vbmBSYPrOZYKIIQQIg3R/WMuLi7IzMzEmjVrMGPGDDx5UrSe1dmzZ/HgwQPJG1gaOb6uqBlBMyBKlW/2XefcPkRtZXcxJnWpbUxzCCGE6CC6B+jixYvo0qULKlasiOTkZIwdOxZubm7YunUr7t69i19//dUU7SxVOgdWwc6Lj6x2CisxzONn7LLzt+b3MCphkCrokrKkkQHDxIQYQ3QPUGRkJEaPHo0bN27AwaG4xkOPHj1w+PBhSRtXWimnMVt6GjwxLWNnS1CuDylLqLwDMTfRAdDp06cxbtw4re0+Pj5ISUmRpFGlnbVUgibSaugjbW4c/XqQsuCL3vXh4mCHBf3Ez5QkxBiiAyC5XI7sbO0y29evX0flytLWHCitYi4XBYqZOXkWbgmRUk0P/gVMDcHVQzijOy04TEqXUW1r4nxUVzSk8g6S+Oh17mCIFVb4tzaiA6A+ffpg7ty5yM/PB1DUTX/37l18+umn6N+/v+QNLM34kmZJyST1EhW2HF1AzWqUnjL0hChRb7h0PuxcCxvGhGDt6BaWborVEx0AffPNN3j+/DmqVKmCly9fomPHjqhVqxacnZ0xb948gxqxcuVK+Pn5wcHBAS1btkRcXBzvvqGhoZDJZFpfPXv2VO0zatQorce7detmUNsIEaqixIu2cqUANacAiBCig52tDdrXrgxnueg5TmWO6E/I1dUV+/btw9GjR3Hx4kU8f/4czZo1Q1iYYYtWbtq0CZGRkVi1ahVatmyJ5cuXIzw8HImJiahSpYrW/lu3bkVeXvHQUUZGBho3bowBAwaw9uvWrRt++eUX1c9yeelYU4pYrz6NvfH7qbuSHY9rCIwSowkhRBoGh4jt2rVDu3btVD+fPXsWUVFR2Llzp6jjLF26FGPHjsXo0aMBAKtWrcKuXbuwdu1aTJ8+XWt/Nzf2uObGjRvh6OioFQDJ5XJ4eXmJagshxmjp7y7p8WiWICGEmI6oIbA9e/Zg6tSp+Oyzz3Dr1i0AwLVr1xAREYEWLVpAoeCuhMsnLy8P8fHxrN4jGxsbhIWF4cSJE4KOER0djcGDB8PJyYm1/eDBg6hSpQrq1q2L999/HxkZGbzHyM3NRXZ2NuvLHOxpnRZCCCHEIgRfgaOjo9G9e3esW7cOixYtQqtWrfDbb7+hdevW8PLyQkJCAv79919RL56eno7CwkJ4enqytnt6egqaUh8XF4eEhAS8++67rO3dunXDr7/+itjYWCxatAiHDh1C9+7dUVhYyHmcBQsWwNXVVfXl6+sr6n0Yilb1JrqcTn5i6SYQQkipJXgI7Ntvv8WiRYswbdo0/PXXXxgwYAB++OEHXLp0CdWqVTNlG3lFR0cjKCgIISEhrO2DBw9WfR8UFIRGjRohICAABw8eRJcuXbSOM2PGDERGRqp+zs7ONksQZEc9QESHs3efsn5+o74nz56EEELEEnwFTkpKUuXZ9OvXD3Z2dvj666+NCn48PDxga2uL1NRU1vbU1FS9+Ts5OTnYuHEjxowZo/d1/P394eHhgZs3b3I+LpfL4eLiwvoyh3IUAJU6Pw5thkbVXHFwaqjRxzpyI53188q3mxl9TEIIIUUEX4FfvnwJR0dHAEUzUeRyOapWrWrUi9vb2yM4OBixsbGqbQqFArGxsWjdurXO527evBm5ubkYNmyY3te5f/8+MjIyjG6v1OR2FACVNt2DqmL7xHbw83DSv7MeX77ZgPWzPf2+EEKIZETNAluzZg2cnYuq3RYUFGDdunXw8PBg7TNp0iRRDYiMjMTIkSPRvHlzhISEYPny5cjJyVHNChsxYgR8fHywYMEC1vOio6MREREBd3f2zJvnz59jzpw56N+/P7y8vJCUlIRPPvkEtWrVQnh4uKi2mRrlABFd2temyuqEEGIqggOg6tWrY/Xq1aqfvby8sGHDBtY+MplMdAA0aNAgPH78GFFRUUhJSUGTJk0QExOjSoy+e/cubGzYd76JiYk4evQo9u7dq3U8W1tbXLx4EevXr0dmZia8vb3RtWtXfPnll1ZXC4iqnxJdqMeHEEJMR8YwtAavpuzsbLi6uiIrK8sk+UB+03epvk9e2FPHnqQsS3v2CiHzioeH6XeFEEJ0E3P9pltMC6JlDYguVCeKEEJMh86wZqbe4TaguWXKB5CSgWYJEkKI6dAZ1szuP32p+r6Cg7SLZ5LSxYkWMySEEJOhAMjM0p7lqr7PLxS3dAgpeyo4UBBECCGmQAGQmWW/yld9Tyt7E33oN4QQQkzDoAAoKSkJn3/+OYYMGYK0tDQAwO7du3H58mVJG1calVOb0t+jIa1WT3T7/nX1Z82iiIQQQowjOgA6dOgQgoKCcOrUKWzduhXPnz8HAFy4cAGzZ8+WvIGljfqwF60FRvTpUKcybszrjuGt/SzdFEIIKVVEX4GnT5+Or776Cvv27YO9vb1qe+fOnXHy5ElJG1ca5VHeDxGJZoMRQoj0RJ9ZL126hL59+2ptr1KlCtLT0zmeQdRR4jMhhBBieaIDoIoVK+LRo0da28+dOwcfHx9JGlWa5RVQAEQIIYRYmugAaPDgwfj000+RkpICmUwGhUKBY8eOYerUqRgxYoQp2liqUABECCGEWJ7oAGj+/PkIDAyEr68vnj9/jvr166NDhw5o06YNPv/8c1O0sVShITBCCCHE8kRXWbO3t8fq1asxa9YsJCQk4Pnz52jatClq165tivaVOrnUA0QIIYRYnMFlZqtXr47q1atL2ZYy4UBimqWbQAghhJR5ogOgyMhIzu0ymQwODg6oVasW3nzzTbi5uRnduNIoOf2FpZtACCGElHmiA6Bz587h7NmzKCwsRN26dQEA169fh62tLQIDA/HDDz9gypQpOHr0KOrXry95g0u6Bt4ueJD5Uv+OhBBCCDEZ0UnQb775JsLCwvDw4UPEx8cjPj4e9+/fxxtvvIEhQ4bgwYMH6NChAyZPnmyK9pZ47s72+ncihBBCiEmJDoC+/vprfPnll3BxcVFtc3V1xRdffIHFixfD0dERUVFRiI+Pl7ShpUVBIWPpJhBCCCFlnugAKCsrS7UAqrrHjx8jOzsbQFGxxLy8PONbVwoVKCgAIoQQQizNoCGwd955B9u2bcP9+/dx//59bNu2DWPGjEFERAQAIC4uDnXq1JG6raUCBUCEEEKI5YlOgv7pp58wefJkDB48GAUFBUUHsbPDyJEjsWzZMgBAYGAg1qxZI21LS4lCBdUBIoQQQixNdADk7OyM1atXY9myZbh16xYAwN/fH87Ozqp9mjRpIlkDS5t8ygEihBBCLM7gQojOzs5o1KiRlG0pEwppCIwQQgixOIMCoDNnzuDPP//E3bt3tZKdt27dKknDSivKASKEEEIsT3QS9MaNG9GmTRtcvXoV27ZtQ35+Pi5fvoz9+/fD1dXVFG0sVQpoMVRCCCHE4gxaDX7ZsmXYsWMH7O3t8e233+LatWsYOHAgrQ0mAA2BEUIIIZYnOgBKSkpCz549ARStDJ+TkwOZTIbJkyfj559/lryBpQ0NgRFCCCGWJzoAqlSpEp49ewYA8PHxQUJCAgAgMzMTL17QQp/6UABECCGEWJ7oJOgOHTpg3759CAoKwoABA/DRRx9h//792LdvH7p06WKKNpYqVAeIEEIIsTzRAdD333+PV69eAQBmzpyJcuXK4fjx4+jfvz8+//xzyRtY2tBaYIQQQojliQqACgoKsHPnToSHhwMAbGxsMH36dJM0rLSiJGhCCCHE8kTlANnZ2WH8+PGqHiAiHuUAEUIIIZYnOgk6JCQE58+fN0FTyobnuQWWbgIhhBBS5okOgD744ANERkbi+++/x4kTJ3Dx4kXWlyFWrlwJPz8/ODg4oGXLloiLi+PdNzQ0FDKZTOtLOTVf0/jx4yGTybB8+XKD2ia1x89yLd0EQgghpMwTnQQ9ePBgAMCkSZNU22QyGRiGgUwmQ2Fhoajjbdq0CZGRkVi1ahVatmyJ5cuXIzw8HImJiahSpYrW/lu3bmUtv5GRkYHGjRtjwIABWvtu27YNJ0+ehLe3t6g2EUIIIaR0Ex0A3b59W9IGLF26FGPHjsXo0aMBAKtWrcKuXbuwdu1azgRrNzc31s8bN26Eo6OjVgD04MEDfPjhh9izZw9v7xAhhBBCyibRAVCNGjUke/G8vDzEx8djxowZqm02NjYICwvDiRMnBB0jOjoagwcPhpOTk2qbQqHA8OHDMW3aNDRo0EDvMXJzc5GbWzw0lZ2dLeJdiDO9eyAW7r5msuMTQgghRD/ROUAAsGHDBrRt2xbe3t64c+cOAGD58uX4559/RB0nPT0dhYWF8PT0ZG339PRESkqK3ufHxcUhISEB7777Lmv7okWLYGdnxxqm02XBggVwdXVVffn6+gp/EyL5VCwPAGjl76ZnT0IIIYSYiugA6Mcff0RkZCR69OiBzMxMVc5PxYoVzZ5oHB0djaCgIISEhKi2xcfH49tvv8W6desgk8kEHWfGjBnIyspSfd27d89UTYZyEryNwLYRQgghRHqiA6AVK1Zg9erVmDlzJmxtbVXbmzdvjkuXLok6loeHB2xtbZGamsranpqaCi8vL53PzcnJwcaNGzFmzBjW9iNHjiAtLQ3Vq1eHnZ0d7OzscOfOHUyZMgV+fn6cx5LL5XBxcWF9mQrDFIVAFP8QQgghliM6ALp9+zaaNm2qtV0ulyMnJ0fUsezt7REcHIzY2FjVNoVCgdjYWLRu3Vrnczdv3ozc3FwMGzaMtX348OG4ePEizp8/r/ry9vbGtGnTsGfPHlHtM4XX8Q9koAiIEEIIsRTRSdA1a9bE+fPntZKhY2JiUK9ePdENiIyMxMiRI9G8eXOEhIRg+fLlyMnJUc0KGzFiBHx8fLBgwQLW86KjoxEREQF3d3fWdnd3d61t5cqVg5eXF+rWrSu6fVJjQD1AhBBCiKWJDoAiIyMxYcIEvHr1CgzDIC4uDn/88QcWLFiANWvWiG7AoEGD8PjxY0RFRSElJQVNmjRBTEyMKjH67t27sLFhd1QlJibi6NGj2Lt3r+jXszSGVsIghBBCLE7GMOIvyb///ju++OILJCUlAQC8vb0xZ84crXyckio7Oxuurq7IysqSPB/or/j7mLL5AjrUqYxf3wnR/wRCCCGECCLm+i26BwgAhg4diqFDh+LFixd4/vw5Z8Vmwk0ZbdIIGCGEEGI5opOgv/rqK1U1aEdHRwp+RKJZYIQQQojliQ6ANm/ejFq1aqFNmzb44YcfkJ6ebop2lVqvChQAgJSsVxZuCSGEEFJ2iQ6ALly4gIsXLyI0NBRLliyBt7c3evbsif/973948eKFKdpYqqw7VtR7di3lmYVbQgghhJRdBi2F0aBBA8yfPx+3bt3CgQMH4Ofnh48//lhv8UICJD0WVyuJEEIIIdIzKABS5+TkhPLly8Pe3h75+flStIkQQgghxKQMCoBu376NefPmoUGDBmjevDnOnTuHOXPmCFrAlBBCCCHE0kRPg2/VqhVOnz6NRo0aYfTo0RgyZAh8fHxM0bZSqVsDL8RcpkCREEIIsSTRAVCXLl2wdu1a1K9fn7VdoVDg33//Ra9evSRrXGmU8DDL0k0ghBBCyjzRAdC8efNYP9+8eRNr167FunXr8PjxY8oD0uP+05eWbgIhhBBS5hmUA/Ty5Uv8+uuv6NChA+rWrYvjx48jKioK9+/fl7p9hBBCCCGSE9UDdPr0aaxZswYbN25EQEAAhg4diuPHj+OHH37QGhIjhBBCCLFWggOgRo0aITs7G2+//TaOHz+OBg0aAACmT59ussYRQgghhJiC4CGwxMREdOjQAZ06daLeHkIIIYSUaIIDoFu3bqFu3bp4//33Ua1aNUydOhXnzp2DjFb1JIQQQkgJIzgA8vHxwcyZM3Hz5k1s2LABKSkpaNu2LQoKCrBu3Tpcv37dlO0khBBCCJGMQbPAOnfujN9++w2PHj3C999/j/379yMwMBCNGjWSun2EEEIIIZIzai0wV1dXfPDBBzhz5gzOnj2L0NBQiZpFCCGEEGI6Ri+GqtSkSRN89913Uh2OEEIIIcRkJAuACCGEEEJKCgqACCGEEFLmUABkZp3qVgYAfNYj0MItIYQQQsouCoDMzNG+qPi2QzlbC7eEEEIIKbtErwbPl+gsk8ng4OCAWrVqoUOHDrC1pQs8FwXDAAAVkCSEEEIsSHQAtGzZMjx+/BgvXrxApUqVAABPnz6Fo6MjnJ2dkZaWBn9/fxw4cAC+vr6SN7ikUwZANhT/EEIIIRYjeghs/vz5aNGiBW7cuIGMjAxkZGTg+vXraNmyJb799lvcvXsXXl5emDx5sinaW+IpiuIf2FAPECGEEGIxonuAPv/8c/z1118ICAhQbatVqxaWLFmC/v3749atW1i8eDH69+8vaUNLC4Z6gAghhBCLE90D9OjRIxQUFGhtLygoQEpKCgDA29sbz549M751pZCyB4hygAghhBDLER0AderUCePGjcO5c+dU286dO4f3338fnTt3BgBcunQJNWvWlK6VpYgqCdrC7SCEEELKMtEBUHR0NNzc3BAcHAy5XA65XI7mzZvDzc0N0dHRAABnZ2d88803kje2NGAoB4gQQgixONE5QF5eXti3bx+uXbuG69evAwDq1q2LunXrqvbp1KmTdC0sZQpfj4HZUAUmQgghxGJEB0BKgYGBCAykasZiHb2ZDgA4desJ+jatZuHWEEIIIWWT6ACosLAQ69atQ2xsLNLS0qBQKFiP79+/X7LGlWYbT9/Dwv6NLN0MQgghpEwSPRDz0Ucf4aOPPkJhYSEaNmyIxo0bs74MsXLlSvj5+cHBwQEtW7ZEXFwc776hoaGQyWRaXz179lTt88UXXyAwMBBOTk6oVKkSwsLCcOrUKYPaZirDW9WwdBMIIYSQMkt0D9DGjRvx559/okePHpI0YNOmTYiMjMSqVavQsmVLLF++HOHh4UhMTESVKlW09t+6dSvy8vJUP2dkZKBx48YYMGCAaludOnXw/fffw9/fHy9fvsSyZcvQtWtX3Lx5E5UrV5ak3YZq6OOChAfZ6FDHsu0ghBBCyjLRPUD29vaoVauWZA1YunQpxo4di9GjR6N+/fpYtWoVHB0dsXbtWs793dzc4OXlpfrat28fHB0dWQHQ22+/jbCwMPj7+6NBgwZYunQpsrOzcfHiRcnabSjb17O/qBAiIYQQYjmiA6ApU6bg22+/VVU0NkZeXh7i4+MRFhZW3CAbG4SFheHEiROCjhEdHY3BgwfDycmJ9zV+/vlnuLq6GjxEJyXlp0az4AkhhBDLET0EdvToURw4cAC7d+9GgwYNUK5cOdbjW7duFXys9PR0FBYWwtPTk7Xd09MT165d0/v8uLg4JCQkqOoPqdu5cycGDx6MFy9eoGrVqti3bx88PDw4j5Obm4vc3FzVz9nZ2YLfg1i0GjwhhBBieaIDoIoVK6Jv376maIto0dHRCAoKQkhIiNZjnTp1wvnz55Geno7Vq1dj4MCBOHXqFGde0YIFCzBnzhxzNBnKSXNUCJEQQgixHNEB0C+//CLZi3t4eMDW1hapqams7ampqfDy8tL53JycHGzcuBFz587lfNzJyQm1atVCrVq10KpVK9SuXRvR0dGYMWOG1r4zZsxAZGSk6ufs7Gz4+voa8I70Uw2BmeTohBBCCBHCovWI7e3tERwcjNjYWNU2hUKB2NhYtG7dWudzN2/ejNzcXAwbNkzQaykUCtYwlzq5XA4XFxfWl6kUrwZPIRAhhBBiKYJ6gJo1a4bY2FhUqlQJTZs21Zm/cvbsWVENiIyMxMiRI9G8eXOEhIRg+fLlyMnJwejRowEAI0aMgI+PDxYsWMB6XnR0NCIiIuDu7s7anpOTg3nz5qFPnz6oWrUq0tPTsXLlSjx48IA1U8xSFKoAyMINIYQQQsowQQHQm2++CblcDgCIiIiQtAGDBg3C48ePERUVhZSUFDRp0gQxMTGqxOi7d+/CRmPhrMTERBw9ehR79+7VOp6trS2uXbuG9evXIz09He7u7mjRogWOHDmCBg0aSNp2QzA0BkYIIYRYnIyRYj57KZOdnQ1XV1dkZWVJPhzW5ZuDSHqcg43vtUIrf3f9TyCEEEKIIGKu3wYvhpqXl8e5Flj16tUNPWSZoAw3KQeIEEIIsRzRAdD169cxZswYHD9+nLWdYRjIZDIUFhZK1rjSSNndRjlAhBBCiOWIDoBGjx4NOzs77Ny5E1WrVqWCfiIVF0K0cEMIIYSQMkx0AHT+/HnEx8cjMDDQFO0p9agSNCGEEGJ5ousA1a9fH+np6aZoS5lAOUCEEEKI5YkOgBYtWoRPPvkEBw8eREZGBrKzs1lfRDdlAEThDyGEEGI5oofAlCu3d+nShbWdkqCFeZD5EgDlABFCCCGWJDoAOnDggCnaUeZcuJeJRtUqWroZhBBCSJkkOgDq2LGjKdpR5jStXsnSTSCEEELKLIMKIWZmZiIuLo6zEOKIESMkaVhp5eFsj/TneShna9F1aAkhhJAyTXQAtGPHDgwdOhTPnz+Hi4sLazq3TCajAEgPhWoWmGXbQQghhJRlorshpkyZgnfeeQfPnz9HZmYmnj59qvp68uSJKdpYqlAdIEIIIcTyRAdADx48wKRJk+Do6GiK9pR6itddQNQDRAghhFiO6AAoPDwcZ86cMUVbygQqhEgIIYRYnugcoJ49e2LatGm4cuUKgoKCUK5cOdbjffr0kaxxpZFyCIwCIEIIIcRyRAdAY8eOBQDMnTtX6zEqhKifMgma4h9CCCHEckQHQJrT3ok4qh4gSgIihBBCLIaK0ZhZ8RCYhRtCCCGElGGie4C4hr7URUVFGdyYskBBSdCEEEKIxYkOgLZt28b6OT8/H7dv34adnR0CAgIoANKDkqAJIYQQyxMdAJ07d05rW3Z2NkaNGoW+fftK0qjSimEYtWnwlm0LIYQQUpZJkgPk4uKCOXPmYNasWVIcrtRSBj8A9QARQgghliRZEnRWVhaysrKkOlyppFCPgAghhBBiMaKHwL777jvWzwzD4NGjR9iwYQO6d+8uWcNKI4V6DxCNgRFCCCEWIzoAWrZsGetnGxsbVK5cGSNHjsSMGTMka1hpxKA4AqIRMEIIIcRyRAdAt2/f5n3s5cuXRjWmtFMfAaP4hxBCCLEcSXKAcnNzsXTpUtSsWVOKw5UJMuoCIoQQQixGcACUm5uLGTNmoHnz5mjTpg3+/vtvAMDatWtRs2ZNLFu2DJMnTzZVO0sF6gEihBBCrIPgIbCoqCj89NNPCAsLw/HjxzFgwACMHj0aJ0+exNKlSzFgwADY2tqasq0lHuUAEUIIIdZBcAC0efNm/Prrr+jTpw8SEhLQqFEjFBQU4MKFCzScIxDNgieEEEKsg+AhsPv37yM4OBgA0LBhQ8jlckyePJmCHxHU4x8ZDYIRQgghFiM4ACosLIS9vb3qZzs7Ozg7O5ukUaUVw9AQGCGEEGINBA+BMQyDUaNGQS6XAwBevXqF8ePHw8nJibXf1q1bpW1hKUIjYIQQQoh1EBwAjRw5kvXzsGHDJG9MaceaBUY9QIQQQojFCA6AfvnlF5M1YuXKlfj666+RkpKCxo0bY8WKFQgJCeHcNzQ0FIcOHdLa3qNHD+zatQv5+fn4/PPP8e+//+LWrVtwdXVFWFgYFi5cCG9vb5O9B0FY0+ApAiKEEEIsRbLFUA21adMmREZGYvbs2Th79iwaN26M8PBwpKWlce6/detWPHr0SPWVkJAAW1tbDBgwAADw4sULnD17FrNmzcLZs2exdetWJCYmok+fPuZ8W5xoGjwhhBBiHWQMY9nJ2S1btkSLFi3w/fffAwAUCgV8fX3x4YcfYvr06Xqfv3z5ckRFReHRo0da+UhKp0+fRkhICO7cuYPq1avrPWZ2djZcXV2RlZUFFxcXcW9Ih6c5eWj65T4AwM153WFna/H4kxBCCCk1xFy/LXoFzsvLQ3x8PMLCwlTbbGxsEBYWhhMnTgg6RnR0NAYPHswb/ABAVlYWZDIZKlasyPl4bm4usrOzWV+mwJoGT11AhBBCiMVYNABKT09HYWEhPD09Wds9PT2RkpKi9/lxcXFISEjAu+++y7vPq1ev8Omnn2LIkCG80eCCBQvg6uqq+vL19RX3RgRiTYM3ySsQQgghRIgSPQYTHR2NoKAg3oTp/Px8DBw4EAzD4Mcff+Q9zowZM5CVlaX6unfvnknay+4BMslLEEIIIUQAwbPATMHDwwO2trZITU1lbU9NTYWXl5fO5+bk5GDjxo2YO3cu5+PK4OfOnTvYv3+/zrFAuVyuqm9kSuxp8BQBEUIIIZZi0R4ge3t7BAcHIzY2VrVNoVAgNjYWrVu31vnczZs3Izc3l7MekTL4uXHjBv777z+4u7tL3nZDMFQKkRBCCLEKFu0BAoDIyEiMHDkSzZs3R0hICJYvX46cnByMHj0aADBixAj4+PhgwYIFrOdFR0cjIiJCK7jJz8/HW2+9hbNnz2Lnzp0oLCxU5RO5ubmxlvMwu9fxD3X+EEIIIZZl8QBo0KBBePz4MaKiopCSkoImTZogJiZGlRh99+5d2NiwO6oSExNx9OhR7N27V+t4Dx48wPbt2wEATZo0YT124MABhIaGmuR9CKHs/6FV4QkhhBDLsngdIGtkqjpA8Xeeov+PxwEAyQt7SnZcQgghhJSgOkBlDcWahBBCiHWgAMiMPF0cLN0EQgghhIACIIsoX87W0k0ghBBCyjQKgMxI8XoIzIZmgRFCCCEWRQGQGTGqafAUARFCCCGWRAGQGSl7gCj+IYQQQiyLAiAzUih7gCzbDEIIIaTMowDIrF7nAFESECGEEGJRFACZkbIHyIbGwAghhBCLogDIjBgaAiOEEEKsAgVAZlScBE0hECGEEGJJFACZEUOrwRNCCCFWgQIgM6JCiIQQQoh1oADIAigJmhBCCLEsCoDMSJUDZOF2EEIIIWUdBUBmpKClMAghhBCrQAGQGTG0FAYhhBBiFSgAMiMqhEgIIYRYBwqAzIpmgRFCCCHWgAIgM6IcIEIIIcQ6UABkRi/yCgEUzwYjhBBCiGVQAGRGcruij/tOxgsLt4QQQggp2ygAMiNlx0+tKs6WbQghhBBSxlEAZEYMqBAiIYQQYg0oADInWgyVEEIIsQoUAJmRMvVZRn1AhBBCiEVRAGRGDPUAEUIIIVaBAiAzYkDT3wkhhBBrQAGQGTFUCJEQQgixChQAmVFxDhAhhBBCLIkCIDOi1eAJIYQQ60ABkBmpeoAoACKEEEIsigIgc1LmANEgGCGEEGJRFg+AVq5cCT8/Pzg4OKBly5aIi4vj3Tc0NBQymUzrq2fPnqp9tm7diq5du8Ld3R0ymQznz583w7sQRlUJmuIfQgghxKIsGgBt2rQJkZGRmD17Ns6ePYvGjRsjPDwcaWlpnPtv3boVjx49Un0lJCTA1tYWAwYMUO2Tk5ODdu3aYdGiReZ6G4IpFEX/UvxDCCGEWJadJV986dKlGDt2LEaPHg0AWLVqFXbt2oW1a9di+vTpWvu7ubmxft64cSMcHR1ZAdDw4cMBAMnJyaZruIFUVYCoC4gQQgixKIv1AOXl5SE+Ph5hYWHFjbGxQVhYGE6cOCHoGNHR0Rg8eDCcnJyMaktubi6ys7NZX6agmgVmkqMTQgghRCiLBUDp6ekoLCyEp6cna7unpydSUlL0Pj8uLg4JCQl49913jW7LggUL4Orqqvry9fU1+phcaBYYIYQQYh0sngRtqOjoaAQFBSEkJMToY82YMQNZWVmqr3v37knQQm2qStAmOTohhBBChLJYDpCHhwdsbW2RmprK2p6amgovLy+dz83JycHGjRsxd+5cSdoil8shl8slOZYu2a/yAQBn72aa/LUIIYQQws9iPUD29vYIDg5GbGysaptCoUBsbCxat26t87mbN29Gbm4uhg0bZupmSmrZvuuWbgIhhBBCYOFZYJGRkRg5ciSaN2+OkJAQLF++HDk5OapZYSNGjICPjw8WLFjAel50dDQiIiLg7u6udcwnT57g7t27ePjwIQAgMTERAODl5aW3Z8nUHmW9sujrE0IIIaSIRQOgQYMG4fHjx4iKikJKSgqaNGmCmJgYVWL03bt3YWPD7qRKTEzE0aNHsXfvXs5jbt++XRVAAcDgwYMBALNnz8YXX3xhmjciULPqFWn4ixBCCLECMkY5N5uoZGdnw9XVFVlZWXBxcZHsuH/F38eUzRfg7eqA4zO6SHZcQgghhIi7fpfYWWAlka1N0fwv/8rOFm4JIYQQUrZRAGRGtBYYIYQQYh0oADIjGmwkhBBCrAMFQGakDIBsqAuIEEIIsSgKgMxIwdAQGCGEEGINKAAyI9VaYBZtBSGEEEIoADIn5Vpg1AVECCGEWBQFQGakmgVm4XYQQgghZR0FQGakWg2eIiBCCCHEoigAMiNVDhBFQIQQQohFUQBkRqpZYBZuByGEEFLWUQBkRjQERgghhFgHCoDMqHgaPEVAhBBCiCVRAGROVAiREEIIsQoUAJlRcRK0RZtBCCGElHkUAJkRQ4UQCSGEEKtAAZAZ0SwwQgghxDpQAGRG1ANECCGEWAcKgMyIFkMlhBBCrAMFQGbE0CwwQgghxCpQAGQBNhQBEUIIIRZFAZAZURI0IYQQYh0oADIjhpKACCGEEKtAAZAZ0VIYhBBCiHWgAMiMaDFUQgghxDpQAGRGDCgHiBBCCLEGFACZkZ2NDHI7G5Szo4+dEEIIsSQZw6hSc8lr2dnZcHV1RVZWFlxcXCzdHEIIIYQIIOb6TV0RhBBCCClzKAAihBBCSJlDARAhhBBCyhwKgAghhBBS5lAARAghhJAyxyoCoJUrV8LPzw8ODg5o2bIl4uLiePcNDQ2FTCbT+urZs6dqH4ZhEBUVhapVq6J8+fIICwvDjRs3zPFWCCGEEFICWDwA2rRpEyIjIzF79mycPXsWjRs3Rnh4ONLS0jj337p1Kx49eqT6SkhIgK2tLQYMGKDaZ/Hixfjuu++watUqnDp1Ck5OTggPD8erV6/M9bYIIYQQYsUsXgeoZcuWaNGiBb7//nsAgEKhgK+vLz788ENMnz5d7/OXL1+OqKgoPHr0CE5OTmAYBt7e3pgyZQqmTp0KAMjKyoKnpyfWrVuHwYMH6z0m1QEihBBCSp4SUwcoLy8P8fHxCAsLU22zsbFBWFgYTpw4IegY0dHRGDx4MJycnAAAt2/fRkpKCuuYrq6uaNmyJe8xc3NzkZ2dzfoihBBCSOll0QAoPT0dhYWF8PT0ZG339PRESkqK3ufHxcUhISEB7777rmqb8nlijrlgwQK4urqqvnx9fcW+FUIIIYSUIBbPATJGdHQ0goKCEBISYtRxZsyYgaysLNXXvXv3JGohIYQQQqyRRQMgDw8P2NraIjU1lbU9NTUVXl5eOp+bk5ODjRs3YsyYMaztyueJOaZcLoeLiwvrixBCCCGll0UDIHt7ewQHByM2Nla1TaFQIDY2Fq1bt9b53M2bNyM3NxfDhg1jba9Zsya8vLxYx8zOzsapU6f0HpMQQgghZYOdpRsQGRmJkSNHonnz5ggJCcHy5cuRk5OD0aNHAwBGjBgBHx8fLFiwgPW86OhoREREwN3dnbVdJpPh448/xldffYXatWujZs2amDVrFry9vREREWGut0UIIYQQK2bxAGjQoEF4/PgxoqKikJKSgiZNmiAmJkaVxHz37l3Y2LA7qhITE3H06FHs3buX85iffPIJcnJy8N577yEzMxPt2rVDTEwMHBwcBLVJWRmAZoMRQgghJYfyui2kwo/F6wBZo/v379NMMEIIIaSEunfvHqpVq6ZzHwqAOCgUCjx8+BAVKlSATCaT9NjZ2dnw9fXFvXv3KNnaxOizNh/6rM2HPmvzoc/afKT6rBmGwbNnz+Dt7a01eqTJ4kNg1sjGxkZv5Ggsmm1mPvRZmw991uZDn7X50GdtPlJ81q6uroL2K9F1gAghhBBCDEEBECGEEELKHAqAzEwul2P27NmQy+WWbkqpR5+1+dBnbT70WZsPfdbmY4nPmpKgCSGEEFLmUA8QIYQQQsocCoAIIYQQUuZQAEQIIYSQMocCIEIIIYSUORQAmcDKlSvh5+cHBwcHtGzZEnFxcTr337x5MwIDA+Hg4ICgoCD8+++/ZmppySfms169ejXat2+PSpUqoVKlSggLC9P7f0OKif29Vtq4cSNkMhktRiyC2M86MzMTEyZMQNWqVSGXy1GnTh06jwgk9rNevnw56tati/Lly8PX1xeTJ0/Gq1evzNTakuvw4cPo3bs3vL29IZPJ8Pfff+t9zsGDB9GsWTPI5XLUqlUL69atk7ZRDJHUxo0bGXt7e2bt2rXM5cuXmbFjxzIVK1ZkUlNTOfc/duwYY2tryyxevJi5cuUK8/nnnzPlypVjLl26ZOaWlzxiP+u3336bWblyJXPu3Dnm6tWrzKhRoxhXV1fm/v37Zm55ySP2s1a6ffs24+Pjw7Rv35558803zdPYEk7sZ52bm8s0b96c6dGjB3P06FHm9u3bzMGDB5nz58+bueUlj9jP+vfff2fkcjnz+++/M7dv32b27NnDVK1alZk8ebKZW17y/Pvvv8zMmTOZrVu3MgCYbdu26dz/1q1bjKOjIxMZGclcuXKFWbFiBWNra8vExMRI1iYKgCQWEhLCTJgwQfVzYWEh4+3tzSxYsIBz/4EDBzI9e/ZkbWvZsiUzbtw4k7azNBD7WWsqKChgKlSowKxfv95UTSw1DPmsCwoKmDZt2jBr1qxhRo4cSQGQQGI/6x9//JHx9/dn8vLyzNXEUkPsZz1hwgSmc+fOrG2RkZFM27ZtTdrO0kZIAPTJJ58wDRo0YG0bNGgQEx4eLlk7aAhMQnl5eYiPj0dYWJhqm42NDcLCwnDixAnO55w4cYK1PwCEh4fz7k+KGPJZa3rx4gXy8/Ph5uZmqmaWCoZ+1nPnzkWVKlUwZswYczSzVDDks96+fTtat26NCRMmwNPTEw0bNsT8+fNRWFhormaXSIZ81m3atEF8fLxqmOzWrVv4999/0aNHD7O0uSwxx7WRFkOVUHp6OgoLC+Hp6cna7unpiWvXrnE+JyUlhXP/lJQUk7WzNDDks9b06aefwtvbW+uPjLAZ8lkfPXoU0dHROH/+vBlaWHoY8lnfunUL+/fvx9ChQ/Hvv//i5s2b+OCDD5Cfn4/Zs2ebo9klkiGf9dtvv4309HS0a9cODMOgoKAA48ePx2effWaOJpcpfNfG7OxsvHz5EuXLlzf6NagHiJRJCxcuxMaNG7Ft2zY4ODhYujmlyrNnzzB8+HCsXr0aHh4elm5OqadQKFClShX8/PPPCA4OxqBBgzBz5kysWrXK0k0rdQ4ePIj58+fjhx9+wNmzZ7F161bs2rULX375paWbRgxAPUAS8vDwgK2tLVJTU1nbU1NT4eXlxfkcLy8vUfuTIoZ81kpLlizBwoUL8d9//6FRo0ambGapIPazTkpKQnJyMnr37q3aplAoAAB2dnZITExEQECAaRtdQhnye121alWUK1cOtra2qm316tVDSkoK8vLyYG9vb9I2l1SGfNazZs3C8OHD8e677wIAgoKCkJOTg/feew8zZ86EjQ31KUiF79ro4uIiSe8PQD1AkrK3t0dwcDBiY2NV2xQKBWJjY9G6dWvO57Ru3Zq1PwDs27ePd39SxJDPGgAWL16ML7/8EjExMWjevLk5mlriif2sAwMDcenSJZw/f1711adPH3Tq1Annz5+Hr6+vOZtfohjye922bVvcvHlTFWQCwPXr11G1alUKfnQw5LN+8eKFVpCjDDwZWlZTUma5NkqWTk0YhimaVimXy5l169YxV65cYd577z2mYsWKTEpKCsMwDDN8+HBm+vTpqv2PHTvG2NnZMUuWLGGuXr3KzJ49m6bBCyT2s164cCFjb2/PbNmyhXn06JHq69mzZ5Z6CyWG2M9aE80CE07sZ3337l2mQoUKzMSJE5nExERm586dTJUqVZivvvrKUm+hxBD7Wc+ePZupUKEC88cffzC3bt1i9u7dywQEBDADBw601FsoMZ49e8acO3eOOXfuHAOAWbp0KXPu3Dnmzp07DMMwzPTp05nhw4er9ldOg582bRpz9epVZuXKlTQNviRYsWIFU716dcbe3p4JCQlhTp48qXqsY8eOzMiRI1n7//nnn0ydOnUYe3t7pkGDBsyuXbvM3OKSS8xnXaNGDQaA1tfs2bPN3/ASSOzvtToKgMQR+1kfP36cadmyJSOXyxl/f39m3rx5TEFBgZlbXTKJ+azz8/OZL774ggkICGAcHBwYX19f5oMPPmCePn1q/oaXMAcOHOA8/yo/35EjRzIdO3bUek6TJk0Ye3t7xt/fn/nll18kbZOMYajfjhBCCCFlC+UAEUIIIaTMoQCIEEIIIWUOBUCEEEIIKXMoACKEEEJImUMBECGEEELKHAqACCGEEFLmUABECCGEkDKHAiBCCJFQcnIyZDIZzp8/b+mmEGJ1Dh8+jN69e8Pb2xsymQx///236GMwDIMlS5agTp06kMvl8PHxwbx580QfhwIgQohRHj9+jPfffx/Vq1eHXC6Hl5cXwsPDcezYMdU+hp7oDDFq1CjIZDIsXLiQtf3vv/+GTCYzSxsIIdxycnLQuHFjrFy50uBjfPTRR1izZg2WLFmCa9euYfv27QgJCRF9HFoNnhBilP79+yMvLw/r16+Hv78/UlNTERsbi4yMDIu1ycHBAYsWLcK4ceNQqVIli7VDSrSyOykNunfvju7du/M+npubi5kzZ+KPP/5AZmYmGjZsiEWLFiE0NBQAcPXqVfz4449ISEhA3bp1AQA1a9Y0qC3UA0QIMVhmZiaOHDmCRYsWoVOnTqhRowZCQkIwY8YM9OnTBwDg5+cHAOjbty9kMpnqZwD4559/0KxZMzg4OMDf3x9z5sxBQUGB6nGZTIYff/wR3bt3R/ny5eHv748tW7bobVdYWBi8vLywYMEC3n2++OILNGnShLVt+fLlrPaNGjUKERERmD9/Pjw9PVGxYkXMnTsXBQUFmDZtGtzc3FCtWjX88ssvWse/du0a2rRpAwcHBzRs2BCHDh1iPZ6QkIDu3bvD2dkZnp6eGD58ONLT01WPh4aGYuLEifj444/h4eGB8PBwve+bkJJu4sSJOHHiBDZu3IiLFy9iwIAB6NatG27cuAEA2LFjB/z9/bFz507UrFkTfn5+ePfdd/HkyRPRr0UBECHEYM7OznB2dsbff/+N3Nxczn1Onz4NAPjll1/w6NEj1c9HjhzBiBEj8NFHH+HKlSv46aefsG7dOq2x/FmzZqF///64cOEChg4disGDB+Pq1as622Vra4v58+djxYoVuH//vlHvcf/+/Xj48CEOHz6MpUuXYvbs2ejVqxcqVaqEU6dOYfz48Rg3bpzW60ybNg1TpkzBuXPn0Lp1a/Tu3VvVK5aZmYnOnTujadOmOHPmDGJiYpCamoqBAweyjrF+/XrY29vj2LFjWLVqlVHvgxBrd/fuXfzyyy/YvHkz2rdvj4CAAEydOhXt2rVT3WTcunULd+7cwebNm/Hrr79i3bp1iI+Px1tvvSX+BSVdWpUQUuZs2bKFqVSpEuPg4MC0adOGmTFjBnPhwgXWPgCYbdu2sbZ16dKFmT9/Pmvbhg0bmKpVq7KeN378eNY+LVu2ZN5//33e9qivPN+qVSvmnXfeYRiGYbZt28aon/Jmz57NNG7cmPXcZcuWMTVq1GAdq0aNGkxhYaFqW926dZn27durfi4oKGCcnJyYP/74g2EYhrl9+zYDgFm4cKFqn/z8fKZatWrMokWLGIZhmC+//JLp2rUr67Xv3bvHAGASExMZhilaibxp06a875OQkk7zvLBz504GAOPk5MT6srOzYwYOHMgwDMOMHTuW9XfCMAwTHx/PAGCuXbsm6vUpB4gQYpT+/fujZ8+eOHLkCE6ePIndu3dj8eLFWLNmDUaNGsX7vAsXLuDYsWOsHp/CwkK8evUKL168gKOjIwCgdevWrOe1bt1a8AyrRYsWoXPnzpg6daro96XUoEED2NgUd5Z7enqiYcOGqp9tbW3h7u6OtLQ0rXYq2dnZoXnz5qqeqwsXLuDAgQNwdnbWer2kpCTUqVMHABAcHGxwuwkpaZ4/fw5bW1vEx8fD1taW9Zjyb6Vq1aqws7NT/Y0AQL169QAU9SAp84KEoACIEGI0BwcHvPHGG3jjjTcwa9YsvPvuu5g9e7bOAOj58+eYM2cO+vXrx3k8KXTo0AHh4eGYMWOGVltsbGxQdBNaLD8/X+sY5cqVY/0sk8k4tykUCsHtev78OXr37o1FixZpPVa1alXV905OToKPSUhJ17RpUxQWFiItLQ3t27fn3Kdt27YoKChAUlISAgICAADXr18HANSoUUPU61EARAiRXP369VnT3suVK4fCwkLWPs2aNUNiYiJq1aql81gnT57EiBEjWD83bdpUcFsWLlyIJk2aaN0ZVq5cGSkpKWAYRjU9XsraPSdPnkSHDh0AAAUFBYiPj8fEiRMBFL33v/76C35+frCzo9MwKTueP3+Omzdvqn6+ffs2zp8/Dzc3N9SpUwdDhw7FiBEj8M0336Bp06Z4/PgxYmNj0ahRI/Ts2RNhYWFo1qwZ3nnnHSxfvhwKhQITJkzAG2+8weoVEoKSoAkhBsvIyEDnzp3x22+/4eLFi7h9+zY2b96MxYsX480331Tt5+fnh9jYWKSkpODp06cAgKioKPz666+YM2cOLl++jKtXr2Ljxo34/PPPWa+xefNmrF27FtevX8fs2bMRFxenCiSECAoKwtChQ/Hdd9+xtoeGhuLx48dYvHgxkpKSsHLlSuzevduIT4Nt5cqV2LZtG65du4YJEybg6dOneOeddwAAEyZMwJMnTzBkyBCcPn0aSUlJ2LNnD0aPHq0VKBJSmpw5cwZNmzZV3cRERkaiadOmiIqKAlA0WWLEiBGYMmUK6tati4iICJw+fRrVq1cHUNRzu2PHDnh4eKBDhw7o2bMn6tWrh40bN4pvjAR5TISQMurVq1fM9OnTmWbNmjGurq6Mo6MjU7duXebzzz9nXrx4odpv+/btTK1atRg7OztWknFMTAzTpk0bpnz58oyLiwsTEhLC/Pzzz6rHATArV65k3njjDUYulzN+fn7Mpk2bdLZJPQla6fbt24y9vT2jecr78ccfGV9fX8bJyYkZMWIEM2/ePK0kaM1jdezYkfnoo49Y22rUqMEsW7ZM9VoAmP/9739MSEgIY29vz9SvX5/Zv38/6znXr19n+vbty1SsWJEpX748ExgYyHz88ceMQqHgfR1CiHRkDKMxCE4IIVZCJpNh27ZtiIiIsHRTCCGlDA2BEUIIIaTMoQCIEEIIIWUOTT8ghFgtGqEnhJgK9QARQgghpMyhAIgQQgghZQ4FQIQQQggpcygAIoQQQkiZQwEQIYQQQsocCoAIIYQQUuZQAEQIIYSQMocCIEIIIYSUORQAEUIIIaTM+T/vU13I7a7HRQAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Plotting running average reward for BanditTenArmedRandomFixed-v0\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACRv0lEQVR4nO3dd3gUVRcG8HfTGwmdUAKB0HsNHQIEIyAIKl2aiqAgSpRPkBJAqSJgQVCqYqEJinSkV4HQpEPoJSG0BBJI2/n+CLtsmd2d2Tab5P09Tx7IZHb27mYzc+bec89VCYIggIiIiCgPcVO6AURERETOxgCIiIiI8hwGQERERJTnMAAiIiKiPIcBEBEREeU5DICIiIgoz2EARERERHkOAyAiIiLKcxgAERERUZ7DAIjIjlQqFcaPH690MygP27lzJ1QqFXbu3Kl0UxSzZMkSqFQqXL16VemmkAtjAEQuT3My03x5eHigZMmS6N+/P27duqV081zO2bNnoVKp4OPjg0ePHindHJfSv39/vc+St7c3KlasiHHjxuHZs2dKN08Rp0+fxptvvomSJUvC29sbJUqUQO/evXH69Gmlm6YnIiJC73dn6os3ICSVh9INIJJq4sSJKFu2LJ49e4aDBw9iyZIl2Lt3L06dOgUfHx+lmwcAePr0KTw8lP2z+uWXXxAcHIyHDx9i1apVeOeddxRtj6vx9vbGggULAABJSUn466+/8PnnnyMuLg6//vqrwq1zrtWrV6Nnz54oWLAg3n77bZQtWxZXr17FwoULsWrVKixbtgxdunRRupkAgNGjR+t9lg8fPoxvvvkGn332GapUqaLdXrNmTVSrVg09evSAt7e3Ek2lnEIgcnGLFy8WAAiHDx/W2/7pp58KAITly5cr1DLXo1arhdDQUCE6Olro0qWLEBERoUg7UlJSFHleS/r16yf4+/vrbVOr1UKjRo0ElUolxMfHK9Qy6dRqtZCammry5zt27BAACDt27DB7nEuXLgl+fn5C5cqVhbt37+r9LDExUahcubLg7+8vxMXF2aPZkj158kTSfitXrpT0OolM4RAY5VjNmzcHAMTFxWm3RUREICIiwmjf/v37IzQ0VPv91atXoVKpMGPGDPz4448ICwuDt7c3GjRogMOHDxs9NiAgALdu3ULnzp0REBCAIkWK4JNPPkFWVpbevoZd8OPHj4dKpcKlS5fQv39/5M+fH0FBQRgwYABSU1P1Hvv06VMMGzYMhQsXRr58+dCpUyfcunVLVrf+vn37cPXqVfTo0QM9evTA7t27cfPmTe3PX3nlFZQrV070sY0bN0b9+vX1tv3yyy+oV68efH19UbBgQfTo0QM3btzQ2yciIgLVq1dHbGwsWrRoAT8/P3z22WcAgL/++gsdOnRAiRIl4O3tjbCwMHz++edG7xsAzJkzB+XKlYOvry/Cw8OxZ88e0d9nWloaYmJiUL58eXh7eyMkJAT/+9//kJaWJuk9MqRSqdCsWTMIgoDLly/r/Wzjxo1o3rw5/P39kS9fPnTo0EFvaGjt2rVQqVQ4efKkdtsff/wBlUqF1157Te9YVapUQffu3bXfL168GK1bt0bRokXh7e2NqlWrYu7cuUbtCw0NxSuvvILNmzejfv368PX1xQ8//AAAuHnzJjp37gx/f38ULVoUw4cPl/w+fPnll0hNTcWPP/6IIkWK6P2scOHC+OGHH5CSkoLp06cDAFatWgWVSoVdu3YZHeuHH36ASqXCqVOntNvOnTuHN954AwULFoSPjw/q16+PtWvX6j1OM7y9a9cuvP/++yhatChKlSolqf3miOUAad7HnTt3at/HGjVqaHOlVq9ejRo1asDHxwf16tXDsWPHjI4r5TVRzsEAiHIszcmtQIECVh/jt99+w5dffolBgwbhiy++wNWrV/Haa68hIyNDb7+srCxERUWhUKFCmDFjBlq2bImvvvoKP/74o6Tn6datGx4/fowpU6agW7duWLJkCSZMmKC3T//+/fHtt9+iffv2mDZtGnx9fdGhQwdZr+fXX39FWFgYGjRogI4dO8LPzw+///679ufdu3fHlStXjIK8a9eu4eDBg+jRo4d226RJk9C3b19UqFABM2fOxEcffYRt27ahRYsWRrlF9+/fR7t27VC7dm3Mnj0brVq1ApB9IQoICEB0dDS+/vpr1KtXD+PGjcPIkSP1Hj937lwMHToUpUqVwvTp09G8eXN07txZL3gDALVajU6dOmHGjBno2LEjvv32W3Tu3BmzZs3SCy7kEvssLV26FB06dEBAQACmTZuGsWPH4syZM2jWrJl2/2bNmkGlUmH37t3ax+3Zswdubm7Yu3evdltiYiLOnTuHFi1a6L3mMmXK4LPPPsNXX32FkJAQvP/++5gzZ45R+86fP4+ePXuibdu2+Prrr1G7dm08ffoUbdq0webNmzF06FCMHj0ae/bswf/+9z9Jr/nvv/9GaGio9kbCUIsWLRAaGor169cDgPa9WLFihdG+y5cvR7Vq1VC9enUA2XlFjRo1wtmzZzFy5Eh89dVX8Pf3R+fOnbFmzRqjx7///vs4c+aM6GfDni5duoRevXqhY8eOmDJlCh4+fIiOHTvi119/xfDhw/Hmm29iwoQJiIuLQ7du3aBWq7WPlfuaKAdQuguKyBLNENg///wjJCYmCjdu3BBWrVolFClSRPD29hZu3Lih3bdly5ZCy5YtjY7Rr18/oUyZMtrvr1y5IgAQChUqJDx48EC7/a+//hIACH///bfeYwEIEydO1DtmnTp1hHr16ultAyDExMRov4+JiREACG+99Zbefl26dBEKFSqk/T42NlYAIHz00Ud6+/Xv39/omKakp6cLhQoVEkaPHq3d1qtXL6FWrVra75OSkgRvb2/h448/1nvs9OnTBZVKJVy7dk0QBEG4evWq4O7uLkyaNElvv//++0/w8PDQ296yZUsBgDBv3jyjNokN1QwaNEjw8/MTnj17JgiCIKSlpQmFChUSGjRoIGRkZGj3W7JkiQBA7/e5dOlSwc3NTdizZ4/eMefNmycAEPbt22fq7REE4cUQWGJiopCYmChcunRJmDFjhqBSqYTq1asLarVaEARBePz4sZA/f35h4MCBeo+Pj48XgoKC9LZXq1ZN6Natm/b7unXrCl27dhUACGfPnhUEQRBWr14tABBOnDhh9r2JiooSypUrp7etTJkyAgBh06ZNettnz54tABBWrFih3ZaSkiKUL1/e4tDQo0ePBADCq6++anIfQRCETp06CQCE5ORkQRAEoWfPnkLRokWFzMxM7T537twR3Nzc9P4+2rRpI9SoUUP7OxaE7KG7Jk2aCBUqVNBu0/xtN2vWTO+YUpgbAtMc98qVK9ptmvdx//792m2bN28WAAi+vr7az74gCMIPP/xgdGypr4lyDvYAUY4RGRmJIkWKICQkBG+88Qb8/f2xdu1am7rMu3fvrnfXr7kbNhwKAYDBgwfrfd+8eXPR/cSIPfb+/ftITk4GAGzatAlA9p2wrg8++EDS8YHs4Zr79++jZ8+e2m09e/bEiRMntMM2gYGBaNeuHVasWAFBELT7LV++HI0aNULp0qUBZA8HqNVqdOvWDffu3dN+BQcHo0KFCtixY4fec3t7e2PAgAFGbfL19dX+//Hjx7h37x6aN2+O1NRUnDt3DgBw5MgR3L9/HwMHDtRLIO/du7dR797KlStRpUoVVK5cWa9drVu3BgCjdolJSUlBkSJFUKRIEZQvXx6ffPIJmjZtir/++gsqlQoAsHXrVjx69Ag9e/bUex53d3c0bNhQ73maN2+OPXv2aF/jiRMn8O6776Jw4cLa7Xv27EH+/Pm1PSSG701SUhLu3buHli1b4vLly0hKStJrc9myZREVFaW3bcOGDShevDjeeOMN7TY/Pz+8++67Ft+Dx48fAwDy5ctndj/NzzWf0+7du+Pu3bt6U+xXrVoFtVqt7YF78OABtm/fru311Lx39+/fR1RUFC5evGg0e3PgwIFwd3e32G5bVa1aFY0bN9Z+37BhQwBA69attZ993e2av29rXhO5Ps4Coxxjzpw5qFixIpKSkrBo0SLs3r3b5lkeuic94MUQyMOHD/W2+/j4GOVJFChQwGg/a54nMDAQ165dg5ubG8qWLau3X/ny5SUdH8jO1ylbtiy8vb1x6dIlAEBYWBj8/Pzw66+/YvLkyQCyL2J//vknDhw4gCZNmiAuLg6xsbGYPXu29lgXL16EIAioUKGC6HN5enrqfV+yZEl4eXkZ7Xf69GmMGTMG27dv115ENTQX+WvXrom+Vg8PD728LU27zp49a/S70Lh7967odl0+Pj74+++/AWTn0EyfPh13797VC0guXrwIANrAylBgYKD2/82bN8e8efNw6dIlxMXFQaVSoXHjxtrAaODAgdizZw+aNm0KN7cX95z79u1DTEwMDhw4YJQPlpSUhKCgIO33hp8LIPt9K1++vDZo06hUqZLF90AT2GgCIVMMA6WXX34ZQUFBWL58Odq0aQMgO3iuXbs2KlasCCB7mEkQBIwdOxZjx44VPe7du3dRsmRJs6/PEQz/DjXvcUhIiOh2zd+3Na+JXB8DIMoxwsPDtUm6nTt3RrNmzdCrVy+cP38eAQEBALITWnV7NjTEkm4BmLzrNDyGrXenUp/HWsnJyfj777/x7Nkz0aDlt99+w6RJk6BSqbS5QStWrECTJk2wYsUKuLm5oWvXrtr91Wo1VCoVNm7cKNp2zfutoRs8aDx69AgtW7ZEYGAgJk6ciLCwMPj4+ODo0aP49NNP9fIrpFKr1ahRowZmzpwp+nPDC5kYd3d3REZGar+PiopC5cqVMWjQIG1Cq6ZtS5cuRXBwsNExdHuqmjVrBgDYvXs3Ll++jLp168Lf3x/NmzfHN998gydPnuDYsWOYNGmS9jFxcXFo06YNKleujJkzZyIkJAReXl7YsGEDZs2aZfTeiL2/tggKCkLx4sX1krfFnDx5EiVLltQGfN7e3tqcl++//x4JCQnYt2+fNrgGXrx3n3zyiVGvlYZhsGvv12eKqb9DS3+f1rwmcn0MgChHcnd3x5QpU9CqVSt899132sTJAgUKiA5LaXoZXFWZMmWgVqtx5coVvQBG05NjyerVq/Hs2TPMnTsXhQsX1vvZ+fPnMWbMGOzbtw/NmjWDv78/XnnlFaxcuRIzZ87E8uXL0bx5c5QoUUL7mLCwMAiCgLJly2rv7OXauXMn7t+/j9WrV+sl/165ckVvvzJlymhfqyZ5GgAyMzNx9epV1KxZU69dJ06cQJs2bYx6PqxVvHhxDB8+HBMmTMDBgwfRqFEjhIWFAQCKFi2qFyyJKV26NEqXLo09e/bg8uXL2mHUFi1aIDo6GitXrkRWVpbee/D3338jLS0Na9eu1euVkDKEp1GmTBmcOnUKgiDovRfnz5+X9PhXXnkF8+fPx969e7VBnK49e/bg6tWrGDRokN727t2746effsK2bdtw9uxZCIKgl4CumWXo6elp8b3LKXLjayLOAqMcLCIiAuHh4Zg9e7a2im9YWBjOnTuHxMRE7X4nTpzAvn37lGqmJJq7yu+//15v+7fffivp8b/88gvKlSuHwYMH44033tD7+uSTTxAQEKBX5K979+64ffs2FixYgBMnThjNoHrttdfg7u6OCRMmGPVSCYKA+/fvW2yT5q5a9/Hp6elGr7F+/fooVKgQ5s+fj8zMTO32X3/91WiIsVu3brh16xbmz59v9HxPnz5FSkqKxXaJ+eCDD+Dn54epU6cCyP59BAYGYvLkyUYzAgHofb6A7GGw7du349ChQ9oAqHbt2siXLx+mTp0KX19f1KtXT7u/2HuTlJSExYsXS25z+/btcfv2baxatUq7TTOtXYoRI0bA19cXgwYNMvp9PnjwAIMHD4afnx9GjBih97PIyEgULFgQy5cvx/LlyxEeHq43hFW0aFFERETghx9+wJ07d4ye1/C9ywly42si9gBRDjdixAh07doVS5YsweDBg/HWW29h5syZiIqKwttvv427d+9i3rx5qFatmlEOiiupV68eXn/9dcyePRv3799Ho0aNsGvXLly4cAEAzPZ23L59Gzt27MCwYcNEf+7t7Y2oqCisXLkS33zzDTw9PdG+fXvky5cPn3zyCdzd3fH666/rPSYsLAxffPEFRo0ahatXr6Jz587Ily8frly5gjVr1uDdd9/FJ598YvY1NWnSBAUKFEC/fv0wbNgwqFQqLF261Cig8vLywvjx4/HBBx+gdevW6NatG65evYolS5YgLCxM77X36dMHK1aswODBg7Fjxw40bdoUWVlZOHfuHFasWKGtlSNXoUKFMGDAAHz//fc4e/YsqlSpgrlz56JPnz6oW7cuevTogSJFiuD69etYv349mjZtiu+++077+ObNm+PXX3/V1hQCsoOcJk2aYPPmzYiIiNDLkXrppZfg5eWFjh07YtCgQXjy5Anmz5+PokWLil5gxQwcOBDfffcd+vbti9jYWBQvXhxLly6Fn5+fpMdXqFABP/30E3r37o0aNWoYVYK+d+8efv/9d21vmIanpydee+01LFu2DCkpKZgxY4bRsefMmYNmzZqhRo0aGDhwIMqVK4eEhAQcOHAAN2/exIkTJyS10ZXkxteU5zl/4hmRPKYqQQuCIGRlZQlhYWFCWFiYdhrtL7/8IpQrV07w8vISateuLWzevNnkNPgvv/zS6JgwmHYuVj1YEF5McTf3WM0+iYmJoq9Jd5puSkqKMGTIEKFgwYJCQECA0LlzZ+H8+fMCAGHq1Kkm35+vvvpKACBs27bN5D6aKeV//fWXdlvv3r0FAEJkZKTJx/3xxx9Cs2bNBH9/f8Hf31+oXLmyMGTIEOH8+fPafVq2bClUq1ZN9PH79u0TGjVqJPj6+golSpQQ/ve//2mnHhtOX/7mm2+EMmXKCN7e3kJ4eLiwb98+oV69esLLL7+st196erowbdo0oVq1aoK3t7dQoEABoV69esKECROEpKQkk69FEEz/LgVBEOLi4gR3d3ehX79+2m07duwQoqKihKCgIMHHx0cICwsT+vfvLxw5ckTvsadPnxYACFWqVNHb/sUXXwgAhLFjxxo939q1a4WaNWsKPj4+QmhoqDBt2jRh0aJFotO3O3ToINrma9euCZ06dRL8/PyEwoULCx9++KGwadMmWRWST548KfTs2VMoXry44OnpKQQHBws9e/YU/vvvP5OP2bp1qwBAUKlUemUodMXFxQl9+/YVgoODBU9PT6FkyZLCK6+8IqxatUq7j7m/bUusmQYv9j4CEIYMGaK3zdT5QcpropxDJQh2ysIkIrs7fvw46tSpg19++QW9e/dWujlOpVarUaRIEbz22muiQ15ERLZgDhCRi3j69KnRttmzZ8PNzU0vgTY3evbsmdHQ2M8//4wHDx6ILm1CRGQr5gARuYjp06cjNjYWrVq1goeHBzZu3IiNGzfi3XfflTS9Oyc7ePAghg8fjq5du6JQoUI4evQoFi5ciOrVq+tNzycishcOgRG5iK1bt2LChAk4c+YMnjx5gtKlS6NPnz4YPXq0Xt2Z3Ojq1asYNmwYDh06hAcPHqBgwYJo3749pk6diqJFiyrdPCLKhRgAERERUZ7DHCAiIiLKcxgAERERUZ6TuxMLrKRWq3H79m3ky5fPbuX2iYiIyLEEQcDjx49RokQJvcWHxTAAEnH79u1cP+uGiIgot7px4wZKlSpldh8GQCLy5csHIPsN1KyCTERERK4tOTkZISEh2uu4OQyARGiGvQIDAxkAERER5TBS0leYBE1ERER5DgMgIiIiynMYABEREVGewwCIiIiI8hwGQERERJTnMAAiIiKiPIcBEBEREeU5DICIiIgoz2EARERERHkOAyAiIiLKcxgAERERUZ7DAIiIiIjyHAZARLmAIAh4lpGldDOIiHIMBkBEucCgpbGoPHYTbjxIVbopREQ5AgMgolxgy5kEAMDKIzcUbgkRUc7AAIgoF3FzUyndBCKiHIEBEFEu4sEAiIhIEgZARLnIyZtJSjeBiChHYABElItocoGIiMg8BkBEuYg7h8CIiCRhAESUizAHiIhIGgZARLkIAyAiImkYABHlIp4e/JMmIpJC8bPlnDlzEBoaCh8fHzRs2BCHDh0yu//s2bNRqVIl+Pr6IiQkBMOHD8ezZ8+0P58yZQoaNGiAfPnyoWjRoujcuTPOnz/v6JdB5BI83BT/kyYiyhEUPVsuX74c0dHRiImJwdGjR1GrVi1ERUXh7t27ovv/9ttvGDlyJGJiYnD27FksXLgQy5cvx2effabdZ9euXRgyZAgOHjyIrVu3IiMjAy+99BJSUlKc9bKIFJP8LEPpJhAR5QgqQRAEpZ68YcOGaNCgAb777jsAgFqtRkhICD744AOMHDnSaP+hQ4fi7Nmz2LZtm3bbxx9/jH///Rd79+4VfY7ExEQULVoUu3btQosWLSS1Kzk5GUFBQUhKSkJgYKAVr4zIuUJHrtf+/+rUDgq2hIhIOXKu34r1AKWnpyM2NhaRkZEvGuPmhsjISBw4cED0MU2aNEFsbKx2mOzy5cvYsGED2rdvb/J5kpKyC8MVLFjQjq0nIiJyPd9uu4gKozfgQUq60k1xeR5KPfG9e/eQlZWFYsWK6W0vVqwYzp07J/qYXr164d69e2jWrBkEQUBmZiYGDx6sNwSmS61W46OPPkLTpk1RvXp1k21JS0tDWlqa9vvk5GQrXhEREZFy0jKz8NXWCwCAup9vZW+wBTkqY3Lnzp2YPHkyvv/+exw9ehSrV6/G+vXr8fnnn4vuP2TIEJw6dQrLli0ze9wpU6YgKChI+xUSEuKI5hMRETnMjnOJSjchR1GsB6hw4cJwd3dHQoJ+6f6EhAQEBweLPmbs2LHo06cP3nnnHQBAjRo1kJKSgnfffRejR4+Gm84MmKFDh2LdunXYvXs3SpUqZbYto0aNQnR0tPb75ORkBkGUI3WpU1LpJhCRQlgJXh7FeoC8vLxQr149vYRmtVqNbdu2oXHjxqKPSU1N1QtyAMDd3R0AoMnlFgQBQ4cOxZo1a7B9+3aULVvWYlu8vb0RGBio90WUU6SmZ2r/H+Ct2D0NESlMwTlNOZKiZ8vo6Gj069cP9evXR3h4OGbPno2UlBQMGDAAANC3b1+ULFkSU6ZMAQB07NgRM2fORJ06ddCwYUNcunQJY8eORceOHbWB0JAhQ/Dbb7/hr7/+Qr58+RAfHw8ACAoKgq+vrzIvVAFrjt3ET/uv4ae3whHk66l0c8iBrt5L1f4/IfmZmT2JKDfLUjMAkkPRAKh79+5ITEzEuHHjEB8fj9q1a2PTpk3axOjr16/r9fiMGTMGKpUKY8aMwa1bt1CkSBF07NgRkyZN0u4zd+5cAEBERITecy1evBj9+/d3+GtyFcOXnwAAfPD7Mfz8VrjCrSFHunb/RY0rPy93BVtCREo6e4cTeORQtA6Qq8oNdYBYFybv0P1dv9OsLMa8UlXB1pBc+y/dQ9y9FPRpVEbppuRZgiAgLvEJyhUOgFsOzqOp9/lW3NeZ/p4Xz/1yrt9MGMjlGpZl/aO8pFCAt9JNIJl6LfgXAFChaAAalSukcGvypqkbz+GH3ZdROMALR8a0Vbo5VlMb9GdcTHiMCsXyOe35f9p/FcFBPoiqJj6RydXkqGnwJF+FYgFKN4GcSIDrdegmpWbg+I1HTNC0YOWRm0o3wSonbjzCo9ScWXTv8NUHaDp1O37YfRkAcO9JznwdGg9T9ZfCaTtrNz5fdwY3HqSaeIT9xF57gJi1pzFoaazDn8teGADlcirk3O7cI1cfYNxfp+y+vpU6lyUK+nq+yPtxxRijzcyd6DxnH3ZeYI0Sc9advK10E2Tbe/EeXp2zD62/2qV0U6zSdd4B3Hr0VOlmONTCvVfQa8FBhz/P3J2XHf4c9sYAiFzWG/MO4OcD1zBzywW7HfPzdWdQ7rMNOB//2G7HVNrTjCzt/y8nut6iv5q76i2n4xVuievRDcbTMtUKtsQ6G07dAQBFl104czsZ0SuOO6WXI6e68cDxQd7TjEzLO7kYBkC53N3HOX9a9OV79ruoL9x7BQAQNXu33Y7pSv446jrDKNM3ncN32y9qv09+mvNOkI6WmcN7I3/797rSTUD7b/Zg9dFbaDsrZ/ZC5Rb7Lt1XugmyMQDKBVLTM5GWmSX6s82nE0S35ySOGsTTnT2VW3i5u8af9O1HT/H9zjjM0Om9Y5FGY7mpbsu2s8qea55lqJGUat/hcrJOTkkzcI2zJVntWUYWqo7bjEpjNmHPxURsOR2P07eTlG6WXd124Bh9blsxOVPtGsMoYr+zEzcfWX28u4+f4ecDV+2eD6Y0V/l9WcNwGHm3C+R4DVx6ROkmONSVeyn49d9ryMhy7c/N42c5o7eXt2Q53Ikbj7T/77PwkNHPm1co7MTWOMbFu08cduwcXPIDAHDs+kO979UC8DQ9C74KF0R8/9ejRtvO2ZB39eaCf3Eh4QkOXXmA73rVtaVpLiUn9wAN/Fk/2PB0gd7HQ1ceKN0Eh2o1YycA4EpiikvX+3JT/qMgSQ5pJplyXCcAEhNS0M85DcmhbLn+/Hv5Ptp9vQex15Q76c7dGWe0bdlh5fMy7j5Os+vxLiRkB8HrTt6x63HtberGc+g1/yAyJd6hp7v4nbwpuy8k4rpB0nFOLiCY0yx4nsvoqnJKXM8AKIebsvGc2Z/nlLFYpdhyB979x4M4eycZr889YMcWyVOmkHGAm5oung+mtA41iyvdBIeKS3yCebvisD/uPvZcvCfpMTl1BOyrLeeNtv24W/o06NhrD/HVlvNId8DMt+M3HkkOQMkxcsp1hwFQLrc/Ludl5gPAJQcMe4kV4pNbnE8QBDSc/A96/uj4uhpSVA42LvX+5Wbji5MrqFe6gNJNcKh2s/do/79Z4pT/LJ3Pn6kkcbVawPi1pzFl41mXubCcuGlbnuHrc/fj2+2X0Gfhv3Zq0Qud5+xD+dEb7X5cMs0w4MxyxYJkIhgA5XKG3dQ5xbMM/V6MsX+esvmYYsMNcq8nW88kICE5DQcuu0Zg+fHKE055nnm74rBkn23d7vee2GdYzJFJ8bbQ/XydlZjvpBvQdKxVQnSfTafjsWT/Vfyw6zJ++feabY10Mf9eeYAnaTkjYdbVuFJl9YjnuUka/5zJGbOPGQDlcu2q54w1WQwdNAgwlh60/cT/LMM4ANK9U1GrBYtd8tfu58yA0hZ3Hz/D1I3nMP7vMzYNWXwvkq9kjZxQMLBUAV9J++mu3WRq2EZ39twfsa5T58leDl1xjZuJnOa/W64z2/fmQ/2bkpGr/1OoJfIwAMrl3HNoYmJGlvjdTWp6JtYcu2nV2kNPRXJjdO/AX5+3H/W+2IrUdNN3pDO32q8qdU7xz5m72v8bLrboDIbDPj6ern/aalZe2uxL3Ry0+GTxoqVn77zoTXKVi14BP0+jbUNahUl6rGHPhavlQbnKMKMlj0RqHlUpbn71c9Ln+mcSsok975bTM9V47KQ6LMFB4quaD/v9GIYvP4Gu8+QnHqeIBDZxiS9yjY5df4THzzJx5OpDo/00nma4ZoKxIy3WGfpSIgAyrJZ89Z7r98L5SSxDoPt+mkqc1q2v4yrXZt315zTy+RgHRWIMJx64WjXsHJO/ojK+ufXycI1L+ktVi4luz8xSY+hvR20eTrcX13i3yGHsWX234piNqDF+i1OK0Zkqsf/P2ezeCGtqA50UKcTXf/FhzDBIGs6LvTymZGap9d5rJerWGAZdPee7RgK6Od4SL0Qudu2XLEGkzMFUCzNSNQwDDFerheRq7THlz+O3jLZdu+/ctQBnbb2AIb8Z1/wyVX9u46l4rDt5B+P/PuPopknCAIhk+90J6/84Inl78b6rotu/23FJ73tLtZXykt8O6f+upV4c6pbOb7c2OGJGoKO5S6wEl2aQl/bjbvvkSTmaLUGCYS6eLdWwHVH4sMv3+/GrCyabGw4droq9iZsP9c+T/l7OrW389baLWC9Sm6tIPvEefFdLeGcAlMs5YqbAVSckAoeXLWTX4yUkP8NJM1N3HVGPxNHumsgZsaej1/SHA6Ve+CoUzWe3Nrzy7V67HctZPCTk3p2Pf4yO3+m/tskbpPWiKMnW9bYM1y00le+nkaUWcPp2kmhuTrcf7F+D6+ydZIxeY/usU3sTW0y4y/f79b4P9JU2DOlopk4Tun8VShaQ1WAAlMv9efy23bt01528bdfjiTE1hqzrxoNUTFp/RtK06LeWHDb7831x0grXmaLElNTwydsc/hx/Htf/XT+UePFzdB7FlA1nHXp8ue4k6X8GZ/1jeRj1g9+Nhw5c0albSXpBz7s2rrd1+5F+4G5pXasxf55Ch2/2YoZI8UVbhYoUEnVVUzYaf+YTDYYiNZ/DpuXtewMpV4qJnp77OmsvjvxD+ZliDIByuIZlC1rc54CdiyE6Y6E7KZfP3gv+xfw9V/D2T5ZPyKdvJ5v9eaaFu1BL4hKdO/auFKmrzdvymTtx45FRQGHoh92XXaoOimH17ZM3kyy+hpQ0ywn1hvWwnO3QlQd45du9aDZtu3Zbso1//53n7NP73lIA9PvzYVh7lVHQJZZI7GoEQcC/l+/jlIRFrjUzw3qGl3Z0s8zaYqIO0ByddANXmKHMACiH+1fCGPhiO2fct6hYRNb+K47cwJSNZ2VdsMRWtDfsAtfkCZ29Yz64kUJqL9krJpZzcNXifPZ2RGK39S0r349Ldx/j1Tn70HjKdov7rj3h+J5IqcTKMlgKcKSkCYld9HXrBa09cRuNp2xz2MSEbWezL2SPde7o7b0SueEwqzMpMatRrtVHb6H7jwdx6pb089zVe8Y3ZPaewWvufG7qGqF7o+AjMpPQ2RgA5QHbzt3FxQTrV+I25C4zcP/fqpP4YddlWQmLP+wyXlfIkcMqUgMgbw/xP9q+iw7ZszmKuPckDcN+P2a292bvJduGCpcdMp9Af/yG9Do3W05nX5yV7iU5fuOR6Hpw5joXstQCbjwwHyQ+Tc/CN9suGm3X7c0c9vsx3El6hmZTLQeMcmVkqbF4/1Wj7fZe9X3vJeUKIeaECV/WVHsXy2m0d8/9zvOJJn8mpXK/K0w2YQCUR7SdtdtuBb52nE9E6Mj1CB25XlZBwkdPbbsDceT0VKn1M/44mvsq8WqM++sU1p64bXaaua2/A0sVYuV8RrPUAr7cfA6Vx27CfhtzuGxhOKSjYe4+YZaEUgtjTFxE3v7pCLr9cEDbOwPYPiwl5rvtl0QvpEG+pmcaiRUbdWWu3gM008q8J5VKhU9eqqi37YSdAw5nlENxNAZAOZjcKYXlPttg9XOZ6u5s/dUuycdYtNe2oTjDKZ/2NPBn/TwiV68G64j8lw3/WV7A08dED5i9uMnIC9h0Oh5zdmQPEQ1aGuuoJllNZaYLyLD0ghhTwfa9J2k4dOWBpNw3W3wt0vsEAGUL++t9r/syT4jU2rKkQahyi+S6Svxz9PpDRM3arddLfvfxM3yz3fLnRIwAoHCA/lT0Z5k5Kzh1BgZAOdix684bOzd15/8gRXoP0L9XHtiUr5MkMg3UUWwJFqV4kJKuHZNPfJxmdvkNMUoVa5NY3sZqa469uOjPljCTSsMZiflijly1bipv5WD7lQlwtiZh+kXuQgq8mEllKXwVSwzfeMp04G1tvpGnxHF6V0mk77fwEM4nPNZO6y//2QaET5I3y9PwtRgOVT6x89+Iq/eeScEAKAdz5ufPVP5NWBF/0e2mtPt6j9ULmyqd62EvKWmZqPv5VtQYvwWJj9PQYNI/qDpus6yTsaV8KEed2H8/dEPSftVLWrcmke4yJLP/uYi/T9zG8MiKZh6hrBNmakutPCLtvRJj62f9QNx9zNlxya6fA02vqOGFr02VoqL7C4KAiwmP9XpTx689Lfn57j9JQ4XRG61oqeVzoyAI6LfoEG4nZU/J79VQ2VlTjw16861ZHkTvIYKAV2oV15sOb/gctpJbv/L+E+Pq4UpjAJRDCYKA935xXre/qR6H+mUsT8M3JCVBTow13eu2cNS6Z5d1pswf1ulBGLHqpORjiJ18apUK0v7f3Kr1D1LSMX3TOVwRmSliL9ZOLzZcu+6D349JqqmjFHOv0ty07XPx5iclVB67ycoWZes5/yC+3Hweq+y4enz6896YXRf0k191qw/rnia+2XYJbWftxmidv/fNp8WnR4sl3r+5UN7EgsEtXyzGaql3IiE5Te919BYJgP4yWGpCELILMrpqnpNusPty9eLw9nDHr+800m57KKO3Xgq5PUCjXHCFeAZAOdSj1AykiPwhlszvK/kYcu4OMzLF9z1nx9llGqZym6ZvMp0QeDf5GUJHrseXm+1XSbeNQX6Tve6mdQua6QaWci5Whj1AFYsF4L2I8trvzZ2c/rfqBL7fGYeXZlnO37K2noi55z9joibTdRsrjP9iZc+iLVx9cVw5QbUlmuGo1UdfBAZvNiqN2iH5td/rJsZqAtffLcz8A4C/RYqryh0u//TlShjToQqA7EDM3N+rYFBpzE2lMjp3xhj0Vq07eQcdvtmLKuNsC04dJV1nuFBsKYr5e66IrodojYOX78v+bJmqDaQkBkA5lKlk0a3RLSQ9/p8zCSg7agPKjVovaf8ME/2dLWXWBJLCmmRnTVVkTVKsPdx9nKZ3EhULOA3df5KGL9adMZugvl/nbtfamRSGvVMXEp6gavEXw07memA0U9ktLUFgC3OxYvtv9oiWZZguM3jN56M/G2nMn6eQ/CwD95+kYcqGs06ZpSI2TV2XqyfTmyLWbrFioV90roFmOgtfanpN5N4siK0uL5dKpcJrdUtpvzfXBMPXIvb38sig6vkHvx+zrYEOpltjx3C5EY1O34nPWJSrx4+WFyTedSHR7jWj7I0BkIvpOm8/Yv6yPERkKsnPQ2KW6jvPZz1JPT+b+iD7e5k/cVmzkOXLs/fIfoxc3/asI2k/3aUfLCUep6Rlot4X/2DB3iuoHrNZ0vGtXXPIsIpqiSAfvbtacyNQhotRAsYnTE2xPWu7zS19rhaJLEy7TmRRRXPEEp/3XbyHel/8gx92X0bN8VtkHc8SQRCw/VwCEnTWYDMcsjO066LpWimu6q0lh0UnAZiq26Jb0K7I85lHUm4WdG0ykwitS/e9FxuK0s2dSjdz8f2fQe/F8RsPHZ7gL9WW09LeCw3NeUn392A4A0wJ/RYdwkwL5R7uKZwX5CK/cgKA9Sfv4PDVh/jpwDUkWaiZY6qMuDXVxaXcrfmZWGV4ysZzZh8vpU6QIAgY99cpSRWrA33kr3a8X6R4X8daJSQ9NnrFce3/Ld3N6+5rrb+O38Lw5cdN3sFpGTTlpWrBKKRz0jMXrIm9h6kGVYvLj96IzCw1Nsk8GWubZ+EzJWVYxBrv/Wr/9bVirz3E4asPsPbEbby15Aia6BQdtJTsbWp2mtRZStawdbHS7efuim5/KOFvuVXl7IRouechqZXDG+qsfzd8+XHt/6e+VgMA4O/94rNtrgfw1C395PV6ZQpaLEypy5F5QO/KLOlwOTH7JlN32NlVlviYuzMOarVgcv3ILSZywpyFAZCL2HMxEd9uf9Gdbqm79WGK+B+3udojGobDJ1J6gcxd/M3daZn6me4F8si1h/j5wDVM+PuMxXZYU/DtVxsutrrVTi3NvDKV4CnHh8uOY82xW1iwx3wwaPjryFILCNA5+Zsr2Cj2KsS2WUrUNefG82VKJnSqhi/fqGn1cczRfb32kKUWjC6a6ZlqvD53P7rOO6A9WesGl4ZTwg3tu6gffD9KTYcgCKhaIsjEIyC7JIKhJSLVm6XKNPO3HL3ihMWbAM2aeM6YoaobnIcUzJ6Kr3tj+JrBSum6ShXUXwTVx1PepfDgFeWqVxu6+LyXXfd346hltvZdumdxYWlDvx++jqG/iV/P5JS6cAQGQC4gKTUDfRYe0rvg7L5gvuvc1AwlKQvM1TAYGpCSzW/u4i9nrF3sMY5eR2u9wdDKkTGRso8Re+0h1jlx7am5FhZ+NFwF2jDulZoLFf98GvA+kV4yW7rRNZW1SxXwNRm0ilUZlkNuIVBLXp2zFzXHb9GrVaMbwKeIBCY/7jZeskVXfj9P7f9/PnAVtSduxccrTpitymsp+LXk0VPrZ/tYmn5t6Sbg83VnsPl0vOg5xVxZAMPPmtzcEc2MRt0/g5sPTZ9XihokCZt6WUsPXkO/RYeMglLdKtxKaFXpRe6l5r3W/dU5aqHRIb8dNdlDaMrSA6YnJ1gz3d+eGAC5gHX/yb+w+ll59ys2NCElADJ3sTKXB2HqRKZblfTDZcctPr+9fN65ulUX9tfn7sd4Mz1UVYqbHwqRy9LFfWWs/sUkyNfTxJ7mNZqyDUN/O2qyx9HanvQC/l4AAF8vd9QtnV90H2sKqc3vW9+6BiF7ZfNX5+wzGXxoFpvcrJOPovvyLxj0iEmp1VOh2IuCh5pgafWxW6Z2BwCLeROWLBbJr5Jq0vqzZn/+xTrLvbSDlsbq5eoA2Tlm5mYNvVa3pN73ctecaxKWXe9G6idK6mdv7J+nsOtCIob9flxv+y8HHTOEK1XtkBfVszXD13/ozCI1NxJg6ebKHMPEcCnM9STLKaTrCAyAXMCEtZZPKroystQ4a2IqsSVXRaYaS6miu+E/0wmqf5vpGTH1d9j+a/FEZ0fPmknRCSyaVzA/fCGHPVakl+Nng7uq7g1CpD/Y4C02lXycJQiid8aGvU9iNL9HT3c3vWnSuqy5+2tbtZjVwWa3Hw7gxI1HZtc6A/QvHrqfF03RPA0p70O8Tm+Sqbvy8kUDLB7HGdRqwWKR0p/M3M3remuJ/jIdloIyW1NWyhWR9x4a1smyFA/948AeH7lV3d3dVPD0ePGGaZYsmbTBfPCqMW2T/UqF5HQMgFyAuRwaMcOXH9fO4pJL7I8teoXl1YZvP3pm8mfmchZKG4y1a4gFYoD+StcAsHJwY4tts5aXlataO6p72VpHx7ZFqQLi77MtTA2zNpj0j8XHXn+eA+TupoJKpcIPfeoZ7XMw7j52nr+Lll/u0CsIaUkzneq21ki1kMCq22tpqlikWi2IBnAFn/d8aczY8qI3J8NET2nn2iWw/eOWZtskRZZaMFpkNbys9EKlcs9D5nodNb9/jakbzV90DXuYbzywriaU1Lwww/YZ1gWyxaW7TyQFxxpyV0UP8PaAt86afFITyE1JSH6GNl/ttHmtxpyIAVAOJHbHPu/NutgwrDmA7GECUwsMRs40Ln73VELS5SIzM7S2nE7A/N2XTQyvWTy0wf76D2gQKr/StDkeOsGLtTkuWWoBqyQEZs4qzGd40dU4fdv0Mg1SWLu+VnqmWvt71wTcJYKMC3RevZ+C/osP49r9VHSdd0Dy8eUsmGpNnpBujoOpj2+mWjCawlsrJD9WDDL9uTDsQdJ4t0WYNonXlHlv1sPeT1uZ3WflkRtGC5jqLq5pidyeCFtvTnRvQAzLLRj2cDqaNUnbYue724+eInLmLkk3CRqvzzWdrC3G090N+XQCvXxWpEM8Sk3H+LWncepWEqZvOo+4xBRMlDC8mdswAMolXq5eHFVLZA8NtK1aDCsHN8G5z19Gt/qlLDwSOKyz/pI1jlx7iEkbzmKySBfsv5flzZawx+zNu8mme6t0L+pjXqki6XhivRP1Qwvi7MSXUbGY6a73MVYu+WEv95/YNr6+xMywhblp7rrDRpoAxFtklo3cAHTxgAYAgDgZtaV6SijYZkg3oDRVWX3LmXij/LYCfp6iw1kj/ziJq2aWHfHycNMLzMW8XD3YYi+fNTW3dJlblFRMBRuH7nR7Xf48rj+MbutrMcdeS9ysF0kL+Ou44ydKNCtfCC/XCNZ+H2BFaZBvtl3Ckv1XMWzZMbMzRnM7BkC5mI+nO97XWR7B0eaLzF7Zc9F8MqPhhVT37rtwgHjPhiWaIY60zCyji5TuHXI+H09t6XxzTPVO+Hq5m5zlZoqsBU9F7sifZWShzkRpBf5MPZPURRHFTvDaY0t8GZrLuthw40c6dVykaFUpu8bMP2elz0L571YSyo1aj60GZfjNrYNW0N8Lj1LT0Wv+Qew4L/5cQ387ZpT836JC9swcw5Xelx2+gbYWlh0xl7RaqsCLIOyP90z3utyVMewiZvQaeWs1SSm5YY5Si4n/IbLkjJtKhXealZV1HLGyF+Zm99lLh5olEOjzYvjRmorumlmfuusS5kUMgFyYPdaeMpWDYwspwz8aphI8NTODDK/xf+rMkJFa1dqQWhDwMCUd1WM2G60mbVi4znA5BblMLRFiipzEX7HLS5+F/+pVpzbnpI0n4xolTdequSjxDl2T06WZFq8EtQAMNMiZazVjp973uitV/3LwGmpP3Ir9cfcx7i/Tq5cnGAxp+T6vir7++VC0LluWHdHNJ/Exs2TEWhvLNFiqam1vNUuZ/nzZy1dbzmPlkRsQBAHPMrL0hmd1hRT0xch2lfF55+qSj+0p0mvnqfM5N1XM1poeqOPj2qJkfl8MaBqKyCpF9X6mdDVlW4gtQutMDIAUZm7Wk+6SBanpmSaTjce+UtXkMdzcVPh7aDPrGyiivoy8HFNdwi9Xz+7CNUwW1E2OtlRl1xS1AGw+HS960Xm9rv6QoK0nfU+ZidRypn6LJaXKuWP775ZxDpCUqdsa9U3kkQHAk7QXJ/H7T/TXTNPtGAgr4g9A2QDIFLVa0E7D1c0VkhqjGhYc1LxseyfJ637GzAVAOY1htWLdYcLQQi9u3IoFWl+P6tvtlzBi1Ums/+8O6n6+FXUmbsExgxuDisUCoFKp4OHuhggZaxuK5aLpVvgWG/4ftfo/1Bi/BauP3oQgCJLzf/L7eWHfyNaI6VhNds9b26rFjLaZW2ngUWq62YKY5nSoUVzW/vYuZiqX652V8hhzPQKai2VGlhpVx21G1XGbZScqAkDx/D5Wt0+uTaf0h01MzVD4aX92kuObC//V265bOG5ku8oAsmfJyJHPx8NkLpHh9Fdbi/G9VDXY7M8NTyRyOvXEgrP7MupmiOWM9DF4v80xN3VZ87LWHLuJel/8gyk6s3x0gwlNG2wNgGqZmEpvi0ZTtqHu51tx/MYjs0NiphjWNyldSH5vqyZANKeczj7eVryPuoUdXYnhn8KV+9m/g5sPU/VuhBqXs23WHwDM/uciUtOzkJKehfwGs9f+GvLiBlHODcKq2JuYaFAbTHeoN7+f8RC+ZgmY6BUncPT6Q8Res5x/Wbaw5c+IOWKf7XgTeZI3HqSi9sSteOXbvVY9V5Xi+Szv5EIYACnMXI/Ao6cZuPEgVa+i6XmRolKWFiS1tkieNQb/clTSFFBNYGSY7KhbaEsz1VPuHY9aEKASHUAyvmu3tQfIUgn9LwwKy8kJYHdfSMSCPeKz66QIK2p84rQ14V0jSy0g6WkGhi/PLqGgWxFZd2Xvas8T83VzFqyxwIYCiKZocmbm775sciadHJaWxRCj+7diajZPjwYvhgms6QHq/oPlRHAlhlF0c5uAF2UCDHNrLM2Qk0L3PGNYzdpX5/wp9/5y0b4rejc5uoG+pd5esUWJxYj14ABAxPNq0OPMjAAA8hLKNeUKdIP7ZxlZuHbf8g2Cp7sK+ST8nb8XESa5PY7GAEhh5nqAvtp8Hs2n79DLV0h4bBy5dzGoompI7jCNrgcp6WhnomihKdcf2CexThP3yK3Xs2DPFXy345KkfUMLvQgSPmtf2agirSXVzazpBBgHXJaWEtD1we/H8MX6s5JXygb0Z1Z9/c9FM3vaRhAE1JognoyteY0qlX7w+vNb4ZKP/+9nbfS+L5LP+mEQS9b/d8fmoaWWMoZOdOkuE/KWiSTcemVeDEUatnNQy3IWn8Ow5o2Y93+x/yKy5jQuVwjjO1bT26YZmjIcnrF3svRv/76o4my4MLClRajF6ObD6eYtWpogIbW3KdlEmzQ3ttZUVDdFbOJD5zn70PLLnThoYkbva3VKYmDzsrg4qb2ka82nL1fW/v+pjB43R2AApDBzPQJiJfPTRO4adIti2dvX/1wQrXJ8bGxbk48J8rX9bhp4kSMwsIW82RkL916RdNIHAD/vF+/duy3CMLNbbVnP1cYgIVGMbg+OYb0TKeISX5xgKxUz38X8Xa862v/bOivIHLFATnPxWHkke5aN4S4tKhbB3N51JR2/WKAP/olugXw+Htj5SYRNbZXiJxsWEAWAb3rUsbyTBaYuZLrDZLo3A9Ner4FPoyqLPUSW3/69jkMyClGK+aanvNf/+7uNtMulaGjKJyzco7++2ombj2Qde1Q76e+J4Tp11gzV6g4xano8gexZqOZITYw3dY1wf35+VAuWK3jbQtMb9Ndx8SVcZnavjdEdsnuhAn2l5fRoZkoOaCrv3G5vDIAUJrdQm6O6qk2t2bXHxJo8hicvXaZyQK9O7SCrTZoLQvmipi/6uosCWqNh2YII8PYwuVyDJVKG53T3saanQT8J1vyfbN3SphOX7aloPuO8Mk3doC83nzf5uHYykiTLF82H/8ZHIdQgB+KD1vYv7fCrTq+ANYL89Lv+pSZC92tcRvv/umWMf3ebP2oBD53fv+4FunOdkrKKQgLiky4+kzn9XYxhT4o1NDd3KQaVug1Lafy4Ow5HdAI2w7/d6mZmLxr6KLKC3vfWLEuSnvniPdVNN7D0ty618rappV8055UvN5/HWDM1x8wV55RDrDCq4SyudtWl/X1v/LA5Tk2Isjm/yVYMgBT23XZ5wxTmVlS2xSqR2hjX76cazTp6s5HlaYvXzPS+tKmc3WPSurLlnhNTXcir32+i/b+5QExMC4OhCj8vDxwd2xar32ti4hH2ZWt3teGFdeKr+sMItiYbm0qyNTwJi/WwrTnmnIJqYidia5KDHWnt0KaS9utY60WCv6bOkS4Pd+MAJ25ye1z4op3snt/LiU9Qc8IWzJE4PCxHIzskKmsCAsNcKMOZQpM3nMN7v74Yshtp0OMjJ2XQsEK5NZ+jTJ1SGLqVOywFwYeuSCsS26GmeFChiYst9SRVLWHdbFpDYiMBhpMTpAb+KpVK8RlgAAMgRaVlZuH3Qy8CmtHtLRflq2BhCMSUCAs9JaNWG98FnhH5wA9qYTmBrbC/6XwNzTF1Cx6aYmp8WLeXw1IFXUMDmoQabfPycJN9Jy2HbpKkNbP4dB9iGHjIWgRVAlN3rV/3qK33fQE/42RHU+u7yeVr4c65Yy3jC8LpCVF2eW57qWYhN0zDUp0usR4bdzeVVYFu66924UlaptkeOjGXJrWzuI+pz00ZCzPjdHtgNENGhkFfPZGeMd3k8QIGs63kLOFieEPibkVxR920BN1OHUvnJksrym//uCX+HNIUxQLFZ/Eevf5IUvtsuenqOu/FNP04kRIcct+tHnY+X9mKAZCCDJNUB7awnND4cjXz065NKR5kn6nwun9MIQXFlwnQnMDECn7d0SkeZykYkHKSl7ucgqVAEAAOfdYGI6IqyTquObp3aClp8pP+NKs3X0x4jHsGy1tI6QXYeiYB3X84IGk2manlDQxP5h4mkh0Xm1kzTqN9DfOf4bOfv2z25/XKGNehMtUeV1fUxMVNQ07SvCO82ag0PNzd8FXXWhb3XdivvlH19s0ftTD7GN1eXk0gYfi7tDSsa1gfTU7um+G7a82N0IhVLxaT1k2ituZmR1e5IgFmh+ZNze4qZNAr7mNljuilu08szho1rOUEAG/UM7380sRXpReadIacedbIJb7fGSf7MdauL6Xb02QL3TtWU3fqWc8Xiqwx3vySDWduG/cw6dKt12G45tbh0ZH4J7oFIk1METVFSs5O0UAfs4uwNpW5GrnumkdSAgRT2s7abdXjBv58BP9eeYCyozZY3NdUj5Lh+2bq5DvBoC6KmCJWLkKb20gpGmfrRdSSBxaS8r/oXAMA8LqZi5pGmyrFMKt7bb1tlvJgKuvUjdEMnRtWWJYy002PjKBRrKyIXLq/It0hNKWC11EGIwleHm5Y8778IX6xhbMNiZ1OZ3StZTLf09UKorpWa8giUwWs5PqhTz2jbYY9BGKzGHQvhKaWqshSC5j9zwWLbej4nfliW4V0LpTfG8weKpLPG+WL5kOtUvktPo81zC1eOf0N/bvhyV1q6H0/yKAnT7fex7LD1gWiCVb+3n/cLS/IFss5AfQr3ALAJytPiO4nxdk7tl90xBiuwWVvk7rY9+41RaSyezmDwohlCjkmSfRhSjp+3B2nN8PQHuTWQorS6dHWBNWGPUA+nu5Y3L+ByWPYEiNKqW8jh25AIHOVHLsRW96njoMmRxw3s9yOpeFPV8AAKIcxtTq1XFHVgo0S/sav1V/3yNLd4bA2FUS333iYiri79j2xlC+aD/tHtsbZifrDI3J6rA1ry5jTvKLpE7nh76BXw9KYr1Ooz/AO7Idd8nv6DDWcvM2qx03ecM7yTjpMFZAsmd8XXerIq5Fkiq1Trk35sY/9iyXq6t2wjOWdZBC7cE/spB9k2StR1LDmTJ3Pt2LyhnMmF/oFzOdimbq4WUqCjW5bEed0hjjF6sY8TDU+75gburYlx0WsR3hgc+unZuv+TpXqAbL2Zska5iqoG1bdd0UMgHIYw6mTL8kcAgJeTEc3PPn8dEC/loSlAOjl6sGi65B9uOw4DpgomqXLVOVbU0rk99Wr2gpkn8BOT4jCiZiXLD5eztBL8SB5gWb48yEzTXVb3XFww7WHDM3sZjm/wpE2fvhi8U5TI4QqlcpoeMMRzOUP6Fo8wLhHQOl8GUOWpuvvvpBotK1ZBfnVpKWQU/F86dvhOD6uLY6NM13ry9zFLb9IgrzGsDYVLA6LidU6Mzd0bcuvfZfI70BT08YauknrhoGZPYbbpDhiIm8nbnJ7vC1zxXtLxHKATDG3uLJSGAC5GEurERvW67Fi0oKWqcqzGoal6sW83awsPm5bUW+b1LyFxzJrIJni7+1hcbmPQB8Ph870CvLzxH/jX8KO50X7Duv0cphb8BYAXqsr7aLvKLpT3CuYqbkkl5TEWUNS86vEpoy7mo8iK1reyQ76i8xsNHRaZGFcU9zdVMjv52U2UDFXnd1SBWRLdG/y5FaBB4BAGUv/mDpXmQvixLzy7R7ce5KmF/QYVnC2V/qCJX4mlkZyd1MZDbHaytxv2rB6ezcXmwEGuEAANGfOHISGhsLHxwcNGzbEoUOHzO4/e/ZsVKpUCb6+vggJCcHw4cPx7Jn+B0vuMV1Jn0bmu9ltXbxT17sWZp2VzC9tDFdszFkJTcJMXzx7NrRcv8hW+Xw8tb1qusXYjlx7iANx9zHmT+NSA/Y+IUkVVa0YvDzctHktW4a3wOIBDexWMwTI7rGTo1igNzrWlLfwra6QAr4omd/XKGHenqKqSe9xtfeq8KZIuVExLC5oDd1kZMOcMF3f9KwNABjf0bqeFN3f3z/RLS3ubzgbtXkF6cVRTd1AfiuzsvWpW8mo/8U/egHV9E365QbkLLRqC3OBqzXT/M0R68XUeK+lfskUc8UalaJoALR8+XJER0cjJiYGR48eRa1atRAVFYW7d8VrxPz2228YOXIkYmJicPbsWSxcuBDLly/HZ599ZvUxcxrDHiDddVXMEcu+t5RfECAxsPGUkNmv6dlqbqKL3x5B1JxeppdZ6F5f/t2HuYDKEsNp4z3nHzSq+3Hhi3bYJuEE7whP0jJxanyUNq+lYrF82l4V3erEtrj5UF4OwP6Rbayazt4zPDu49XB3w64REdj0YQtZ647JMbNbbcx703gCgSlyyzRYQ0pu0nydxWqtFVb4RWDi62X677V15WK48EU79H++zEG3+vJ6OHV7kKQEd4bD1f7e0qd9myp8aO3acLqjXpoFnzXk9IzZEqeYWzdPzpCVrTaflr6GoVIUDYBmzpyJgQMHYsCAAahatSrmzZsHPz8/LFq0SHT//fv3o2nTpujVqxdCQ0Px0ksvoWfPnno9PHKPqZT7Vi5poVtTJrptRZQrIu1ut311+fWDLkucIWJqtWJdmjvn70wEKYbBi9xFSQHzQZQ1M0VsuYOXsiigl4eb7JXupTCcMSemeskgk1NSrVkOQIy/zBwvuW+3pqr46A4vks493LOLWhpW/LYXf28PvGzwt7RrRITJ/ef3lR4sWUvsIl7foHigPRLPi+nUErM0rVr3syV3BtJlncRaa4atvT3c0aic6TIWunqFiweP5axcosFcHpqchVajJQyfhpso1WGuyr4T4x+n9YDaQrEAKD09HbGxsYiMjHzRGDc3REZG4sAB8ZkJTZo0QWxsrDbguXz5MjZs2ID27dtbfUwASEtLQ3Jyst6Xo+kWBLSWpXWhdJlaT8ac6BXSpjpLSS7WzC4ylatjeMEyLOYlhblgwrBAmxSGd0vrPmiGQ6OlzSSLbuv4/I+/hogvt2B48ROTkWn6RG24JtXW4eaL2Zli6gRoaiaj3GBwYf8GuDKlvSIl9XVfg7mp6uYu/paKBEplGCTM7FYLH0aKz9C0RcuKRRAc6IPmFQojxEIFa11iSyhYw9ywm6GlbzdEVQnnvOAg8XNXISt77nRzgMz1xBiqUzq/3nB4j3DLQ/Z+Jnq6zJ3rn9kxhQIA/jCzhJA9V6l3FMUCoHv37iErKwvFiun3HhQrVgzx8eJdZ7169cLEiRPRrFkzeHp6IiwsDBEREdohMGuOCQBTpkxBUFCQ9iskxPHJWvb4bFhaMkCXn4SLhO66RHJIuXCZS4wWKxn/XoT8BS/NtSK/n/wAyPACXr1kkOgioGLk5r/I9ffQZkbr8GhI6X3KMlOkxHAZB2uXXzH1Gd8wLHvWmW6yprUVzs199kzlcfSUcHGxZPmgRqhQNMCoBpQYU8MsYsuJAMbru8nVsVYJk8snSGLm3HTwszZY+nZDWYeTs0TKUzO5SpVk1HnydHdDHwtDudVKBFr1WTD3mP2XXsx+NawHZm5pjDXvN8Wv7zRE8SAfDI+sKCl4SkiWP4pwxM4lKMzl21U0OG9Yu+C0IymeBC3Hzp07MXnyZHz//fc4evQoVq9ejfXr1+Pzzz+36bijRo1CUlKS9uvGDccsOKpL7irwYlpJWFBUo2u9UggvWxCfvGS6Z+LvE7dtbpMp5rqGM0WCo4JW9ADZe5aXK3fhVgw2feIxVcxQl6l11gyZ6mWSRvx3HuTniatTO+CMTk2nglb00FnSsVYJTH3NOECxx6oZpQr4YWt0S/RyQHJ938ahWPp2OGLHRFreWYQK2Rcfsdcuhb2r9U6yMLNV1+V7pofdG5aVl5PXrX6I2SrS64c1Rz4fy7O9uhqVZjB9Llv/3x3t/w3fx5Wx4tcVzXBb8SBfHBjVxqbeO7HyELr+Om7fc7y5m61uBnmX9lgw194UC4AKFy4Md3d3JCQk6G1PSEhAcLD43eDYsWPRp08fvPPOO6hRowa6dOmCyZMnY8qUKVCr1VYdEwC8vb0RGBio9+VoP+2/avJnpu7sDUm509fw8XTHikGNMbS1/bvGAcu9UVLWoXI1YjVC7MXWHBVza4BJSeC09Nk5NSEKR8e2lfRZNDU0IeVX/lXXWmhYtqBRKQV7ERtKyO9r/2DLnAmdxHt0zAWhzSsUkTUM841Ob5emV0zKMIqu9yPC0L5GsMW1t+QKKeiHw6MjcWai+IK1ZXXybcz1FH/ykrz1+dzdVHi/pfyeZA3NMPyYDlW1M816NAiR3JNhmPRsal0tucv5aIidU+WUhxjQNNSq59VlqncTMA4A5QxhOotiAZCXlxfq1auHbdteVLhVq9XYtm0bGjduLPqY1NRUuBksv+Dunn2yFwTBqmMqxVwP0M8DwiVdIO3xcZJSQ0SjhJkFVc0VTQMgqUve1Awxpdiz5IChr20oKmjpDl1KYGxpHaoAbw/RXjixuiym7uwaS5hF93q9Ulg+qLHVORdSGBZgS7ViSri5z74lpgIRucU2zelUqwR6NSyNQS3LWd1z+b+XK+P73vUcUi+rSD5v+JmYOaY708tcPS9fL3cs6Cuv2vcTkeVGpNo/qjVOxLyEID9PrBrcBJ93ro5xHavijXohmPpaDYtLo0xc92JdPHM3gOUlTmQxZDjMPqZDFRN7ipPS+2WJuSFow4+RqaWTlCQpe3Dt2rWSD9ipUyfJ+0ZHR6Nfv36oX78+wsPDMXv2bKSkpGDAgAEAgL59+6JkyZKYMmUKAKBjx46YOXMm6tSpg4YNG+LSpUsYO3YsOnbsqA2ELB3TVZxPMF0VNMjPEwESpnLaYwbRuFeqYomZ3iiNEzEvIdDMLCsfT3eEhxY0OdtEykVZysrmuYVhEFO+aIDJBUYN7fu0tc3PX9bK+kPjOlY1WpDX1PRea/KuHGHt0KZ6C8E+emq+wrmYnSNa2dSG/k1C8du/1/WL/Nl5qElKPpIrqlEyCHsu3gNgebZmmyryCmDakiDv7eGuPScVC/TRq9HWI7w0fj903dRDjSw3swagnNwmXRNfrYbt516Ud5FSMkCXqYKJUiwZ0MBi8VnDWW+eHq7XAyTp09G5c2e971UqlV5Eq3shzsqSfnfVvXt3JCYmYty4cYiPj0ft2rWxadMmbRLz9evX9Xp8xowZA5VKhTFjxuDWrVsoUqQIOnbsiEmTJkk+pqtIfGw+gS1DQs0Ie9yoSb3bs/RhB4DX65W0abrtiKhK2HH+LgY2l7n6swU1S7leCXbDu/Rf3m6ISRvOol/jMvhh92VsPZNg4pG2T2V9LyLM6t4Hsd4esUVzbTm52pvhjcLU12pi9dFbso5ha7AyvlM1jO9UDaEj19t0nNxoSKvy+H5n9np5dwxq5xiSe9Nn7qbNVsevP5K8r7mbTKkpD4YMbyql1Pjp0SBEuyCzNTNtNSIkDLUZLpfi6YI9QJJapFartV9btmxB7dq1sXHjRjx69AiPHj3Chg0bULduXWzatEl2A4YOHYpr164hLS0N//77Lxo2fDHDYOfOnViyZIn2ew8PD8TExODSpUt4+vQprl+/jjlz5iB//vySj5lTSAk4nFnUSorKwbblTlUKzodzn7+Mke2kFXeUauKrtq/i/UMf+9ZzMfzVBQf54NuedVA/tKDZgo6A7b93qcUzxRjWrypXxB+hBjVTYjpW1UtwdjX27nmRQ87MzbxCt17UPAkFGzvXlj5b1RF1tjSO3RDP6RFzzgHrgBlOHpFyXojp+CIfTcoQtS0Mq8rnihygjz76CF9//TWioqK0CcNRUVGYOXMmhg0b5og25klSFjnNMDOV2R7k9jDZo6dFTmK3JYc+a4PV7zexy/RLOTU9pDBXkt7SBVpKpdtKVk5dtyTYIBcmPukZ3mqqv6bcAIPvczp71hlaN6yZ3Y6VG+kurWCqh2K8iaRyZwspIL0Wkik/2nBjZbjGoJQOFl8vd+we0QrrPmiGUnZovzllCurfGLnb8dxuL7JbFBcXZ9TjAgBBQUG4evWqHZpEANDARJVPPQ6eWCW3erIj77akej8ie/2Z3g1Lo2igj91mtNi7t83Se9XOTOVuKblS9UPtO5NHw7AHIzU9y+plA5TQwIr3ZbWFqsdyhBUJwJbhLXDoM2kFNfOywQZrSWkE2iF51x6irKxdpat0IeuDEMMeIKkTbUsX8tNbq9BRDG/kHDmpxFqyA6AGDRogOjpab6p5QkICRowYgfBwx6y/kxcV8PfCnv+ZT7z0caE8C1fxyUuVsH5YM7sMe+kyV8TMGpZm6nzfuy42ftjc6uM7qgqr2Fpdpio7uxLN6t5fdJaeKHxy/EvY/nFLo4JutqpYLB+K2lKoMI8I9BXvedPNW3RGxXVTpCz1Mmr1SYc9v2HP5D9nTecNWmKpCO7QVvLLCRie41yxFIrsAGjhwoW4c+cOSpcujfLly6N8+fIoXbo0bt26hYULFzqijXlWSEE/bP84e7HM/CJVY13lTsiSS5Pa6X0fanDXM/31mnZ7Ljc3FaqVCLJ7EUNnF0VUqVSiJe0X9Zc2DVhKEr01xMbxfXNAIH5gZBvs/bSVdsbN973rWkw+DfTxlLzWHtmfuR7ozR+1wKh2lc0WOnQ0c2tuafx+6AbumVn30ZaYwHBY3pEZEZ9EyavBJCbMBf+WZAdAFSpUwMmTJ/H3339j2LBhGDZsGNatW4f//vsP5ctbX3QqrxvWRrxAYbkiAbg6tQPaVtHPCbLXgpWAfi0gw3FlWzUtX8io12B0h6p633dr4PilR2yVakM9EXtqXVnabMZ9l+455PkNc7SKBTp+tXN78PVy18t5aF+juI1VrnOGsy6cjG6JuQCjUnA+DGoZpmjpDHNFAHWZ6401V/hRruM3Hln9WDm9M9YmMzcoK22BWmeSFQBlZGTAw8MDp0+fxksvvaQNgNq2besS+R851ZgOVfCRiQBIw3BKodSaMVLo9m6ILUthi0L+xhfIJg6efeAYOevzrVtl+ETMS9r/R1SyrQK14VCgYS9VTpvlFFXtRUBZzsraSK7M18s9xwZBNq1nJsLU6unWklpC5OZD01P77XnZjEu0/ppw12BdsbY6k3AM1+mzVETVFCUWLbZEVgDk6emJ0qVLy6r1Q8YMawC907ycxT8mKes7WUv3omXv3BGxGUuuvMaWKa7QZjnrTj1+9qLHysfTDb+83RDtqgdj+hu2DTcavg/mZrPlBKV1VjVPy3C9JE17cOS5Iyfx83ZHGxnrJ9rL0Wump8vb83RravFfKU7fTtL7/n9RlfDTW+FoVz3YqOK1q5VfsYXsIbDRo0fjs88+w4MH9l1VNi8xt+KxKY6MnnVPkPbuARJbTkH3Irrug5wxLfhBivyVl+3tcxmJ3bpd624qFZpVKIy5b9aTvJq9KYY9vYbfOyr52lHe0Sm6WdjOpQ5chakEft1goKELDk/Y287ziVjQrz6+7lEb+0baXk1dKsM/Cd1zuT3/XuQsjm2oeQX9nuEKxfKhZcUimPtmPaNlanLWX7h5sgOg7777Drt370aJEiVQqVIl1K1bV++LLEvUSYorIJLcLGZOb8e9t7ol3s/cTrbrsX1Exuh1ew1c+WZCd3ikko1FHu3B2l4oR/bSJD8vd6+pW5XTagAVC/RB1efDeG/ULantDc1NAYGp9ATdejr/Xsn9N7T5vD2gUqnwau2STp25mGUQ5LxWt6T2/1VFJjpYy5ZyFGFF/Y3WzDNU9PkNgu55MaeT3a1guCwGyTdz63nZj7FngUBDur005+OTEW7Hk7/YuVd3uM/BtRxt0rdxKDafzp5aas8TprO74R0ZZGqWPvmmZx0cu/7Iqjo7SlsxuDFO30pCg9CCaBxWCL8cvK6tJ5Vbbf6ohV7PxNbhLZRrjJPUKaPMZ9Mw0XncK1XRoUZxVC0RKFpWwplWDW6MjafiMaRVeXSqVRIDFh/CR5HipQW2Dm+Ji3cfo55C76MjyA6AYmJiHNGOPOViwotkNVfoTtS9Q7T3H6SlacRi0/tdhb2rP2tEv+Tc2iXOmKDg4+nu8NL6jhLg7YGGz9c4K180n8tUGnaUc5+/DB9Pd9x48GJiRUhBx1YFzssMZ9Z6uLtpP29Kqx9aEPWfJ4dXCs6H/aNMF+gM8vPU7ptbuF5t6jzgrk4StLW1fCpbuYKwJZqib1fvpWi3yen9MBw6MFXR+Mc+9TDt9RoufeKtWCwfJnepIbn2jiFTpQ2qlZBehVWTgNi9vuuXCqCcQTNUUqqAL9rXCMYb9UrlqGre1lJqtL1ooLfVC55aokRSd24iuwcoKysLs2bNwooVK3D9+nWkp6fr/ZzJ0fJ80dm6isX2XlzPTZVdeEyzCGvEjJ3anzUsJz3q/753XSw7fAMbT93By9WCTfY+vGSHMvLOIGfmlaFSdhg2692wDJqXL4JSBVyn2nL9MgVwxMzMFnI9+Xw88PhZpt5wqEqlwve97bvIryuzZ80dOYrm80GFogE4ceMRhpsYXrJWuxrFse3cXbseMy+R3QM0YcIEzJw5E927d0dSUhKio6Px2muvwc3NDePHj3dAE3O3klZe2Ow9K0xzbrj5MNXoZ3JOHIUCvDGkVXms+6A5hrY2X9sot6tc3LiXrq2ERW4NlS7kJ7nmiDN81a2W0k0gmX4f2AiNyxXCmvdzf/FHUx6mplveyQHm7YrTzvz19rTvoIvrnBXMs7TAs1Jkt+rXX3/F/Pnz8fHHH8PDwwM9e/bEggULMG7cOBw8eNARbczVTK14bMmh0Y5ZTLH/4sMOOW5eVLNUfqNttqz+LEez8oUddmxNLyHlHNVLBuH3dxuhtoOGYnICw+nczpL4JA3r/7sDAFh55IZdj62pll23dH67Htde1g5titaVi2K9i5Y7kd2NEB8fjxo1shcUDAgIQFJSdgGlV155BWPHjrVv63KpcoX9cfl5jk1+P/kB0Aety8PPy3lVNXPKXUZO4KyK6UvfDseaY7fsOqNPg1Xfyd5GtquMqRvPOeTYRfN54+7jNKt6X+3BT2etvLjEFDN7ylfA3wvnPn8ZXgrPJjOlZqn8WNS/gdLNMEn2u1aqVCncuZMdzYaFhWHLli0AgMOHD8PbO3cWErM3zcq7dVw0ajdU3UJ9CHI9KpUKr9Utpbf+lb24QlVsyl38DRbUndHVfsOsGz9sjoX96qNXuPX5fHLM7l5b7/v6ZRw7c8rH092lhshzEtkBUJcuXbBt2zYAwAcffICxY8eiQoUK6Nu3L9566y27NzA30iwPkZFlXREcZyfEdqpdwqnPR66N51qyN91p4b0alsYb9UrZ7diFArzRpkoxpwXuhrl/lgoMknJkj6NMnTpV+//u3bujTJky2L9/PypUqICOHTvatXG5lWYtFbnF9Za+HY6Dl+/jjXrOnRJdRKGxc3JNuWktIHINFYvlw89vhUMA0LKibQv2uppnmVw701XZnEjSqFEjNGrUyB5tyTM0FVj9ZebxNK9QxGjNFmdgzod95JbqwgyAyBFa5LLARyM1jQGQq5IdAJUuXRoRERFo2bIlIiIiEBaWO07qzqRdAI/XkTylXfXiSjfBLnSHEno6Ka+CyNXVCsmvt66ixpO0TO3/rZ31S44hOwdo8uTJ8PHxwbRp01ChQgWEhITgzTffxPz583Hx4kVHtDHX0VTV4Z103uLmmhM1ZNNNpfj90HXlGkLkQv4a0lQ0d+nrbS+uixs+bO7MJpEFsnuA3nzzTbz55psAgDt37mDXrl1Yt24d3n//fajVamRlsbvPEk0PkKsmk/5zJkHpJuRKgiss/GYHHBIl0lc8yEf7/xSdHh9Dfl65f8mRnMSqHKDU1FTs3bsXO3fuxI4dO3Ds2DFUr14dERERdm5e7qS5ELpqD9A7Px9Rugm50unbSbmupEDV4oFKN4FIMcfHtcWfx26hq85afebWVfN00Xo9eZXsAKhJkyY4duwYqlSpgoiICIwcORItWrRAgQIFHNG+XEmzOjDvpPOWAO/cV0H593c5AYLyrvx+XujftKzeNnM9vQyAXIvsAOjcuXPw9/dH5cqVUblyZVSpUoXBj0xqbQ+Qsu3QVaqAL24+fKp0M3K1xmGFLO+UQ1yZ0h5qgUURiQxVKR6IKsUDcfZOstHP+PfiWmSHo/fv38f27dvRqFEjbN68GU2bNkXJkiXRq1cvzJ8/3xFtzHVe5AC5zh9D/yahSjch1/Nwd53ft61UKhVP5kQi3N1U2DDMNde+In2yAyCVSoWaNWti2LBhWLVqFTZu3Ii2bdti5cqVGDx4sCPamOsILpgEvfLITdHtn7xU0cktyb3cXSjgJSLHYXpDziB7COzo0aPYuXMndu7cib179+Lx48eoUaMGPvjgA7Rs2dIRbcx1NENgrvRH0jM8BOP/PmO0vV2N3FG7xhWwx4SIyHXIDoDCw8NRp04dtGzZEgMHDkSLFi0QFJS7ZrY4mmYIzIXiH5Ozk8KKBDi5JbkXAyAiItchOwB68OABAgM59dUWrlgIkasJO4aXuxvSny96yyEwIiLXITsHKDAwEI8ePcKCBQswatQoPHjwAED20NitW7fs3sDcyBULIbpSMJabHB4Tqf0/g0wiItchuwfo5MmTaNOmDfLnz4+rV69i4MCBKFiwIFavXo3r16/j559/dkQ7cxVXLITIa7NjBPl64tznL3P4i4jIxcjuAYqOjsaAAQNw8eJF+Pi8KP/dvn177N69266Ny61O304CANx85Dp1d56mcwkTR/HxdGcBNKI8pligt9JNIAtkn5UPHz6MQYMGGW0vWbIk4uPj7dKo3G7fpfsAgPUn7yjckhcSn6QZbRNb2ZiIiCx7tXZJpZtAFsgOgLy9vZGcbFzh8sKFCyhSpIhdGkXOV7qgn9E23sEQEVmnU60SSjeBLJAdAHXq1AkTJ05ERkYGgOxaNtevX8enn36K119/3e4NzI00Kwe3quQ6AWNGlvECNrll9XIiImeLS3yidBPIAtkB0FdffYUnT56gaNGiePr0KVq2bIny5csjICAAkyZNckQbc52m5QsDAMLLus7aUJdF/lh9vUyvakxERKYlP81QuglkgexZYEFBQdi6dSv27t2LkydP4smTJ6hbty4iIyMtP5gAABcSHgMAnqZnKtySFwJ9jVcq792QOUBERNZIF+lVJ9ciOwDSaNasGZo1e7Hg29GjRzFu3DisW7fOLg3LzU7ezJ4FNn/PFUS/VEnh1mQznJJfs1QQe4CIiKxUt3R+ve97hoco0xAySdYQ2ObNm/HJJ5/gs88+w+XLlwEA586dQ+fOndGgQQOo1WqHNDK3eprhOlPPG4fpD8ctGRCuUEuIiHK+fD4v+hcqFA1ATMdqCraGxEjuAVq4cKG26OHDhw+xYMECzJw5Ex988AG6d++OU6dOoUqVKo5sa67TrX4ppZugFeCt/1Eo6O+lUEuIiHK+wgEvZtEuHtAAPp7sUXc1knuAvv76a0ybNg337t3DihUrcO/ePXz//ff477//MG/ePAY/MpQr7A/AuNeFiIhyh/x+XnitTkm0qx6MUgWMy4yQ8iT3AMXFxaFr164AgNdeew0eHh748ssvUaqU6/Ri5BSX76UAyC6I2KUO3z8iotxoZvfaSjeBzJDcA/T06VP4+WVHsSqVCt7e3ihevLjDGpYX7LmYqHQTRFUtHqh0E4iIiBxK1iywBQsWICAgAACQmZmJJUuWoHDhwnr7DBs2zH6ty+VqlAxSugmiPD24bhUREeVukgOg0qVLY/78+drvg4ODsXTpUr19VCoVAyAZervoWluFmABNRES5nOQA6OrVqw5sRt5SOTgfzsU/hqeba/W0zHuzHpbsv4IvOldXuilEREQOZXUhRLKeZo0tg9qDinu5ejBerh6sdDOIiIgczrW6IPIIAdkRkIvFP0RERHkGAyAFXL2Xmv0fRkBERESKYACkgPSs7CVD4u4ar8BOREREjscASEHnn68KT0RERM5lVQAUFxeHMWPGoGfPnrh79y4AYOPGjTh9+rTsY82ZMwehoaHw8fFBw4YNcejQIZP7RkREQKVSGX116NBBu8+TJ08wdOhQlCpVCr6+vqhatSrmzZsn/0U6gYpjYERERIqQHQDt2rULNWrUwL///ovVq1fjyZPsYZwTJ04gJiZG1rGWL1+O6OhoxMTE4OjRo6hVqxaioqK0QZWh1atX486dO9qvU6dOwd3dXbtEBwBER0dj06ZN+OWXX3D27Fl89NFHGDp0KNauXSv3pTqcq80CIyIiyitkB0AjR47EF198ga1bt8LL60XBvNatW+PgwYOyjjVz5kwMHDgQAwYM0PbU+Pn5YdGiRaL7FyxYEMHBwdqvrVu3ws/PTy8A2r9/P/r164eIiAiEhobi3XffRa1atcz2LCnFjREQERGRImQHQP/99x+6dOlitL1o0aK4d++e5OOkp6cjNjYWkZGRLxrj5obIyEgcOHBA0jEWLlyIHj16wN/fX7utSZMmWLt2LW7dugVBELBjxw5cuHABL730ksnjpKWlITk5We/LGRj/EBERKUN2AJQ/f37cuXPHaPuxY8dQsmRJyce5d+8esrKyUKxYMb3txYoVQ3x8vMXHHzp0CKdOncI777yjt/3bb79F1apVUapUKXh5eeHll1/GnDlz0KJFC5PHmjJlCoKCgrRfISEhkl+HLQK8WYeSiIhICbIDoB49euDTTz9FfHw8VCoV1Go19u3bh08++QR9+/Z1RBtFLVy4EDVq1EB4eLje9m+//RYHDx7E2rVrERsbi6+++gpDhgzBP//8Y/JYo0aNQlJSkvbrxo0bjm4+AOC1uqWc8jxERESkT3YXxOTJkzFkyBCEhIQgKysLVatWRVZWFnr16oUxY8ZIPk7hwoXh7u6OhIQEve0JCQkIDja/HENKSgqWLVuGiRMn6m1/+vQpPvvsM6xZs0Y7M6xmzZo4fvw4ZsyYoTfcpsvb2xve3t6S224rbw83pGWq4enOMTAiIiIlyO4B8vLywvz58xEXF4d169bhl19+wblz57B06VK4u7vLOk69evWwbds27Ta1Wo1t27ahcePGZh+7cuVKpKWl4c0339TbnpGRgYyMDLgZLDLq7u4OtVotuW2O9nwpMKiYBERERKQIq5NQSpcujdKlS9v05NHR0ejXrx/q16+P8PBwzJ49GykpKRgwYAAAoG/fvihZsiSmTJmi97iFCxeic+fOKFSokN72wMBAtGzZEiNGjICvry/KlCmDXbt24eeff8bMmTNtaqs9pWdmB2NujH+IiIgUITsAio6OFt2uUqng4+OD8uXL49VXX0XBggUtHqt79+5ITEzEuHHjEB8fj9q1a2PTpk3axOjr168b9eacP38ee/fuxZYtW0SPuWzZMowaNQq9e/fGgwcPUKZMGUyaNAmDBw+W+Uod72FKBooH+SrdDCIiojxHJQiCYHm3F1q1aoWjR48iKysLlSpVAgBcuHAB7u7uqFy5Ms6fPw+VSoW9e/eiatWqDmm0oyUnJyMoKAhJSUkIDAy0+/FDR64HABwa3QZF8/nY/fhERER5kZzrt+wcoFdffRWRkZG4ffs2YmNjERsbi5s3b6Jt27bo2bMnbt26hRYtWmD48OFWv4DcTDfe9HDjUmxERERKkH0F/vLLL/H555/rRVZBQUEYP348pk+fDj8/P4wbNw6xsbF2bWhuodbpb2MOEBERkTJkB0BJSUmia3UlJiZqKyjnz58f6enptrcuF1Lr9ABxFhgREZEyrBoCe+utt7BmzRrcvHkTN2/exJo1a/D222+jc+fOALKrNFesWNHebc0VsnS6gNgDREREpAzZs8B++OEHDB8+HD169EBmZmb2QTw80K9fP8yaNQsAULlyZSxYsMC+Lc0ldFPO3RkBERERKUJ2ABQQEID58+dj1qxZuHz5MgCgXLlyCAgI0O5Tu3ZtuzUwt9EdAuNq8ERERMqwuhBiQEAAatasac+25An6OUAKNoSIiCgPsyoAOnLkCFasWIHr168bJTuvXr3aLg3LrfRngTECIiIiUoLsJOhly5ahSZMmOHv2LNasWYOMjAycPn0a27dvR1BQkCPamKuo1RwCIyIiUprsAGjy5MmYNWsW/v77b3h5eeHrr7/GuXPn0K1bN5vXBssL9HOAFGwIERFRHiY7AIqLi0OHDh0AZK/onpKSApVKheHDh+PHH3+0ewNzG00HkErFOkBERERKkR0AFShQAI8fPwYAlCxZEqdOnQIAPHr0CKmpqfZtXS6kWQqDw19ERETKkZ0E3aJFC2zduhU1atRA165d8eGHH2L79u3YunUr2rRp44g25iqaHiDdgohERETkXLIDoO+++w7Pnj0DAIwePRqenp7Yv38/Xn/9dYwZM8buDcxtzt5JVroJREREeZ6sACgzMxPr1q1DVFQUAMDNzQ0jR450SMNyqzKF/JRuAhERUZ4nKwfIw8MDgwcP1vYAkXyaxOd83lbXoCQiIiIbyU6CDg8Px/Hjxx3QlLxBMw3ejXPgiYiIFCO7G+L9999HdHQ0bty4gXr16sHf31/v51wew7wXs8AUbggREVEeJjsA6tGjBwBg2LBh2m0qlQqCIEClUiErK8t+rcuFstTZ/3IaPBERkXJkB0BXrlxxRDvyDA6BERERKU92AFSmTBlHtCPPUHMIjIiISHGyk6ABYOnSpWjatClKlCiBa9euAQBmz56Nv/76y66Ny43UHAIjIiJSnOwAaO7cuYiOjkb79u3x6NEjbc5P/vz5MXv2bHu3L9dRcykMIiIixckOgL799lvMnz8fo0ePhru7u3Z7/fr18d9//9m1cbnRixwghRtCRESUh8m+DF+5cgV16tQx2u7t7Y2UlBS7NCo3Yw8QERGR8mQHQGXLlhUthLhp0yZUqVLFHm3K1TRroLozACIiIlKM7Flg0dHRGDJkCJ49ewZBEHDo0CH8/vvvmDJlChYsWOCINuYqT9IyAQCMf4iIiJQjOwB655134OvrizFjxiA1NRW9evVCiRIl8PXXX2uLJJJp8UnZ66jFJXK4kIiISClWrcjZu3dv9O7dG6mpqXjy5AmKFi1q73blWjFrTyvdBCIiojxPdg7QF198oa0G7efnx+BHpvRMtdJNICIiyvNkB0ArV65E+fLl0aRJE3z//fe4d++eI9pFRERE5DCyA6ATJ07g5MmTiIiIwIwZM1CiRAl06NABv/32G1JTUx3RxlzF053Zz0REREqzqhxftWrVMHnyZFy+fBk7duxAaGgoPvroIwQHB9u7fbnOsNYVlG4CERFRnmdzPWJ/f3/4+vrCy8sLGRkZ9mhTrlYwwAsAEFWtmMItISIiyrusCoCuXLmCSZMmoVq1aqhfvz6OHTuGCRMmID4+3t7ty3U0hRBZCZqIiEg5sqfBN2rUCIcPH0bNmjUxYMAA9OzZEyVLlnRE23IlgUthEBERKU52ANSmTRssWrQIVatW1duuVquxYcMGvPLKK3ZrXG6k1nQBMf4hIiJSjOwAaNKkSXrfX7p0CYsWLcKSJUuQmJjIPCALnoc/7AEiIiJSkFU5QE+fPsXPP/+MFi1aoFKlSti/fz/GjRuHmzdv2rt9uc6LHCBl20FERJSXyeoBOnz4MBYsWIBly5YhLCwMvXv3xv79+/H9998bDYmROOYAERERKU9yAFSzZk0kJyejV69e2L9/P6pVqwYAGDlypMMalxupnwdAjH+IiIiUI3kI7Pz582jRogVatWrF3h4bvMiBZgRERESkFMkB0OXLl1GpUiW89957KFWqFD755BMcO3YMKnZlyCIwB4iIiEhxkgOgkiVLYvTo0bh06RKWLl2K+Ph4NG3aFJmZmViyZAkuXLjgyHbmGmrmABERESnOqllgrVu3xi+//II7d+7gu+++w/bt21G5cmXUrFnT3u3LdRIfpwEA3GxehISIiIisZdNlOCgoCO+//z6OHDmCo0ePIiIiwk7Nyr28PbPf8oTkNIVbQkRElHfZrR+idu3a+Oabb+x1uFzL19MdABDgLbsGJREREdkJB2KcTJMEHeTrqWxDiIiI8jAGQE4msA4QERGR4hgAOZlmLTDGP0RERMphAORkmiEw1k8iIiJSjuxMXFOJziqVCj4+PihfvjxatGgBd3d3mxuXGwnaPiAiIiJSiuwAaNasWUhMTERqaioKFCgAAHj48CH8/PwQEBCAu3fvoly5ctixYwdCQkLs3uCc7kUPkLLtICIiystkD4FNnjwZDRo0wMWLF3H//n3cv38fFy5cQMOGDfH111/j+vXrCA4OxvDhwyUdb86cOQgNDYWPjw8aNmyIQ4cOmdw3IiICKpXK6KtDhw56+509exadOnVCUFAQ/P390aBBA1y/fl3uS3WIFzlAjICIiIiUIjsAGjNmDGbNmoWwsDDttvLly2PGjBkYNWoUSpUqhenTp2Pfvn0Wj7V8+XJER0cjJiYGR48eRa1atRAVFYW7d++K7r969WrcuXNH+3Xq1Cm4u7uja9eu2n3i4uLQrFkzVK5cGTt37sTJkycxduxY+Pj4yH2pDsG1wIiIiJQnewjszp07yMzMNNqemZmJ+Ph4AECJEiXw+PFji8eaOXMmBg4ciAEDBgAA5s2bh/Xr12PRokUYOXKk0f4FCxbU+37ZsmXw8/PTC4BGjx6N9u3bY/r06dptusGa0jgNnoiISHmye4BatWqFQYMG4dixY9ptx44dw3vvvYfWrVsDAP777z+ULVvW7HHS09MRGxuLyMjIF41xc0NkZCQOHDggqS0LFy5Ejx494O/vDwBQq9VYv349KlasiKioKBQtWhQNGzbEn3/+afY4aWlpSE5O1vtyFO0QGCMgIiIixcgOgBYuXIiCBQuiXr168Pb2hre3N+rXr4+CBQti4cKFAICAgAB89dVXZo9z7949ZGVloVixYnrbixUrpu1JMufQoUM4deoU3nnnHe22u3fv4smTJ5g6dSpefvllbNmyBV26dMFrr72GXbt2mTzWlClTEBQUpP1yZPK2tgfIYc9ARERElsgeAgsODsbWrVtx7tw5XLhwAQBQqVIlVKpUSbtPq1at7NdCExYuXIgaNWogPDxcu02tVgMAXn31VW0Sdu3atbF//37MmzcPLVu2FD3WqFGjEB0drf0+OTnZYUHQkWsPAQDnEywPERIREZFjWL0iZ+XKlVG5cmWrn7hw4cJwd3dHQkKC3vaEhAQEBwebfWxKSgqWLVuGiRMnGh3Tw8MDVatW1dtepUoV7N271+TxND1ZznDs+iMAwM7ziU55PiIiIjImOwDKysrCkiVLsG3bNty9e1fb66Kxfft2Scfx8vJCvXr1sG3bNnTu3BlAdg/Otm3bMHToULOPXblyJdLS0vDmm28aHbNBgwY4f/683vYLFy6gTJkyktpFREREuZ/sAOjDDz/EkiVL0KFDB1SvXt2mZN7o6Gj069cP9evXR3h4OGbPno2UlBTtrLC+ffuiZMmSmDJlit7jFi5ciM6dO6NQoUJGxxwxYgS6d++OFi1aoFWrVti0aRP+/vtv7Ny50+p2EhERUe4iOwBatmwZVqxYgfbt29v85N27d0diYiLGjRuH+Ph41K5dG5s2bdImRl+/fh1ubvp52ufPn8fevXuxZcsW0WN26dIF8+bNw5QpUzBs2DBUqlQJf/zxB5o1a2Zze4mIiCh3UAmaaUkSlShRAjt37kTFihUd1SbFJScnIygoCElJSQgMDLTrsUNHrtf+/+rUDmb2JCIiIjnkXL9lT4P/+OOP8fXXX0Nm3ERERETkMmQPge3duxc7duzAxo0bUa1aNXh6eur9fPXq1XZrHBEREZEjyA6A8ufPjy5dujiiLUREREROITsAWrx4sSPaQUREROQ0snOAyD6aljeewk9ERETOIakHqG7duti2bRsKFCiAOnXqmK39c/ToUbs1LjdqXqEw9ly8h9fqlFK6KURERHmWpADo1Vdf1S4VoanaTLZxY98bERGRYiQFQDExMaL/J/k01QPcbKigTURERLaxejHU9PR00bXASpcubXOjcjP18wjIliVEiIiIyDayA6ALFy7g7bffxv79+/W2C4IAlUqFrKwsuzUuN9IEQG6Mf4iIiBQjOwAaMGAAPDw8sG7dOhQvXpw9GTKpOQRGRESkONkB0PHjxxEbG4vKlSs7oj25nsAeICIiIsXJnotUtWpV3Lt3zxFtyRM0PUDsOSMiIlKO7ABo2rRp+N///oedO3fi/v37SE5O1vsi87RJ0Aq3g4iIKC+TPQQWGRkJAGjTpo3ediZBS8McICIiIuXJDoB27NjhiHbkHZocIBZCJCIiUozsAKhly5aOaEeewRwgIiIi5VlVCPHRo0c4dOiQaCHEvn372qVhudWLOkAMgIiIiJQiOwD6+++/0bt3bzx58gSBgYF6PRkqlYoBkAUvcoCUbQcREVFeJjsT5eOPP8Zbb72FJ0+e4NGjR3j48KH268GDB45oY64isAeIiIhIcbIDoFu3bmHYsGHw8/NzRHtyvXPxjwFwGjwREZGSZAdAUVFROHLkiCPakqccuHxf6SYQERHlWbJzgDp06IARI0bgzJkzqFGjBjw9PfV+3qlTJ7s1LjerH1pQ6SYQERHlWbIDoIEDBwIAJk6caPQzFkK0rFxhf1y+lwI/L3elm0JERJRnyQ6ADKe9kzxcCoOIiEh5rEfsZM9nwYOTwIiIiJQjuwdIbOhL17hx46xuTF4gaCIg9gEREREpRnYAtGbNGr3vMzIycOXKFXh4eCAsLIwBkAXC8z4g9gAREREpR3YAdOzYMaNtycnJ6N+/P7p06WKXRuVmAleDJyIiUpxdcoACAwMxYcIEjB071h6Hy9U0ARDDHyIiIuXYLQk6KSkJSUlJ9jpcrqVZCoMdQERERMqRPQT2zTff6H0vCALu3LmDpUuXol27dnZrWG6lnQXGPiAiIiLFyA6AZs2apfe9m5sbihQpgn79+mHUqFF2a1hupR0CY/xDRESkGNkB0JUrV0z+7OnTpzY1Ji/gLDAiIiLl2SUHKC0tDTNnzkTZsmXtcbhcTa1NgmYEREREpBTJAVBaWhpGjRqF+vXro0mTJvjzzz8BAIsWLULZsmUxa9YsDB8+3FHtzDU4BEZERKQ8yUNg48aNww8//IDIyEjs378fXbt2xYABA3Dw4EHMnDkTXbt2hbs7F/i0JPP5WmoMgIiIiJQjOQBauXIlfv75Z3Tq1AmnTp1CzZo1kZmZiRMnTkDFq7lkj1IzAACPn2Uq3BIiIqK8S/IQ2M2bN1GvXj0AQPXq1eHt7Y3hw4cz+LHSqiM3lW4CERFRniU5AMrKyoKXl5f2ew8PDwQEBDikUXlBhWJ874iIiJQieQhMEAT0798f3t7eAIBnz55h8ODB8Pf319tv9erV9m1hLuXjyXwpIiIipUgOgPr166f3/Ztvvmn3xuQlXu52W4WEiIiIZJIcAC1evNiR7chzPNyZO0VERKQUdkMoxIM9QERERIrhVVghXuwBIiIiUgwDIIV4sgeIiIhIMbwKK4QBEBERkXJ4FXYitWYlVDAAIiIiUhKvwk6UJegGQMwBIiIiUgoDICfKzGIPEBERkSvgVdiJ0rPU2v8zACIiIlIOr8JOlKETAHl5cAiMiIhIKQyAnCiDPUBEREQuwSWuwnPmzEFoaCh8fHzQsGFDHDp0yOS+ERERUKlURl8dOnQQ3X/w4MFQqVSYPXu2g1ovXUbmixwgVoImIiJSjuJX4eXLlyM6OhoxMTE4evQoatWqhaioKNy9e1d0/9WrV+POnTvar1OnTsHd3R1du3Y12nfNmjU4ePAgSpQo4eiXIYleDpAbh8CIiIiUongANHPmTAwcOBADBgxA1apVMW/ePPj5+WHRokWi+xcsWBDBwcHar61bt8LPz88oALp16xY++OAD/Prrr/D09HTGS7Eo0NdD5/+u0SYiIqK8SNEAKD09HbGxsYiMjNRuc3NzQ2RkJA4cOCDpGAsXLkSPHj3g7++v3aZWq9GnTx+MGDEC1apVs3iMtLQ0JCcn6305gqfbi7fbgz1AREREilE0ALp37x6ysrJQrFgxve3FihVDfHy8xccfOnQIp06dwjvvvKO3fdq0afDw8MCwYcMktWPKlCkICgrSfoWEhEh/ETIIOv9XqRgAERERKUXxITBbLFy4EDVq1EB4eLh2W2xsLL7++mssWbJEcpAxatQoJCUlab9u3LjhkPYKOpWg2QFERESkHEUDoMKFC8Pd3R0JCQl62xMSEhAcHGz2sSkpKVi2bBnefvttve179uzB3bt3Ubp0aXh4eMDDwwPXrl3Dxx9/jNDQUNFjeXt7IzAwUO/LEXSWAmMPEBERkYIUDYC8vLxQr149bNu2TbtNrVZj27ZtaNy4sdnHrly5EmlpaXjzzTf1tvfp0wcnT57E8ePHtV8lSpTAiBEjsHnzZoe8DqkEvUEwIiIiUoqH5V0cKzo6Gv369UP9+vURHh6O2bNnIyUlBQMGDAAA9O3bFyVLlsSUKVP0Hrdw4UJ07twZhQoV0tteqFAho22enp4IDg5GpUqVHPtiLHke/3D4i4iISFmKB0Ddu3dHYmIixo0bh/j4eNSuXRubNm3SJkZfv34dbm76HVXnz5/H3r17sWXLFiWabDXNEBiHv4iIiJSlEnQzcwkAkJycjKCgICQlJdk1H+hO0lM0nrIdHm4qXJrc3m7HJSIiInnX7xw9CyynEbRDYOwBIiIiUhIDICfSdrUx/iEiIlIUAyAnUj9PAmL8Q0REpCwGQArgEBgREZGyGAA5kaCdBaZsO4iIiPI6BkBOpBY4BEZEROQKGAA5kSYJmkNgREREymIA5ETakkuMf4iIiBTFAMiJND1AjH+IiIiUxQDIiTQ9QG5cDIyIiEhRDICciCNgREREroEBkBNph8CYBE1ERKQoBkBOpJkGzxEwIiIiZTEAciKBi4ERERG5BAZATsRK0ERERK6BAZATcQiMiIjINTAAUoCKQ2BERESKYgDkRBwCIyIicg0MgJxIgGYIjBEQERGRkhgAOVFKWhYA4Najpwq3hIiIKG9jAOREFxIeK90EIiIiAgMgp6peMlDpJhAREREYADmV+nkSdLnC/so2hIiIKI9jAOREWc8jIOZAExERKYsBkBNpCiG6sxIiERGRohgAOZGmDhCnwRMRESmLAZATaYbAMtWChT2JiIjIkRgAOdGBy/cBAJfuPlG4JURERHkbAyAnWnH4htJNICIiIjAAcqq3mpVVuglEREQEBkBOVfZ5/Z8GoQUUbgkREVHexgDIiTTT4FWcBUZERKQoBkBOpJkGz/CHiIhIWQyAnEgz+Z0dQERERMpiAOREwvMuIBZCJCIiUhYDICfSDoEx/iEiIlIUAyAnEp4PgqmYBURERKQoBkBOxB4gIiIi18AAyIleBECMgIiIiJTEAMiJtHWAFG4HERFRXscAyIk4DZ6IiMg1MABypucREKfBExERKYsBkBO9mAVGRERESmIA5ERqzgIjIiJyCQyAnIizwIiIiFwDAyAn4hAYERGRa2AA5EQshEhEROQaGAA5kSBwKQwiIiJXwADIiTR1gNz4rhMRESmKl2In0g6BsQeIiIhIUQyAnEh4EQERERGRghgAOZGalaCJiIhcAgMgJ9KuBaZoK4iIiMglAqA5c+YgNDQUPj4+aNiwIQ4dOmRy34iICKhUKqOvDh06AAAyMjLw6aefokaNGvD390eJEiXQt29f3L5921kvxyTtLDBGQERERIpSPABavnw5oqOjERMTg6NHj6JWrVqIiorC3bt3RfdfvXo17ty5o/06deoU3N3d0bVrVwBAamoqjh49irFjx+Lo0aNYvXo1zp8/j06dOjnzZZnF+IeIiEhZHko3YObMmRg4cCAGDBgAAJg3bx7Wr1+PRYsWYeTIkUb7FyxYUO/7ZcuWwc/PTxsABQUFYevWrXr7fPfddwgPD8f169dRunRpB70Sy9TPe4CYA0RERKQsRXuA0tPTERsbi8jISO02Nzc3REZG4sCBA5KOsXDhQvTo0QP+/v4m90lKSoJKpUL+/PltbbJNBCYBERERuQRFe4Du3buHrKwsFCtWTG97sWLFcO7cOYuPP3ToEE6dOoWFCxea3OfZs2f49NNP0bNnTwQGBoruk5aWhrS0NO33ycnJEl+BPMnPMgCwDhAREZHSFM8BssXChQtRo0YNhIeHi/48IyMD3bp1gyAImDt3rsnjTJkyBUFBQdqvkJAQh7R378V7AJgETUREpDRFA6DChQvD3d0dCQkJetsTEhIQHBxs9rEpKSlYtmwZ3n77bdGfa4Kfa9euYevWrSZ7fwBg1KhRSEpK0n7duHFD/ouRoEPN4gjy9URklaIOOT4RERFJo2gA5OXlhXr16mHbtm3abWq1Gtu2bUPjxo3NPnblypVIS0vDm2++afQzTfBz8eJF/PPPPyhUqJDZY3l7eyMwMFDvyxHebRGGEzEv4eXqxR1yfCIiIpJG8Vlg0dHR6NevH+rXr4/w8HDMnj0bKSkp2llhffv2RcmSJTFlyhS9xy1cuBCdO3c2Cm4yMjLwxhtv4OjRo1i3bh2ysrIQHx8PIHsGmZeXl3NeGBEREbksxQOg7t27IzExEePGjUN8fDxq166NTZs2aROjr1+/DjeD5dPPnz+PvXv3YsuWLUbHu3XrFtauXQsAqF27tt7PduzYgYiICIe8DiIiIso5VIJ2hU7SSE5ORlBQEJKSkhw2HEZERET2Jef6naNngRERERFZgwEQERER5TkMgIiIiCjPYQBEREREeQ4DICIiIspzGAARERFRnsMAiIiIiPIcBkBERESU5zAAIiIiojyHARARERHlOQyAiIiIKM9RfDFUV6RZHi05OVnhlhAREZFUmuu2lGVOGQCJePz4MQAgJCRE4ZYQERGRXI8fP0ZQUJDZfbgavAi1Wo3bt28jX758UKlUdj12cnIyQkJCcOPGDa4072B8r52H77Xz8L12Hr7XzmOv91oQBDx+/BglSpSAm5v5LB/2AIlwc3NDqVKlHPocgYGB/INyEr7XzsP32nn4XjsP32vnscd7bannR4NJ0ERERJTnMAAiIiKiPIcBkJN5e3sjJiYG3t7eSjcl1+N77Tx8r52H77Xz8L12HiXeayZBExERUZ7DHiAiIiLKcxgAERERUZ7DAIiIiIjyHAZARERElOcwAHKAOXPmIDQ0FD4+PmjYsCEOHTpkdv+VK1eicuXK8PHxQY0aNbBhwwYntTTnk/Nez58/H82bN0eBAgVQoEABREZGWvzd0AtyP9cay5Ytg0qlQufOnR3bwFxE7nv96NEjDBkyBMWLF4e3tzcqVqzI84hEct/r2bNno1KlSvD19UVISAiGDx+OZ8+eOam1Odfu3bvRsWNHlChRAiqVCn/++afFx+zcuRN169aFt7c3ypcvjyVLlti3UQLZ1bJlywQvLy9h0aJFwunTp4WBAwcK+fPnFxISEkT337dvn+Du7i5Mnz5dOHPmjDBmzBjB09NT+O+//5zc8pxH7nvdq1cvYc6cOcKxY8eEs2fPCv379xeCgoKEmzdvOrnlOY/c91rjypUrQsmSJYXmzZsLr776qnMam8PJfa/T0tKE+vXrC+3btxf27t0rXLlyRdi5c6dw/PhxJ7c855H7Xv/666+Ct7e38OuvvwpXrlwRNm/eLBQvXlwYPny4k1ue82zYsEEYPXq0sHr1agGAsGbNGrP7X758WfDz8xOio6OFM2fOCN9++63g7u4ubNq0yW5tYgBkZ+Hh4cKQIUO032dlZQklSpQQpkyZIrp/t27dhA4dOuhta9iwoTBo0CCHtjM3kPteG8rMzBTy5csn/PTTT45qYq5hzXudmZkpNGnSRFiwYIHQr18/BkASyX2v586dK5QrV05IT093VhNzDbnv9ZAhQ4TWrVvrbYuOjhaaNm3q0HbmNlICoP/9739CtWrV9LZ1795diIqKsls7OARmR+np6YiNjUVkZKR2m5ubGyIjI3HgwAHRxxw4cEBvfwCIiooyuT9ls+a9NpSamoqMjAwULFjQUc3MFax9rydOnIiiRYvi7bffdkYzcwVr3uu1a9eicePGGDJkCIoVK4bq1atj8uTJyMrKclazcyRr3usmTZogNjZWO0x2+fJlbNiwAe3bt3dKm/MSZ1wbuRiqHd27dw9ZWVkoVqyY3vZixYrh3Llzoo+Jj48X3T8+Pt5h7cwNrHmvDX366acoUaKE0R8Z6bPmvd67dy8WLlyI48ePO6GFuYc17/Xly5exfft29O7dGxs2bMClS5fw/vvvIyMjAzExMc5odo5kzXvdq1cv3Lt3D82aNYMgCMjMzMTgwYPx2WefOaPJeYqpa2NycjKePn0KX19fm5+DPUCUJ02dOhXLli3DmjVr4OPjo3RzcpXHjx+jT58+mD9/PgoXLqx0c3I9tVqNokWL4scff0S9evXQvXt3jB49GvPmzVO6abnOzp07MXnyZHz//fc4evQoVq9ejfXr1+Pzzz9XumlkBfYA2VHhwoXh7u6OhIQEve0JCQkIDg4WfUxwcLCs/SmbNe+1xowZMzB16lT8888/qFmzpiObmSvIfa/j4uJw9epVdOzYUbtNrVYDADw8PHD+/HmEhYU5ttE5lDWf6+LFi8PT0xPu7u7abVWqVEF8fDzS09Ph5eXl0DbnVNa812PHjkWfPn3wzjvvAABq1KiBlJQUvPvuuxg9ejTc3NinYC+mro2BgYF26f0B2ANkV15eXqhXrx62bdum3aZWq7Ft2zY0btxY9DGNGzfW2x8Atm7danJ/ymbNew0A06dPx+eff45Nmzahfv36zmhqjif3va5cuTL+++8/HD9+XPvVqVMntGrVCsePH0dISIgzm5+jWPO5btq0KS5duqQNMgHgwoULKF68OIMfM6x5r1NTU42CHE3gKXBZTbtyyrXRbunUJAhC9rRKb29vYcmSJcKZM2eEd999V8ifP78QHx8vCIIg9OnTRxg5cqR2/3379gkeHh7CjBkzhLNnzwoxMTGcBi+R3Pd66tSpgpeXl7Bq1Srhzp072q/Hjx8r9RJyDLnvtSHOApNO7nt9/fp1IV++fMLQoUOF8+fPC+vWrROKFi0qfPHFF0q9hBxD7nsdExMj5MuXT/j999+Fy5cvC1u2bBHCwsKEbt26KfUScozHjx8Lx44dE44dOyYAEGbOnCkcO3ZMuHbtmiAIgjBy5EihT58+2v010+BHjBghnD17VpgzZw6nwecE3377rVC6dGnBy8tLCA8PFw4ePKj9WcuWLYV+/frp7b9ixQqhYsWKgpeXl1CtWjVh/fr1Tm5xziXnvS5TpowAwOgrJibG+Q3PgeR+rnUxAJJH7nu9f/9+oWHDhoK3t7dQrlw5YdKkSUJmZqaTW50zyXmvMzIyhPHjxwthYWGCj4+PEBISIrz//vvCw4cPnd/wHGbHjh2i51/N+9uvXz+hZcuWRo+pXbu24OXlJZQrV05YvHixXdukEgT22xEREVHewhwgIiIiynMYABEREVGewwCIiIiI8hwGQERERJTnMAAiIiKiPIcBEBEREeU5DICIiIgoz2EARERkR1evXoVKpcLx48eVbgqRy9m9ezc6duyIEiVKQKVS4c8//5R9DEEQMGPGDFSsWBHe3t4oWbIkJk2aJPs4DICIyCaJiYl47733ULp0aXh7eyM4OBhRUVHYt2+fdh9rT3TW6N+/P1QqFaZOnaq3/c8//4RKpXJKG4hIXEpKCmrVqoU5c+ZYfYwPP/wQCxYswIwZM3Du3DmsXbsW4eHhso/D1eCJyCavv/460tPT8dNPP6FcuXJISEjAtm3bcP/+fcXa5OPjg2nTpmHQoEEoUKCAYu2wJ67sTrlBu3bt0K5dO5M/T0tLw+jRo/H777/j0aNHqF69OqZNm4aIiAgAwNmzZzF37lycOnUKlSpVAgCULVvWqrawB4iIrPbo0SPs2bMH06ZNQ6tWrVCmTBmEh4dj1KhR6NSpEwAgNDQUANClSxeoVCrt9wDw119/oW7duvDx8UG5cuUwYcIEZGZman+uUqkwd+5ctGvXDr6+vihXrhxWrVplsV2RkZEIDg7GlClTTO4zfvx41K5dW2/b7Nmz9drXv39/dO7cGZMnT0axYsWQP39+TJw4EZmZmRgxYgQKFiyIUqVKYfHixUbHP3fuHJo0aQIfHx9Ur14du3bt0vv5qVOn0K5dOwQEBKBYsWLo06cP7t27p/15REQEhg4dio8++giFCxdGVFSUxddNlNMNHToUBw4cwLJly3Dy5El07doVL7/8Mi5evAgA+Pvvv1GuXDmsW7cOZcuWRWhoKN555x08ePBA9nMxACIiqwUEBCAgIAB//vkn0tLSRPc5fPgwAGDx4sW4c+eO9vs9e/agb9+++PDDD3HmzBn88MMPWLJkidFY/tixY/H666/jxIkT6N27N3r06IGzZ8+abZe7uzsmT56Mb7/9Fjdv3rTpNW7fvh23b9/G7t27MXPmTMTExOCVV15BgQIF8O+//2Lw4MEYNGiQ0fOMGDECH3/8MY4dO4bGjRujY8eO2l6xR48eoXXr1qhTpw6OHDmCTZs2ISEhAd26ddM7xk8//QQvLy/s27cP8+bNs+l1ELm669evY/HixVi5ciWaN2+OsLAwfPLJJ2jWrJn2JuPy5cu4du0aVq5ciZ9//hlLlixBbGws3njjDflPaNelVYkoz1m1apVQoEABwcfHR2jSpIkwatQo4cSJE3r7ABDWrFmjt61NmzbC5MmT9bYtXbpUKF68uN7jBg8erLdPw4YNhffee89ke3RXnm/UqJHw1ltvCYIgCGvWrBF0T3kxMTFCrVq19B47a9YsoUyZMnrHKlOmjJCVlaXdVqlSJaF58+ba7zMzMwV/f3/h999/FwRBEK5cuSIAEKZOnardJyMjQyhVqpQwbdo0QRAE4fPPPxdeeuklvee+ceOGAEA4f/68IAjZK5HXqVPH5OskyukMzwvr1q0TAAj+/v56Xx4eHkK3bt0EQRCEgQMH6v2dCIIgxMbGCgCEc+fOyXp+5gARkU1ef/11dOjQAXv27MHBgwexceNGTJ8+HQsWLED//v1NPu7EiRPYt2+fXo9PVlYWnj17htTUVPj5+QEAGjdurPe4xo0bS55hNW3aNLRu3RqffPKJ7NelUa1aNbi5vegsL1asGKpXr6793t3dHYUKFcLdu3eN2qnh4eGB+vXra3uuTpw4gR07diAgIMDo+eLi4lCxYkUAQL169axuN1FO8+TJE7i7uyM2Nhbu7u56P9P8rRQvXhweHh7avxEAqFKlCoDsHiRNXpAUDICIyGY+Pj5o27Yt2rZti7Fjx+Kdd95BTEyM2QDoyZMnmDBhAl577TXR49lDixYtEBUVhVGjRhm1xc3NDdk3oS9kZGQYHcPT01Pve5VKJbpNrVZLbteTJ0/QsWNHTJs2zehnxYsX1/7f399f8jGJcro6deogKysLd+/eRfPmzUX3adq0KTIzMxEXF4ewsDAAwIULFwAAZcqUkfV8DICIyO6qVq2qN+3d09MTWVlZevvUrVsX58+fR/ny5c0e6+DBg+jbt6/e93Xq1JHclqlTp6J27dpGd4ZFihRBfHw8BEHQTo+3Z+2egwcPokWLFgCAzMxMxMbGYujQoQCyX/sff/yB0NBQeHjwNEx5x5MnT3Dp0iXt91euXMHx48dRsGBBVKxYEb1790bfvn3x1VdfoU6dOkhMTMS2bdtQs2ZNdOjQAZGRkahbty7eeustzJ49G2q1GkOGDEHbtm31eoWkYBI0EVnt/v37aN26NX755RecPHkSV65cwcqVKzF9+nS8+uqr2v1CQ0Oxbds2xMfH4+HDhwCAcePG4eeff8aECRNw+vRpnD17FsuWLcOYMWP0nmPlypVYtGgRLly4gJiYGBw6dEgbSEhRo0YN9O7dG998843e9oiICCQmJmL69OmIi4vDnDlzsHHjRhveDX1z5szBmjVrcO7cOQwZMgQPHz7EW2+9BQAYMmQIHjx4gJ49e+Lw4cOIi4vD5s2bMWDAAKNAkSg3OXLkCOrUqaO9iYmOjkadOnUwbtw4ANmTJfr27YuPP/4YlSpVQufOnXH48GGULl0aQHbP7d9//43ChQujRYsW6NChA6pUqYJly5bJb4wd8piIKI969uyZMHLkSKFu3bpCUFCQ4OfnJ1SqVEkYM2aMkJqaqt1v7dq1Qvny5QUPDw+9JONNmzYJTZo0EXx9fYXAwEAhPDxc+PHHH7U/ByDMmTNHaNu2reDt7S2EhoYKy5cvN9sm3SRojStXrgheXl6C4Slv7ty5QkhIiODv7y/07dtXmDRpklEStOGxWrZsKXz44Yd628qUKSPMmjVL+1wAhN9++00IDw8XvLy8hKpVqwrbt2/Xe8yFCxeELl26CPnz5xd8fX2FypUrCx999JGgVqtNPg8R2Y9KEAwGwYmIXIRKpcKaNWvQuXNnpZtCRLkMh8CIiIgoz2EARERERHkOpx8QkcviCD0ROQp7gIiIiCjPYQBEREREeQ4DICIiIspzGAARERFRnsMAiIiIiPIcBkBERESU5zAAIiIiojyHARARERHlOQyAiIiIKM/5P3po60gB8L1/AAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Plotting running average reward for CliffWalking-v0\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj4AAAHHCAYAAAC/R1LgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACAuElEQVR4nO3dd1zU9R8H8Nexjr0UBJUhOHBgKu6Jk1zlKGeOLNOy5ag0V1bOTG2YmZaWWab+tHJn7pkbJyrLzVBkicz7/v7AO24v7jjgXs/Hg8fjvvPefBXuzWe9RYIgCCAiIiKyAjaWDoCIiIiorDDxISIiIqvBxIeIiIisBhMfIiIishpMfIiIiMhqMPEhIiIiq8HEh4iIiKwGEx8iIiKyGkx8iIiIyGow8SEyAZFIhE8++cTSYZAVO3jwIEQiEQ4ePGjpUCxm7dq1EIlESExMtHQoVI4x8aFyS/pLTPplZ2eHGjVqYPTo0bh3756lwyt3rl27BpFIBEdHR6Snp1s6nHJl9OjRCv+XxGIx6tati1mzZiE3N9fS4VnElStX8Morr6BGjRoQi8WoXr06hg8fjitXrlg6NAWRkZEK/3aavviHB+nLztIBEOny6aefolatWsjNzcXJkyexdu1aHD16FJcvX4ajo6OlwwMAPH36FHZ2lv1x+vXXX+Hn54fHjx9j8+bNeP311y0aT3kjFouxevVqAEBGRgb++usvfPbZZ4iLi8P69estHF3Z2rJlC4YOHQpvb2+89tprqFWrFhITE/Hjjz9i8+bN2LBhA/r372/pMAEA06dPV/i/fPr0aXz99df4+OOPUb9+fdn+xo0bo2HDhhgyZAjEYrElQqWKQiAqp9asWSMAEE6fPq2w/6OPPhIACH/88YeFIit/JBKJEBwcLEyaNEno37+/EBkZaZE4njx5YpH31WXUqFGCi4uLwj6JRCK0bt1aEIlEQlJSkoUi059EIhFycnI0Hj9w4IAAQDhw4IDW+8TGxgrOzs5CWFiYkJKSonAsNTVVCAsLE1xcXIS4uDhThK237Oxsvc7btGmTXt8nkSbs6qIKp0OHDgCAuLg42b7IyEhERkaqnDt69GgEBwfLthMTEyESibB48WL88MMPCA0NhVgsRosWLXD69GmVa11dXXHv3j3069cPrq6u8PHxwZQpU1BUVKRwrnJT+yeffAKRSITY2FiMHj0anp6e8PDwwKuvvoqcnByFa58+fYp3330XVatWhZubG1544QXcu3fPoOb7Y8eOITExEUOGDMGQIUNw+PBh3L17V3a8T58+CAkJUXttmzZt0Lx5c4V9v/76KyIiIuDk5ARvb28MGTIEd+7cUTgnMjISjRo1wtmzZ9GxY0c4Ozvj448/BgD89ddf6N27N6pXrw6xWIzQ0FB89tlnKs8NAJYvX46QkBA4OTmhZcuWOHLkiNp/z7y8PMyePRu1a9eGWCxGQEAAPvzwQ+Tl5en1jJSJRCK0b98egiAgPj5e4diuXbvQoUMHuLi4wM3NDb1791boAvr7778hEolw8eJF2b7//e9/EIlEGDBggMK96tevj8GDB8u216xZgy5dusDX1xdisRgNGjTAihUrVOILDg5Gnz59sGfPHjRv3hxOTk5YuXIlAODu3bvo168fXFxc4Ovri4kTJ+r9HL744gvk5OTghx9+gI+Pj8KxqlWrYuXKlXjy5AkWLVoEANi8eTNEIhEOHTqkcq+VK1dCJBLh8uXLsn0xMTF46aWX4O3tDUdHRzRv3hx///23wnXSbuxDhw7hrbfegq+vL2rWrKlX/NqoG+MjfY4HDx6UPcfw8HDZWKgtW7YgPDwcjo6OiIiIwPnz51Xuq8/3RBUHEx+qcKS/1Ly8vIy+x2+//YYvvvgC48aNw+eff47ExEQMGDAABQUFCucVFRUhKioKVapUweLFi9GpUyd8+eWX+OGHH/R6n0GDBiErKwvz58/HoEGDsHbtWsyZM0fhnNGjR+Obb75Br169sHDhQjg5OaF3794GfT/r169HaGgoWrRogb59+8LZ2Rm///677PjgwYORkJCgktzdunULJ0+exJAhQ2T75s6di5EjR6JOnTpYsmQJ3n//fezbtw8dO3ZUGTv06NEj9OzZE02aNMGyZcvQuXNnAMUfQK6urpg0aRK++uorREREYNasWZg6darC9StWrMDbb7+NmjVrYtGiRejQoQP69eunkLQBgEQiwQsvvIDFixejb9+++Oabb9CvXz8sXbpUIakwlLr/S+vWrUPv3r3h6uqKhQsXYubMmbh69Srat28vO799+/YQiUQ4fPiw7LojR47AxsYGR48ele1LTU1FTEwMOnbsqPA9BwUF4eOPP8aXX36JgIAAvPXWW1i+fLlKfNevX8fQoUPRvXt3fPXVV2jSpAmePn2Krl27Ys+ePXj77bcxffp0HDlyBB9++KFe3/O2bdsQHBws+wNCWceOHREcHIwdO3YAgOxZbNy4UeXcP/74Aw0bNkSjRo0AFI8bat26Na5du4apU6fiyy+/hIuLC/r164etW7eqXP/WW2/h6tWrav9vmFJsbCyGDRuGvn37Yv78+Xj8+DH69u2L9evXY+LEiXjllVcwZ84cxMXFYdCgQZBIJLJrDf2eqAKwdJMTkSbSrq5///1XSE1NFe7cuSNs3rxZ8PHxEcRisXDnzh3ZuZ06dRI6deqkco9Ro0YJQUFBsu2EhAQBgFClShUhLS1Ntv+vv/4SAAjbtm1TuBaA8Omnnyrcs2nTpkJERITCPgDC7NmzZduzZ88WAAhjxoxROK9///5ClSpVZNtnz54VAAjvv/++wnmjR49Wuacm+fn5QpUqVYTp06fL9g0bNkx47rnnZNsZGRmCWCwWJk+erHDtokWLBJFIJNy6dUsQBEFITEwUbG1thblz5yqcd+nSJcHOzk5hf6dOnQQAwvfff68Sk7oumXHjxgnOzs5Cbm6uIAiCkJeXJ1SpUkVo0aKFUFBQIDtv7dq1AgCFf89169YJNjY2wpEjRxTu+f333wsAhGPHjml6PIIglHR1paamCqmpqUJsbKywePFiQSQSCY0aNRIkEokgCIKQlZUleHp6CmPHjlW4PikpSfDw8FDY37BhQ2HQoEGy7WbNmgkvv/yyAEC4du2aIAiCsGXLFgGAEB0drfXZREVFCSEhIQr7goKCBADC7t27FfYvW7ZMACBs3LhRtu/JkydC7dq1dXYBpaenCwCEF198UeM5giAIL7zwggBAyMzMFARBEIYOHSr4+voKhYWFsnMePHgg2NjYKPx8dO3aVQgPD5f9GwtCcRdd27ZthTp16sj2SX+227dvr3BPfWjr6pLeNyEhQbZP+hyPHz8u27dnzx4BgODk5CT7vy8IgrBy5UqVe+v7PVHFwRYfKve6desGHx8fBAQE4KWXXoKLiwv+/vvvUjWNDx48WOGvfOlfv8pdHgAwfvx4he0OHTqoPU8dddc+evQImZmZAIDdu3cDKP7LV94777yj1/2B4m6ZR48eYejQobJ9Q4cORXR0tKx7xt3dHT179sTGjRshCILsvD/++AOtW7dGYGAggOJmf4lEgkGDBuHhw4eyLz8/P9SpUwcHDhxQeG+xWIxXX31VJSYnJyfZ66ysLDx8+BAdOnRATk4OYmJiAABnzpzBo0ePMHbsWIWB4cOHD1dpzdu0aRPq16+PsLAwhbi6dOkCACpxqfPkyRP4+PjAx8cHtWvXxpQpU9CuXTv89ddfEIlEAIC9e/ciPT0dQ4cOVXgfW1tbtGrVSuF9OnTogCNHjsi+x+joaLzxxhuoWrWqbP+RI0fg6ekpaxFRfjYZGRl4+PAhOnXqhPj4eGRkZCjEXKtWLURFRSns27lzJ/z9/fHSSy/J9jk7O+ONN97Q+QyysrIAAG5ublrPkx6X/j8dPHgwUlJSFKbKb968GRKJRNbilpaWhv3798taOaXP7tGjR4iKisLNmzdVZmOOHTsWtra2OuMurQYNGqBNmzay7VatWgEAunTpIvu/L79f+vNtzPdE5R9ndVG5t3z5ctStWxcZGRn46aefcPjw4VLP2pD/ZQeUdHU8fvxYYb+jo6PKOAgvLy+V84x5H3d3d9y6dQs2NjaoVauWwnm1a9fW6/5A8XicWrVqQSwWIzY2FgAQGhoKZ2dnrF+/HvPmzQNQ/OH1559/4sSJE2jbti3i4uJw9uxZLFu2THavmzdvQhAE1KlTR+172dvbK2zXqFEDDg4OKudduXIFM2bMwP79+2UfnlLSD/dbt26p/V7t7OwUxmVJ47p27ZrKv4VUSkqK2v3yHB0dsW3bNgDFY2QWLVqElJQUhUTk5s2bACBLqJS5u7vLXnfo0AHff/89YmNjERcXB5FIhDZt2sgSorFjx+LIkSNo164dbGxK/sY8duwYZs+ejRMnTqiM98rIyICHh4dsW/n/BVD83GrXri1L1qTq1aun8xlIExppAqSJcoL0/PPPw8PDA3/88Qe6du0KoDhpbtKkCerWrQuguDtJEATMnDkTM2fOVHvflJQU1KhRQ+v3Zw7KP4fSZxwQEKB2v/Tn25jvico/Jj5U7rVs2VI2+LZfv35o3749hg0bhuvXr8PV1RVA8UBV+ZYMKXWDaQFo/CtT+R6l/WtU3/cxVmZmJrZt24bc3Fy1ycpvv/2GuXPnQiQSycb+bNy4EW3btsXGjRthY2ODl19+WXa+RCKBSCTCrl271MYufd5S8kmDVHp6Ojp16gR3d3d8+umnCA0NhaOjI86dO4ePPvpIYfyEviQSCcLDw7FkyRK1x5U/wNSxtbVFt27dZNtRUVEICwvDuHHjZANVpbGtW7cOfn5+KveQb5lq3749AODw4cOIj49Hs2bN4OLigg4dOuDrr79GdnY2zp8/j7lz58quiYuLQ9euXREWFoYlS5YgICAADg4O2LlzJ5YuXarybNQ939Lw8PCAv7+/wqBsdS5evIgaNWrIEj2xWCwb0/Ldd98hOTkZx44dkyXVQMmzmzJlikorlZRykmvq708TTT+Hun4+jfmeqPxj4kMViq2tLebPn4/OnTvj22+/lQ2I9PLyUtv9JG1VKK+CgoIgkUiQkJCgkLhIW2502bJlC3Jzc7FixQpUrVpV4dj169cxY8YMHDt2DO3bt4eLiwv69OmDTZs2YcmSJfjjjz/QoUMHVK9eXXZNaGgoBEFArVq1ZH/JG+rgwYN49OgRtmzZojCoNyEhQeG8oKAg2fcqHRQNAIWFhUhMTETjxo0V4oqOjkbXrl1VWjqM5e/vj4kTJ2LOnDk4efIkWrdujdDQUACAr6+vQpKkTmBgIAIDA3HkyBHEx8fLuks7duyISZMmYdOmTSgqKlJ4Btu2bUNeXh7+/vtvhVYIfbrqpIKCgnD58mUIgqDwLK5fv67X9X369MGqVatw9OhRWfIm78iRI0hMTMS4ceMU9g8ePBg///wz9u3bh2vXrkEQBIWB5dJZg/b29jqfXUVRGb8n4qwuqoAiIyPRsmVLLFu2TLbqbmhoKGJiYpCamio7Lzo6GseOHbNUmHqR/hX53XffKez/5ptv9Lr+119/RUhICMaPH4+XXnpJ4WvKlClwdXVVWJxv8ODBuH//PlavXo3o6GiVGVEDBgyAra0t5syZo9IqJQgCHj16pDMm6V/R8tfn5+erfI/NmzdHlSpVsGrVKhQWFsr2r1+/XqUrcdCgQbh37x5WrVql8n5Pnz7FkydPdMalzjvvvANnZ2csWLAAQPG/h7u7O+bNm6cyww+Awv8voLi7a//+/Th16pQs8WnSpAnc3NywYMECODk5ISIiQna+umeTkZGBNWvW6B1zr169cP/+fWzevFm2Tzo9XR8ffPABnJycMG7cOJV/z7S0NIwfPx7Ozs744IMPFI5169YN3t7e+OOPP/DHH3+gZcuWCl1Vvr6+iIyMxMqVK/HgwQOV91V+dhVBZfyeiC0+VEF98MEHePnll7F27VqMHz8eY8aMwZIlSxAVFYXXXnsNKSkp+P7779GwYUOVMSblSUREBAYOHIhly5bh0aNHaN26NQ4dOoQbN24AgNbWjfv37+PAgQN499131R4Xi8WIiorCpk2b8PXXX8Pe3h69evWCm5sbpkyZAltbWwwcOFDhmtDQUHz++eeYNm0aEhMT0a9fP7i5uSEhIQFbt27FG2+8gSlTpmj9ntq2bQsvLy+MGjUK7777LkQiEdatW6eSSDk4OOCTTz7BO++8gy5dumDQoEFITEzE2rVrERoaqvC9jxgxAhs3bsT48eNx4MABtGvXDkVFRYiJicHGjRtla90YqkqVKnj11Vfx3Xff4dq1a6hfvz5WrFiBESNGoFmzZhgyZAh8fHxw+/Zt7NixA+3atcO3334ru75Dhw5Yv369bE0goDi5adu2Lfbs2YPIyEiFMVA9evSAg4MD+vbti3HjxiE7OxurVq2Cr6+v2g9WdcaOHYtvv/0WI0eOxNmzZ+Hv749169bB2dlZr+vr1KmDn3/+GcOHD0d4eLjKys0PHz7E77//Lmv9krK3t8eAAQOwYcMGPHnyBIsXL1a59/Lly9G+fXuEh4dj7NixCAkJQXJyMk6cOIG7d+8iOjparxjLk8r4PVm9sp9IRqQfTSs3C4IgFBUVCaGhoUJoaKhsOuyvv/4qhISECA4ODkKTJk2EPXv2aJzO/sUXX6jcE0rTx9Wt9isIJVPVtV0rPSc1NVXt9yQ/3fbJkyfChAkTBG9vb8HV1VXo16+fcP36dQGAsGDBAo3P58svvxQACPv27dN4jnRq+F9//SXbN3z4cAGA0K1bN43X/e9//xPat28vuLi4CC4uLkJYWJgwYcIE4fr167JzOnXqJDRs2FDt9ceOHRNat24tODk5CdWrVxc+/PBD2RRi5WnIX3/9tRAUFCSIxWKhZcuWwrFjx4SIiAjh+eefVzgvPz9fWLhwodCwYUNBLBYLXl5eQkREhDBnzhwhIyND4/ciCJr/LQVBEOLi4gRbW1th1KhRsn0HDhwQoqKiBA8PD8HR0VEIDQ0VRo8eLZw5c0bh2itXrggAhPr16yvs//zzzwUAwsyZM1Xe7++//xYaN24sODo6CsHBwcLChQuFn376Se007N69e6uN+datW8ILL7wgODs7C1WrVhXee+89Yffu3QataHzx4kVh6NChgr+/v2Bvby/4+fkJQ4cOFS5duqTxmr179woABJFIpLCchLy4uDhh5MiRgp+fn2Bvby/UqFFD6NOnj7B582bZOdp+tnUxZjq7uucIQJgwYYLCPk2/H/T5nqjiEAmCiUZZEpHJXLhwAU2bNsWvv/6K4cOHWzqcMiWRSODj44MBAwao7doiIioNjvEhsrCnT5+q7Fu2bBlsbGwUBsZWRrm5uSpdYL/88gvS0tLUliAhIiotjvEhsrBFixbh7Nmz6Ny5M+zs7LBr1y7s2rULb7zxhl7TtCuykydPYuLEiXj55ZdRpUoVnDt3Dj/++CMaNWqkMM2eiMhU2NVFZGF79+7FnDlzcPXqVWRnZyMwMBAjRozA9OnTFdaNqYwSExPx7rvv4tSpU0hLS4O3tzd69eqFBQsWwNfX19LhEVElxMSHiIiIrAbH+BAREZHVYOJDREREVqNyDyAwgkQiwf379+Hm5maypfGJiIjIvARBQFZWFqpXr65QGFgZEx8l9+/fr/QzaYiIiCqrO3fuoGbNmhqPM/FR4ubmBqD4wUkrExMREVH5lpmZiYCAANnnuCZMfJRIu7fc3d2Z+BAREVUwuoapcHAzERERWQ0mPkRERGQ1mPgQERGR1WDiQ0RERFaDiQ8RERFZDSY+REREZDWY+BAREZHVYOJDREREVoOJDxEREVkNJj5ERERkNZj4EBERkdVg4kNERERWg4kPERERGSQrtwD5hRJLh2EUJj5EREQm8DS/CG+tP4vvD8Uhr7DI0uGYTVxqNsI/+Qd1Z+yydChGYeJDRESkxvnbj/H6z2dw69ETXL6XgbG/nEFsSrbG87/85zp2XkrCgl0xqDdjNz7eeqkMoy0bgiCg65eHZNu3H+VYMBrjMPEhIiJSo/93x/HvtWR0+uIg+nxzFHuvJuONdWc0nv9fQprC9m//3db5HmlP8vHi8mM4EJNS6niNsfvyA0QtPYzHT/L1Ov/IzYcK29su3jdHWGbFxIeIiEhJQZH68SvxqU80XhPm56ay71F2ntb3afbZXkTfScera08jeOoOBE/dgdyCsusmG//rOVxPzkLTz/bqdb7yc9l6/p45wjIrJj5ERERyzt5KQ53pho9fufv4qcq+ASuOqz034eETXE/KUnus4ew9Br+3NoIgyJKqzNwCAMDdxzkqCVbak3ykZmlP1F77WbHFS13XX2ZuAb7edxMJDzUniZZUaRKfxMREvPbaa6hVqxacnJwQGhqK2bNnIz9fv+Y7IiKyLrEpWZiyKRoZTwtk+wqLJBi44oTe9zh4PQU7Lj6AIAg4Ef9I5fitRzkqrSTpOfnovPggopYdVnvPIomg9/vr43ZayTicxp/8g15fHUH7hQcQNnO3wnnNPtuLFnP/lSVH+npdKRlqNXcfluy9gc6LD+JUQhpqTduhMcmzBDtLB2AqMTExkEgkWLlyJWrXro3Lly9j7NixePLkCRYvXmzp8IiIqBw5nZiGl78vTnA2n72LxAW9AQDH41STF2W5BUVwtLeFIAgYvea0zvMznhagqqsYQHHrS5NP9etW0ue+Hk72OmPt9MVBhX1XH2RqvebWwxyE1/RQ2X/kZqrstZvYDll5hQCAf68lo7BIAjvb4raUp3ItSYNWFj/jqGWHZc/Y0ipNi8/zzz+PNWvWoEePHggJCcELL7yAKVOmYMuWLZYOjYiIygGJpKTLR5r0SN1ILm6RGPnTKZ33OXSjOAFIz1HfMvK/N9sqbO+6nCR7vV/PQcy/n9I+MDp46g48N+cf/HA4Tut5S/+9odf7ydt+Sf2A5RE/ljyb1zrUUji29ngigOLETpO9V5MNjsUcKk3io05GRga8vb0tHQYREVnYnbQchHy8U+PxHksP4/0N5/W617h1ZzF/1zWcVNO1BQARQV4K2zP/vAwAOJOYpjJGRpNpWy7hWOxDlf1FEgHZz1paAGDezhhZMpeSlYuc/EKF81MytY/ZUcfeRjU1yFLq/hrfKRQrR0TItr89EAsA6Lf8mMb7jv1Fv+/d3CpNV5ey2NhYfPPNNzq7ufLy8pCXV/IfIzNTexMgERFVLPLdWtr8eUH/qdkrD8VrPZ64oDeCp+6QbR+LfYjhq/9TOc/T2R7nZnSHjY0IABSuGb76Pywf1gzhNTyQ8bQAfb89qvU9W87dBwBwsLXB0amd0WPpYY2tUvI+79cIM54lZwDg5GAra7kRiUSy+OU52tsiqqGfbDs9pwCFRRJE383Q+l6jfjqFTnV9MKZ9La3nmZNI0NYuVQ5MnToVCxcu1HrOtWvXEBYWJtu+d+8eOnXqhMjISKxevVrrtZ988gnmzJmjsj8jIwPu7u7GBU1EROXCb//dNnohwQ+fr4cx7Wrh15O3MLRlIF7+/oTO8TFrX22ByHq+AIC+3xzFpXuaE4GE+b1kiYXU3qvJZd4yopykyftzQjvce/wUE347J9sXN68XbNUkao1reuCiXOIzuXtdfLlXfVfbzbk9YW9r2k6nzMxMeHh46Pz8LveJT2pqKh490j7YLCQkBA4ODgCA+/fvIzIyEq1bt8batWtho6bJTp66Fp+AgAAmPkRElYCmD3QAGN4qEGlP8hXG4EidmdFNNiBZn3ttGt8GRRIBrWp5y5KZgiKJxmnxbUKq4Pc3WqvsFwQBtaZp7pIz1h9vtEatqi5oOW8fnm/oh6+GNsHX+25ieKsgVPd0wsHrKXoN1AagMEj57uMctF94QOWc3e93QJifu6zlSPl7ivnseTja25biO1Klb+JT7ru6fHx84OPjo9e59+7dQ+fOnREREYE1a9boTHoAQCwWQywW6zyPiIgqhoIiCR6k52L0WvUDlbe+1Rb303PRqZ4PXMV2uHg3HS98WzI25dthTVWSHl1aBKuOJ9XWorHopcZq94tEIsTP66V1PJIxWoVUAaCYtHwQVdJTElnPF5O618USDS00mtT0cla7P8yvOPFQbtGSMnXSY4hyn/jo6969e4iMjERQUBAWL16M1NSSaXd+fn5ariQiospE2+KD299pj0Y1PNA0sGRf45qeCuf0bOSv9tr9kzuhi1ydKqkmAZ6qJ2ux9a22CPBWnzAAkI33Uadb/WroEuaLqIbVcOTmQxy6kYrFLz+H60lZ6PX1EbXXqFtRWp13u9bRmfhsf6e9zvv8IDfoWWpE6yCsO3kLALB8WDO94jGXSpP47N27F7GxsYiNjUXNmjUVjpXz3jwiIjIRTYvvJczvhay8Qrg7ql/35uuhTfHu7+fxSutA2fgVZSE+rmjg764yzqdViP6zh3e+2wENquseRnHkw87osEi1C+nTFxuiuqcTAKBf0xro17QGAKBBdXe1Y3UMXTvnqyFN8N6GC2qPdahTFY1qqK7v0zvcHzsuPZBt92io2tjwWb9G+KxfI2TmFmj8Nygr5X6MT1nTt4+QiIjKl7XHEvDJtqsq++v7u2PXex1M+l5FEgGhz7qjLs+JgqtYfTvC/fSnaLtgv2zb0EG98anZ+PP8PXy9Pxb+Ho44Ma2r1vNTsnJls7tWjWyO7g2q6f1eUpvO3MGTvEKVZ7nhjdZo/azLTN7jJ/kKtb4stVBhpRncXNaY+BARVUzqBh+3r10VP49pqbEVpzQycwuQXyjROR5IPq7ysnqxPlIyc2WDoWf0qa9xPA8APMh4itfWnsHKERFau/HMiYmPkZj4EBFVTMqJz5EPO1vsQ1jeibhHWH0kHt+90gxiO8sN6q3sKs2sLiIiImOUh6QHANqEVkGbUNUuIrKMSl2ygoiIKjZBEDDyp1NoNe9frefJD2pu4O9eobqUqGwx8SEionLrQUYuDt9IRXJmHu6k5Wg87376U9nrjePblEVoVEEx8SEionJLviXn0r0MrDmWgIt305FfKFE470FGLgCguoejxhlWRADH+BARUTmWlVtSbfyt9ecUjsl3Z03ZGA0AuP8sASLShC0+RERULuUXSrRWVX+SV5IUPXqSXxYhUSXAxIeIiMqlbw/Eaj1+KjGtjCKhyoSJDxERlUsbT9/RevzVZ9XEcwuKyiIcqiQ4xoeIiMqlpEzd43USHj7B5rMlCdI/EzuaMySqBJj4EBFRuZOlodioss6LDypsW7oAJpV/7OoiIqJyp5lc0Ut5Ih0lt/w8HM0QDVUmTHyIiKhcySssQkFRSRnJ09O7yV5f/iTKEiFRJcKuLiIiKlfm7bimsO3jJlZYs2f9660wfPV/ZR0WVRJs8SEionLl5xO3tB5vV7sq2tVWLfq5emRzc4VElQgTHyIiKrcuftJD7f4lg5qo7OvWoJqZo6HKgF1dRERUbmmapeXrJsZzNT3g6miHX19rBZGuUc9EzzDxISKiCkckEuGvt9tbOgyqgNjVRURE5VKAt5OlQ6BKiIkPERGVS+93rWvpEKgSYuJDRETlUqFEYukQqBJi4kNEROWGIJQsXNiwuocFI6HKioObiYioXLh8LwNpT/Jl20FVnC0YDVVWTHyIiMjipm25hN9P3VbY5yrmRxSZHru6iIjIonILilSSHgBcm4fMgokPERFZVKFE0H0SkYkw8SEiIovKL+TsLSo77EAlIiKLKJIICP14p6XDICvDxIeIiMrUV//exJlbaXijY4ilQyErxMSHiIjKTG5BEZb+ewMA4Oxga+FoyBpxjA8REZWZ8b+elb3ecyVZ5XhUw2oAgHFsDSIzYYsPERGVmYPXU7UeXz6sGWJTs1GvmlsZRUTWhokPERGVibzCIp3n2NnaIMzPvQyiIWvFri4iIioTx2Ifaj0+f0B4GUVC1owtPkREVCbGrD2jdn/8vF4QANjacKVmMj8mPkREZFE2THioDLGri4iILGbf5E6WDoGsDBMfIiKyiD6N/RHq42rpMMjKsKuLiIjK1EfPhyGqYTUEVXGxdChkhZj4EBGR2fX86ojsdecwH4SwpYcshF1dRERkdtceZMpe12bSQxbExIeIiMwqNiVLYdvOlh89ZDns6iIiIpMTBAH9vzuO5kFeiH/4xNLhEMkw8SEiIpNbuPs6LtxJx4U76WgbWkW2/++321kwKiJ2dRERkRl8fyhO9vp43CPZ68Y1PS0QDVEJJj5ERERkNZj4EBERkdVg4kNERERWg4kPERGZlEQiqN3f97nqZRwJkSomPkREZFKxqdlq94f5uZVxJESqmPgQEZFJ2dmI1O5/pVVQGUdCpIrr+BARkUk9zikAAIhEQPy8Xhj7y1k08HeDh7O9hSMjYuJDREQmlvYkHwAgCIBIJMLqUc0tHBFRCXZ1ERGRSR24nmLpEIg0YuJDREQm1TzICwDg7GBr4UiIVDHxISIikzqd+BgAkJNfZOFIiFQx8SEiIpP6/dRtS4dApBETHyIiMpnrSVmWDoFIKyY+RERktEfZeZj112UkZ+YCAKKWHbZwRETaVcrEJy8vD02aNIFIJMKFCxcsHQ4RUaUV8fm/+OXELbSat8/SoRDppVImPh9++CGqV2dNGCKisjTix/8Utns28rNQJESaVbrEZ9euXfjnn3+wePFiS4dCRGRVjtx8qLA9pn0tC0VCpFmlWrk5OTkZY8eOxZ9//glnZ2e9rsnLy0NeXp5sOzMz01zhERFZlaSMXEuHQKSi0rT4CIKA0aNHY/z48WjeXP/l0efPnw8PDw/ZV0BAgBmjJCKyHvVYjZ3KoXKf+EydOhUikUjrV0xMDL755htkZWVh2rRpBt1/2rRpyMjIkH3duXPHTN8JEZH1aFTDHaE+rpYOg0hFue/qmjx5MkaPHq31nJCQEOzfvx8nTpyAWCxWONa8eXMMHz4cP//8s9prxWKxyjVERKSbdAq7suufPw8HWxuIRKIyjohIN70Sn7///lvvG77wwgtGB6OOj48PfHx8dJ739ddf4/PPP5dt379/H1FRUfjjjz/QqlUrk8ZERETAibhHaveL7Viji8ovvRKffv36KWyLRCIIgqCwLVVUZJnaLIGBgQrbrq7FTayhoaGoWbOmJUIiIqrU3v/jgqVDIDKYXmN8JBKJ7Ouff/5BkyZNsGvXLqSnpyM9PR07d+5Es2bNsHv3bnPHS0RERGQ0g8f4vP/++/j+++/Rvn172b6oqCg4OzvjjTfewLVr10waoLGCg4MVWqWIiMj8PJzsLR0CkVYGz+qKi4uDp6enyn4PDw8kJiaaICQiIqqo1rzawtIhEGllcOLTokULTJo0CcnJybJ9ycnJ+OCDD9CyZUuTBkdERBVD9Owe2PluBzQL9LJ0KERaGdzV9eOPP2LAgAEIDAyULfZ3584d1KlTB3/++aep4yMionKuU10feDjZs5uLKgSDE586derg4sWL2Lt3L2JiYgAA9evXR7du3bhmAxGRlZAfQ8mFCqkiMSjxKSgogJOTEy5cuIAePXqgR48e5oqLiIjKsQ2nS1a5f7k5lwyhisOgMT729vYIDAy02Fo9RERUPpxKSJO9DqqiX1FoovLA4MHN06dPx8cff4y0tDTdJxMRUaV0MyVL9porNVNFYvAYn2+//RaxsbGoXr06goKC4OLionD83LlzJguOiIjKp8v3MmWvbW04vpMqDoMTH+XyFUREREQVhcGJz+zZs80RBxERVRCbz961dAhERjN4jA8REVm3KZuiZa871KlqwUiIDGdwi09RURGWLl2KjRs34vbt28jPz1c4zkHPRETW48jNh5YOgcggBrf4zJkzB0uWLMHgwYORkZGBSZMmYcCAAbCxscEnn3xihhCJiKi8+uj5MEuHQGQQgxOf9evXY9WqVZg8eTLs7OwwdOhQrF69GrNmzcLJkyfNESMREZVT4zuFWDoEIoMYnPgkJSUhPDwcAODq6oqMjAwAQJ8+fbBjxw7TRkdEROUaSxVRRWNw4lOzZk08ePAAABAaGop//vkHAHD69GmIxWLTRkdEROVWjwbVLB0CkcEMTnz69++Pffv2AQDeeecdzJw5E3Xq1MHIkSMxZswYkwdIRERlR774qCZNAjwBAC82qWHmaIhMz+BZXQsWLJC9Hjx4MIKCgnD8+HHUqVMHffv2NWlwRERUdh5kPEWb+fsxrFUg5vUP13jehTvpAIC8QtZtpIqn1Ov4tG7dGpMmTWLSQ0RUwS3afR0A8Nt/tzWeUyQpaRE6Gsup7FTxGJz4BAYGYuTIkfjxxx8RFxdnjpiIiMgCsnILZa+vJ2Xh+WWHsedKksI5H2+5JHvduZ5vmcVGZCoGJz7z5s2Do6MjFi5ciDp16iAgIACvvPIKVq1ahZs3b5ojRiIiKgP/XkuWvX79l9OIScrCuHVnAQCPsvMgCAL+OHNHdo6vGye0UMVj8BifV155Ba+88goA4MGDBzh06BC2b9+Ot956CxKJBEVF7PMlIqro7qQ9lb0+EJOCV9eextCWgejeoBr2Xi1OkJoHe1sqPCKjGZz4AEBOTg6OHj2KgwcP4sCBAzh//jwaNWqEyMhIE4dHRERl4VF2nsZjr649DQD4/ZTi2B9bG67hQxWPwYlP27Ztcf78edSvXx+RkZGYOnUqOnbsCC8vL3PER0REZSDh4RNLh0BUJgwe4xMTEwMXFxeEhYUhLCwM9evXZ9JDRFTBPcjItXQIRGXC4MTn0aNH2L9/P1q3bo09e/agXbt2qFGjBoYNG4ZVq1aZI0YiIjKzd34/b+kQiMqESNBnmU4NBEHA2bNn8e2332L9+vWVYnBzZmYmPDw8kJGRAXd3d0uHQ0RUJoKnGlZrsYqLA87O7G6maIgMp+/nt8FjfM6dO4eDBw/i4MGDOHr0KLKyshAeHo533nkHnTp1KlXQRERUMWx+s62lQyAyisGJT8uWLdG0aVN06tQJY8eORceOHeHh4WGO2IiIqByKntUDHs72lg6DyCgGJz5paWnsAiIiqmTC/NwQk5Sl87yohtWY9FCFZvDgZnd3d6Snp2P16tWYNm0a0tLSABR3gd27d8/kARIRkXkJgqA26bkyJ0ph+9V2wfj+lYiyCovILAxu8bl48SK6du0KT09PJCYmYuzYsfD29saWLVtw+/Zt/PLLL+aIk4iIzKTrl4fU7ncRK35EfPR8GEQiLlpIFZvBLT6TJk3Cq6++ips3b8LR0VG2v1evXjh8+LBJgyMiIvPKL5QgXs/FCx1sDf7IICp3DP5ffPr0aYwbN05lf40aNZCUlKTmCiIiKq/+S3iksD29V30AQM9Gfirn2rBEBVUCBnd1icViZGZmquy/ceMGfHx8TBIUERGVjcIixaXcmgZ6In5eL1mS82KT6vjrwn0MbRloifCITM7gFp8XXngBn376KQoKCgAAIpEIt2/fxkcffYSBAweaPEAiIjKfQzdSFbYzcwsUWna+GtIUVz+NwvwB4WUdGpFZGJz4fPnll8jOzoavry+ePn2KTp06oXbt2nB1dcXcuXPNESMREZnJrUeK43saVVddl83ZweDOAaJyy+D/zR4eHti7dy+OHj2KixcvIjs7G82aNUO3bt3MER8REZlR78bVceB6cavPyhER8HV31HEFUcVmdBrfvn17tG/fXrZ97tw5zJo1C9u3bzdJYEREZH7TtlyUvY5qqDqgmaiyMaira8+ePZgyZQo+/vhjxMfHAwBiYmLQr18/tGjRAhKJxCxBEhGReRQUGV2nmqhC0rvF58cff5QtVvj48WOsXr0aS5YswTvvvIPBgwfj8uXLqF+/vjljJSIiIioVvVt8vvrqKyxcuBAPHz7Exo0b8fDhQ3z33Xe4dOkSvv/+eyY9RETlUMbTAsSnZqs9lltQVMbREFme3i0+cXFxePnllwEAAwYMgJ2dHb744gvUrFnTbMEREZFxHmXnYd3JW1j2700AwMe9wvBGx1CFc17+/oQlQiOyKL1bfJ4+fQpnZ2cAxWv3iMVi+Pv7my0wIiIy3pd7b8iSHgCYtzNG5ZxL9zLKMiSicsGgWV2rV6+Gq6srAKCwsBBr165F1apVFc559913TRcdEREZ5bf/bls6BKJySe/EJzAwEKtWrZJt+/n5Yd26dQrniEQiJj5ERBXQH2+0tnQIRGVC78QnMTHRjGEQEZG5bT57Fy9FlIzLrFvNFTeSiwc+twqpYqmwiMqUwSUriIioYpqyKVphu1Wt4mRnYDNOUiHrwcSHiMiKJGfmyl4XFBUvOhtcxdlS4RCVOSY+RERWpMfSw7LX0lWb7Wz5UUDWg//biYisSMbTAtlraWV2e1uRpcIhKnNMfIiIrJSTgy0A4EkeV3Am62FU4hMXF4cZM2Zg6NChSElJAQDs2rULV65cMWlwRERkPvbPurg8ne0tHAlR2TE48Tl06BDCw8Px33//YcuWLcjOLp4KGR0djdmzZ5s8QCIiMsy7v59X2G4a6Kn2vP0xxX+4iu3Y+E/Ww+D/7VOnTsXnn3+OvXv3wsHBQba/S5cuOHnypEmDIyIiw/0dfV/2evXI5tg8vq3W86Pvpps5IqLyw+DE59KlS+jfv7/Kfl9fXzx8+NAkQRERkWm0rV0FtjYi2Vo9rWp5q5zTNMCrrMMishiDEx9PT088ePBAZf/58+dRo0YNkwRFRESmIbYrHsDcLMgTAPBfQhoAQCIRZOekZueVeVxElmJw4jNkyBB89NFHSEpKgkgkgkQiwbFjxzBlyhSMHDnSHDESEZGRbG2Kp6rfe/xUtk8QBOy49EDlHCJrYFB1dgCYN28eJkyYgICAABQVFaFBgwYoKirCsGHDMGPGDHPESEREpeRkbyt7XWvaToVj1T2dyjocIosxuMXHwcEBq1atQlxcHLZv345ff/0VMTExWLduHWxtbXXfwMx27NiBVq1awcnJCV5eXujXr5+lQyIiKjPdlxxSuz8iWPM4nl6N/MwVDlG5Y3CLj1RgYCACAwNNGUup/e9//8PYsWMxb948dOnSBYWFhbh8+bKlwyIiKjM3U7Jlr8d1CpG9DvDSXI+LJSvImhic+EyaNEntfpFIBEdHR9SuXRsvvvgivL1VZw6YU2FhId577z188cUXeO2112T7GzRoUKZxEBGVF9N61pe9lgiCljOJrIfBic/58+dx7tw5FBUVoV69egCAGzduwNbWFmFhYfjuu+8wefJkHD16tEyTjnPnzuHevXuwsbFB06ZNkZSUhCZNmuCLL75Ao0aNNF6Xl5eHvLySGQ2ZmZllES4RkcnN33lN4zFHe8sPRSAqDwxu33zxxRfRrVs33L9/H2fPnsXZs2dx9+5ddO/eHUOHDsW9e/fQsWNHTJw40RzxahQfHw8A+OSTTzBjxgxs374dXl5eiIyMRFpamsbr5s+fDw8PD9lXQEBAWYVMRGRSKw/HazxWzd2xDCMhKr8MTny++OILfPbZZ3B3d5ft8/DwwCeffIJFixbB2dkZs2bNwtmzZ00S4NSpUyESibR+xcTEQCKRAACmT5+OgQMHIiIiAmvWrIFIJMKmTZs03n/atGnIyMiQfd25c8ckcRMRVQQ1OKOLrIzBXV0ZGRlISUlR6cZKTU2VdRN5enoiPz/fJAFOnjwZo0eP1npOSEiIbFFF+bjEYjFCQkJw+/ZtjdeKxWKIxWKTxEpEVJ55OtsjPadAYd9fb7ezUDRElmFw4vPiiy9izJgx+PLLL9GiRQsAwOnTpzFlyhTZ1PFTp06hbt26JgnQx8cHPj4+Os+LiIiAWCzG9evX0b59ewBAQUEBEhMTERQUZJJYiIjKqwt30nWec35md4U1fN7tUhtVXfmHH1kXgxOflStXYuLEiRgyZAgKCwuLb2Jnh1GjRmHp0qUAgLCwMKxevdq0kerg7u6O8ePHY/bs2QgICEBQUBC++OILAMDLL79cprEQEZW1fsuPKWw3quGuco5IpLhC86AWHNNI1sfgxMfV1RWrVq3C0qVLZQOKQ0JC4OrqKjunSZMmJgvQEF988QXs7OwwYsQIPH36FK1atcL+/fvh5cUCfERkXf6e0F7nOT5ubO0h62P0Aoaurq5o3LixKWMpNXt7eyxevBiLFy+2dChERBZlo6H+Vs9Gfth1OQlASQFTImtiVOJz5swZbNy4Ebdv31YZxLxlyxaTBEZERKa34pUIHLyegubBZbvILFF5YfB09g0bNqBt27a4du0atm7dioKCAly5cgX79++Hh4eHOWIkIiItHmQ8VdiOnt1D6/mR9XzhKja6wZ+oQjM48Zk3bx6WLl2Kbdu2wcHBAV999RViYmIwaNCgcle7i4jIGrSZv1/2eky7WvBwsrdgNETlm8GJT1xcHHr37g2guFL7kydPIBKJMHHiRPzwww8mD5CIiPTXr2l1S4dAVK4ZnPh4eXkhKysLAFCjRg1Z9fP09HTk5OSYNjoiItLqh8NxCttOrMlFpJXBnbwdO3bE3r17ER4ejpdffhnvvfce9u/fj71796Jr167miJGIiDSYtzNGYVt5rR4iUmRw4vPtt98iNzcXQHFdLHt7exw/fhwDBw7EjBkzTB4gERHpz4crMRNpZVDiU1hYiO3btyMqKgoAYGNjg6lTp5olMCIi0k4QBJV9Hs4c2EykjUFjfOzs7DB+/HhZiw8REVnGtQeZCnW3ACB2bk8LRUNUcRg8uLlly5a4cOGCGUIhIiJ99fzqiMo+O1uDf6UTWR2Dx/i89dZbmDRpEu7cuYOIiAi4uLgoHC9vZSyIiIiIpAxOfIYMGQIAePfdd2X7RCIRBEGASCRCUVGR6aIjIiK9bB7fxtIhEFUIBic+CQkJ5oiDiIhKoVENlgwi0ofBiU9QUJA54iAiolIQ23F8D5E+jPpJWbduHdq1a4fq1avj1q1bAIBly5bhr7/+MmlwRESkXg1PJ4VtLlxIpB+DE58VK1Zg0qRJ6NWrF9LT02Vjejw9PbFs2TJTx0dEVKEVSQScvZWGV9ecwv6YZI3nFRZJDLrvvfSnuk8iIhUGJz7ffPMNVq1ahenTp8PWtqQmTPPmzXHp0iWTBkdEVJEVFEkQ+vFODFxxAgeup2LM2jM4eD1F5bykjFw0/XQvZv112QJRElkXgxOfhIQENG3aVGW/WCzGkydPTBIUEVFlcPlehsq+fddUE5/VR+KRlVeIX07c0uu+RRLVFZuJSD8GJz61atVSu4Dh7t27Ub9+fVPERERUKaQ/LVDZ5+ygWj09JilL9rpIIiA7rxBxqdka73snLcc0ARJZIYNndU2aNAkTJkxAbm4uBEHAqVOn8Pvvv2P+/PlYvXq1OWIkIqqQbj1UbQXPzC1U2Xc09qHs9fWkLIxacwqpWXn4++12eOHbY7ARAfHze8vOiVx8UPZ6Ws8wDGsVaNrAiSoxgxOf119/HU5OTpgxYwZycnIwbNgwVK9eHV999ZVscUMiIgKy81STnN9P3cb8AeEar0nNzkNqVh4A4LPtVwEAEgG4+zgHNb2cVc4f1ynURNESWQeDEx8AGD58OIYPH46cnBxkZ2fD19fX1HEREVV4bUKr6HVeNXcxkjOLk51RP52S7T+d+Fj2muN6iEzD4DE+n3/+uWz1ZmdnZyY9REQaPMzO1+u8RtV1r7qclJFb2nCICEYkPps2bULt2rXRtm1bfPfdd3j48KHui4iIrNCphDS9zlM3CFrZCLmWICIynsGJT3R0NC5evIjIyEgsXrwY1atXR+/evfHbb78hJ4czDYiIpH48qlrbsFe4n8q+s7ceq+xTll8ogSAIEAQBw58NZh7QrEbpgySyMkaVrGjYsCHmzZuH+Ph4HDhwAMHBwXj//ffh56f6A01EREBEkBcAwNvFweh79Ft+DCN+PCVLlHzcxCaJjcialLqqnYuLC5ycnODg4ICCAt3NtURE1qhjHR8AwK8nbyM9R7+xP8qi72bgaOxD2bo/Px5RbVEiIu2MSnwSEhIwd+5cNGzYEM2bN8f58+cxZ84cJCUlmTo+IqJKZ/qfpilN0SLY2yT3IbImBk9nb926NU6fPo3GjRvj1VdfxdChQ1GjBvuZiYi0eSzXytOxTlXZ69JMUz+dqN/gaSIqYXDi07VrV/z0009o0KCBwn6JRIKdO3eiT58+JguOiKiyOHwzVfbazqaksf3QDdXaXfoa1Ta4NCERWSWDu7rmzp2rkPTExsbi448/Rs2aNdG/f3+TBkdEVBkMal4T8akl5St+OlYyNqdIYvx9O9fjOmpEhjJqjM/Tp0/xyy+/oGPHjqhXrx6OHz+OWbNm4e7du6aOj4jIrDKeFuD9Dedx6Eaq7pM1KCySIHjqDnRfckjt8aAqLgrbV+5nyl5LBOO7utrV1m9laCIqYVDic/r0aYwbNw5+fn5YtmwZXnzxRYhEInz33XcYP348qlWrZq44iYjM4rk5/+DPC/cVSkUYStqCczMlGylZxSssS+TG7rSq5Y3N49uovTYnX7Welyy2mh44O6ObxuMikciYcImsmt5jfBo3bozMzEwMGzYMx48fR8OGDQEAU6dONVtwREQVweazJa3de64kY0TrIBRISvqw6vq5wd3RXu21gd6qhUcB4MqcKLiIjSqnSERa6N3ic/36dXTs2BGdO3dWGdhMRFQZaGt90eZGcrbs9cw/L2PZvzdQUFTS4mNvo/lXrbRhKLiKM2b3Lf7duuPd9gpJzwdR9YyKi4hU6Z34xMfHo169enjzzTdRs2ZNTJkyBefPn2dTKxFVWI+fKC4kKJ+slMayf2/iUXaebFtL3iObzm5na4NX29VC4oLeaKhUtNTJ3tYkcRGRAYlPjRo1MH36dMTGxmLdunVISkpCu3btUFhYiLVr1+LGjRvmjJOIyOSyco1r4dFHgdx0LbGd5sRFlvjYaP4j8sKddJV9ns7qu86ISDujZnV16dIFv/76Kx48eIBvv/0W+/fvR1hYGBo3bmzq+IiIzCYzV7HMjqQUiwkq23tVdX2emX1UhwkUPntPGy2t58o1ueb2b4Qd73YoZYRE1qlUtbo8PDzw1ltv4cyZMzh37hwiIyNNFBYRkfmdv61YFb2gNIvqKFm4O0Zln9hO9Vfu02fjirLzNLc+3XqUI3vdu7E/hrcKQg1PJxNESWR9Sl2kVKpJkyb4+uuvTXU7IiKzW7T7usJ2ly8PySqfm8Pdx09V9o3/9RwA4HZajsoxqaaBnrLXQ1sEmjwuImtissSHiKiiyVJqZcnOK8TAFceRX6h/y49gwAKESRkliY8h18mf216uzhcRGY6JDxGRkkdP8nSf9Mzuy0l6nys/W+vyvUwtZypyd+JAZiJTYeJDRBXajeQsBE/dgRNxj0x2z9/+u633uanZ+idJg1sGyF2Xq/d1z9X01PtcItKOiQ8RVWg9lh4GAAxdddKg6x5kqI63kfpmf6ze9zFk7R/5KeuPnxTgv3j9krXnAjzxy5iW2D+5k97vRUTqGbweuqYBzCKRCI6OjqhduzY6duwIW1suuEVE5de0LZdMcp9CA2aC2duW/K3p5WKPwT/on6x1rOtjUFxEpJ7Bic/SpUuRmpqKnJwceHl5AQAeP34MZ2dnuLq6IiUlBSEhIThw4AACAgJ03I2IyHRiU7JQ29dNr3PzCkwzdX3HpQdaj9f3d5e9lk98XBxYh4vIEgzu6po3bx5atGiBmzdv4tGjR3j06BFu3LiBVq1a4auvvsLt27fh5+eHiRMnmiNeIiKNui05rPe53RtUM8l7XrybofW4m6NiglOvWnFiVqS0WKK/h6NJ4iEi7Qz+k2PGjBn43//+h9DQUNm+2rVrY/HixRg4cCDi4+OxaNEiDBw40KSBEhGZUoPq7lqPF0kE2GopI6Gvz/s1UtiW3rPQhKtEE5H+DG7xefDgAQoLVVcYLSwsRFJS8bTO6tWrIysrq/TRERGZyR0tCwYCwEMDZmtp0rC6O+pWU+x6s7OVJj6KXW3NAr1K/X5EpJvBiU/nzp0xbtw4nD9/Xrbv/PnzePPNN9GlSxcAwKVLl1CrVi3TRUlEZGIfbL6o9Xhpy1ckLuittp6WdGaX8mywvMKiUr0fEenH4MTnxx9/hLe3NyIiIiAWiyEWi9G8eXN4e3vjxx9/BAC4urriyy+/NHmwRERlZfSa0zrPUR6nI1XV1UHjNXY2xb92d1xUHBR94HqqAdERkbEMHuPj5+eHvXv3IiYmBjdu3AAA1KtXD/Xq1ZOd07lzZ9NFSESkgbrEIz41GyE+rqW+d2xKts5ztl+8L3vds5EfalV1wXcH4/DVkKYar7meXDwMQHn80PrXWxkZKREZwuj5lGFhYQgLCzNlLEREBlFXU+vA9VSTJD76mPnnZdnrj3vVR4C3MyZ0rg0XseZfrc8FeOLwjVRsPX9Ptm/b2+0RXtND4zVEZDoGJz5FRUVYu3Yt9u3bh5SUFEiUBujt37/fZMEREWmjLvG5n655RWapxIdPFLZ/e70Vhq3+z6D3TsrIRWZuyUSPAG9nANCa9ACAg63qCIMqWrrGiMi0DB7j89577+G9995DUVERGjVqhOeee07hi4iorBRIVBOfH48mICdfdeapvMjFBxW229auirn9G6k/WYOztx4bdL7Ulfuq6/5wDR+ismNwi8+GDRuwceNG9OrVyxzxEBHprVBDnawLt9PRtnZVg+41vFUQYlOyseZYosZzBEGASFQ8Nic7r8Cg+0s9yFAtTiq9JxGZn8EtPg4ODqhdu7Y5YiEiMoimKecJj56o3a/LzN4NFLblC5nmFhSh25JDCJ66A9ui78PGyGRFbMfa0ESWZPBP4OTJk/HVV19BELjqKBFZVsZT9a0uIuiflMzoXV/22sZGhFAfF9n24RslU8wPXk9FXGpxQvXO7+dx5X6moeECADrUMawliohMy+CurqNHj+LAgQPYtWsXGjZsCHt7e4XjW7ZsMVlwRETavP3bObX7Z/11GcNaBao9duFOusJ278b+CtsTOtfGpI3RAICkjJLVm5XHDa09nmhgtMX+vZZi1HVEZBoGt/h4enqif//+6NSpE6pWrQoPDw+FL0u6ceMGXnzxRVStWhXu7u5o3749Dhw4YNGYiMh8Eh+pLztRKBGQnad+gPPDLMVSFF7OijOqkjJLxuCcu10ygHn1kQRjw1TQrb6vwnaYn37V5InINAxu8VmzZo054jCJPn36oE6dOti/fz+cnJywbNky9OnTB3FxcfDz87N0eERUhm49eoKG1VX/GAus4qyw7Whvq7AtP938kFxX19UHmru2mgR46h3XczU9FVp9hrQI0PtaIiq9SjPK7uHDh7h58yamTp2Kxo0bo06dOliwYAFycnJw+fJl3TcgogrjSV4hCnXU0nJSSmikJHLjE/dP7qRy3N3JXmWfLsrdZ9p8vf+mwnZTFiclKlN6tfg0a9YM+/btg5eXF5o2bap16uW5c+r73M2tSpUqqFevHn755Rc0a9YMYrEYK1euhK+vLyIiIiwSExGZ3oOMp2gzX3Wh1OGtArH+v9uybQ1ltFBQWHzARgS1KzzX9HQyTaAaKBcnDa/BFZuJypJeic+LL74IsVgMAOjXr5854zGaSCTCv//+i379+sHNzQ02Njbw9fXF7t274eWl+S+qvLw85OWV9PlnZho3U4OIysaSf26o3a88vbxQzeKGALD8QCwAzYmRoev/lJaNDdfwISpLeiU+s2fPVvu6LEydOhULFy7Ues61a9dQr149TJgwAb6+vjhy5AicnJywevVq9O3bF6dPn4a/v7/aa+fPn485c+aYI3QiMoO4VNXioc4OtlBuiJa27CjbfSXJHGERUQVhdJHS/Px8tbW6AgPVTyE11uTJkzF69Git54SEhGD//v3Yvn07Hj9+DHd3dwDAd999h7179+Lnn3/G1KlT1V47bdo0TJo0SbadmZmJgAAONiQqr87dTlfZl5NfhDcjQ/HLiVuyfcsPxOL7EeWvm/uXMS0x8qdTlg6DyGoZnPjcuHEDr732Go4fP66wX7qUe1FRkcmCAwAfHx/4+PjoPC8np3haq42N4nhtGxsbleRMnlgslnXjEVH5N6ZdLfx0THVqub+HE3zdxEh5Nl29LFt2fhnTUu9zO9Yt+X3GGl1EZc/gxOfVV1+FnZ0dtm/fDn9//3JTY6ZNmzbw8vLCqFGjMGvWLDg5OWHVqlVISEhA7969LR0eEZmItnVvUpTW6FFnZp8G+Gz7VXg6Gz57SxP5ZEYfiQt6IzO3AO6OpouBiPRjcOJz4cIFnD17FmFhYeaIx2hVq1bF7t27MX36dHTp0gUFBQVo2LAh/vrrL1aNJ6pENC1MqC/Js1HNXer56jizRJGmkdAAXBzUT5vXhUkPkWUYnPg0aNAADx8+NEcspda8eXPs2bPH0mEQkRnN33VNZd+LTarrff2G08VT3recv4clg5vodY2mYqgf9wrD4BamHddIROZl8AKGCxcuxIcffoiDBw/i0aNHyMzMVPgiIjKnAC9nlX3z+ocDAP6dpLogoTJpoVFtFg4MV9jOLVA/dvGNjqHwMGLBQyKyHIMTn27duuHkyZPo2rUrfH194eXlBS8vL3h6empdL4eIKidB0NwNVFp30nLw0eaLiE3Jku0bGFFT5TwXcXHjdW3fkgUJNZWCaFXLGwBQQ8tChW1DFdfymbIpWv+giahcM7iri0U/iUhq0e4Y/HXhPv5+ux2quJp+duRrP5/GjeRs7L6ShOjZPQAAhUXaE60G/u64+iATJ+IfqT3eLMgL/yWkIaqh5vp9ts8WFZTW7ZKvrdWhTlUcuVk+u/uJSDeDE59OnXQ3JRORdfjuYBwA4KdjCfggyvQTHm4kFy9WmPG0QLZP04rMUtJiorc0VG6X1urStmCydBXo/CIJHmYrzhRbMqgJun55EH2e039cERGVH0YtYJieno5Tp06pXcBw5MiRJgmMiMwjeOoOAEDC/F4mW46iUMusJ23O3nqMUT+dwl9vt0OomrpZ6txILu72shEVl52YPyBcxxWKpD1z2kpFyC8H1vzzfxWO+biJcW5md9jZVpoaz0RWxeDEZ9u2bRg+fDiys7Ph7u6u8ItTJBIx8SEqxx7JtV6cufUYLYK9jb6XfHX0gzGpmNazvsH3GLiieCHUrl8eQuIC/dbbqv5sbE5UQz8sHdwEjhqqsAPAW+vP4rvhiqs3S8ckacv5dDQqMekhqsAM/umdPHkyxowZg+zsbKSnp+Px48eyr7S0NHPESEQmIt8u46QlYdCHfPfT9eQskw9y1jSF/J8ryQAAJwdbrUkPAOy8lKQwMBooKU6qXNRUXmZugcZjRFSxGZz43Lt3D++++y6cnVWnlBJR+bbi2ZgcAMgrLF15GeXurYM3Ukt1P2WaFg28l/4UAHDu1mO97nP38VOFbekYH22dfIHe/P1GVFkZnPhERUXhzJkz5oiFiMzsx6MlNa4GrjiB47HGz07KL1RskTmbqDkRySsswr5rycgyoCVFUwNS3WrFY4GGtNRv4cA8pTjXHEsEAKw+olrvS0pXSxIRVVwGj/Hp3bs3PvjgA1y9ehXh4eGwt1dcvOuFF14wWXBEZF7DVv+n99gaZeduKyY63i4OGs+dtDEaOy4+AADEzu2p1xiZW2nqFxqs7umEG8nZqKLh/ba93R59vz1ash19X+3U9XwNXWlEVLkZnPiMHTsWAPDpp5+qHDNHdXYiKp88nTUnOsqkSQ8ARN9NR0SQ7kHV15Oy1O5PzyluNbKzVd9ZFV7TQ2G7ZS3jB3ATUeVjcFeXRCLR+MWkh8h6KA+OtrfT/OtEfub4vfRctec8Ulov50GG+vMu3EkHABy9qX6BQmXODkat2oGlg1WLG7/evpZR9yKi8oNzMomsnLGzsf5TWhk5R0vVdPlxygFeJaUi3BxLkpLYlGyFazaevqP1/U9qWJlZ2c/HE9XuXzSwsdbrIuuqVm8/nciZq0QVncF/Cqnr4pI3a9Yso4MhorJ3+V6mSveQPtZoSCh0UZhGLpcQfbHnOla8EgEft+LSF8qDkn85kYhu9avJtj/upXndoD3vd0TUssMAgEv3MpCUkQs/D0cAxTW67qU/RV0/N61x2qrpSnuqoVgpEVUcBic+W7duVdguKChAQkIC7OzsEBoaysSHqILJN7KLum41V5yML2kBycnX7z5ZuSUtQ/Z2NsCzHq4ztx5jzNrT2PZOewAl09alZv11BV/vi0WIjwviU5+giqvmMUbyxUoB4PK9DFniI+hRsgIAxGq67rQN4CaiisHgxOf8+fMq+zIzMzF69Gj079/fJEERUdnRVfRTk77PVVdIfH4+kYiJ3evqvO6VH//DpvFt0CLYW1YEVOrSvQyt1z7MzpPVzrLXMLgZKCkyKlXTu6R7Tbr+kLYFDAHA3kY18RnROljrNURU/plkjI+7uzvmzJmDmTNnmuJ2RFSGkjLVDyLW5X9n7ypsS2dbye6bkYseSw9h0xnVsTovf38Cj7LzUKRmfJG+Y47upD3VfdIzzvYlf+OlZBUnTrqms6ur5cUWH6KKz2SDmzMyMpCRof2vNSIqf97bcMGo687dTtd6vPX8fbiRnI0PNl9Uezw1O09tkpNXKEFOvuaB0lLxD9Wv86POwyfFyY78+/0Xb/hAZVOX5SCismdwV9fXX3+tsC0IAh48eIB169ahZ8+eJguMiCquaw8ydZ5zN+2p2rIUpxLSMPKnUzqvd7TX/++2b/bdxJpXW2LPlSTZPk0LIGoTXNXF4GuIqHwxOPFZunSpwraNjQ18fHwwatQoTJs2zWSBEZVXeYVFSMrIRVAVfghqMmH9OZ3nXH2QCXXluDQVJy2No89Kc+y+XJL4tKtT1aB7bH2rrawyPBFVXAYnPgkJmuvbPH2qf587UUWUnJmLVvP2AQBm922AV9tVnAXtJBqKfppKr/CSshBP9Oiquno/U21MNbz0Sy4MabEpeDaA20Fuppa9rmldSpoGehl0PhGVTyYZ45OXl4clS5agVq2K8yFAZIz5O6/JXs/ZdtWCkRjuvT8umPX+Nb1KKprrMxRm95UkWaV0edm5upMmAPBw0p74yK/5IyWfZynP/CIi66B34pOXl4dp06ahefPmaNu2Lf78808AwE8//YRatWph6dKlmDhxorniJCoXlNeWqUi2Rd836/1/OBwve/04J1+va56oWfvnt/9u63Wtp7O91uPyY4Aa1XAHAGyWm4lmp2a6OhFVfnr/5M+aNQsrVqxAcHAwEhMT8fLLL+ONN97AsmXLsGTJEiQmJuKjjz4yZ6xEFnc68bHuk0wgPScfwVN36DVWxpSCp+5A8NQdep3bo4Fqi4rUsJaBRsdw/lktLl2aBHhqPf5ULqnKL1QdN8Tq7ETWSe/EZ9OmTfjll1+wefNm/PPPPygqKkJhYSGio6MxZMgQ2Nra6r4JEWklCAKOxz1Ez6+OAAB2XHqg4wrTka+VlaDHVPHkrDyNx9Stz6Mvfd4bgMrih8rmDQiXvb6RnI2ULMX1iuTrhGni7FD8e21gs5p6xURE5Z/eg5vv3r2LiIgIAECjRo0gFosxceJEiHSsfkpE+jtwPQVj1p4p8/eVSAQs3XtDtm2nY/xLfqEE0WpaZgatPAERgP8SzF/MU90Cg/KquTsqbLecu0/xej1+d52b2R03k7NlXWVEVPHp3eJTVFQEB4eSwYR2dnZwdXXVcgWVV5fvZWDh7hhka6mmTZaxPyZFZV9mboGaM03rp2MJCq1Ld9JytJ7/VENdrlMJaWWS9JiCg5paXMoc7W0RXtODf+ARVSJ6t/gIgoDRo0dDLC6unJybm4vx48fDxUVxLZMtW7aYNkIyuT7fHAVQ/OH1yQsNLRwNyfv1pOrA3s+3X8Wil54z6ftUcxcjObOkq+rzHdcUjp9OfIy2tQ1b56Y0OtX1waEbqSr7fd3EshITALBscBO8/8cFjGgdVGaxEVHlonfiM2rUKIXtV155xeTBUNnSZ3VdKnFZRwFNdZIzc+HrJi5Vi8FJI0or6HJialeEfLxT43F7O+3xCjDtmkA/j2mpdlB1fX93pGSVJET9mtZAr3B/vVpriIjU0TvxWbNmjTnjIAu4ysTHIIZ2OW05dxeTNkZjROsgfNavkdHvezstBzeTs1CnmpvR91BmYyPCutdaYsSP6ktDtAmpovV6+VIT/ZpUx58XzDNVfmBETZWWICY9RFQa/A1ixbL0XCiOinnqWDBP2aSN0QCAdSdvlfq9uy89XOp7KNP2/ehqz5GftaU8iNiUfFzFZrs3EVknJj5kUhKJYPbSCJbyV/Q9lX2aBgGX52fQqpY3AO1FPnVVIZdv8WlUw8M0ganBxZWJyNSY+JDJFEkEhHy8EyEf79T5wVkRpWWrrkac8VR991ee0oJ5ln4ehXKL9YU/S1S0jTvSlbfJJz5pT/RbpXnRwMYAgDA/N8zu20CvkhERQSX1sVoGe+v1PvpYOSLCZPcioorF4CKlRJqsOlJSsuDc7ceICDLdB5Wp7br0AD+fSMTPY1pCbKff4puNa3pgk1zJAwDYezVZbYvH9ouKY14KJQLsbUvffHH4RiqOxj7EB1H1YK9jAT95j3NKErT2z6qSa8s7inRkPtLjzg62iEnK0npu29AqeKV1EHo28kOYvxtCfVzhIrbDq+1qITYlGx5OxaUnXm9fC6uPKhZBtrO1wZ73O2LD6dt4u3Ntre+jzgdR9fDFnusq+6Ma+qk5m4isAVt8rJwpWyL+uZIke52jYZ2X8uLN9edwMj4NzT/7V+9rAqsUL90gv+LvmmMJyMhRbPV5kPEUH2y+qLBPVyIh1TTQU+vxkT+dwg+H47Hh9B297if1RG7NpudqFr+HtpIN6oqHypO2aBVJBHi7aK+Z9dvY1ugV7g+RSITGNT3hIi55frV9XeHjVjyOp3+zGmqvr+fnhtl9G6KKEeN9hrcyvnQGEVVOTHys3B65ZKW05D/b7z4uv8U8bySXtFBk5RXqnfyN+ql4BpT8oPDM3EI89+k/eJRdstaMuhWN9U0E9W0Tumfg8xXLjefxcike1KycsMmTfyQX76bjq39vKoxnkiZSeYUSOGgpV+Ptov+AcH27zAzBCuxEpIyJj5WL17Mukj7kWzUM/WAuS4lK3/OdtNLHKp3BBUBhwT2p3AL9Ep9zt9M1HpMfMJ2uZ/VzqcKi4mvlBzRrK/kg3+LzwrfHsPTfG+iw6ADO3y4u0tr/u+Oy44FVnDTeZ9d7HfSOsYG/6ctCMPEhImVMfKyM8qKFjnqOb9HHJbkF/srzCv9+Hqaffi2/1szjJ6otKVO3XDL4nsr1sk7GP5K9TtVQIPTIzVQET92h0pJX+Cxpsrcp+ZGv5qb5OUiTWOW1i+QTHqnuDTSPlzFkqrsxXVm66FOPi4isCxMfK/Pa2tMK2zdTtA9MNdauy6brQjM15RlXNib+Kdh/XbXe1mE15RiUXbybLnvdItgLAd7OCseHrf5P9jpLqc5aYZEEhUUS2YKE49adVTkOALZyA6wDqyjeX560xUddTa7vD8UpbDuaaUHBYC3x6Us573mupgc2jmtT6vsSUcXFWV1W5n5GrsK2u5P2ganGik3JNst9TeHz7VcVtrdFP8CbkaEmu7+uyuaaLPv3pux1kwBPjGgdjI5fHFB77pnEkjIW7Rbsx7107d11qc/GIKVrGdcDAPa2IhQUCVh5KB7+Hk7wclYdo7NgV4zCtp2tDRa//Bwu3k3HqYQ0nbO89LV6VPNS30OkNGrqr7fbl/qeRFSxscXHyrUox1POzSX6rmLNrYW7YzScWcKQumaB3sa1VMhXZq9V1RWBVZxxbmZ3tedKBODK/QxcvpehMenZfPYuVhyMQ15hEe6n56o9R1nBs7FA/yWkoedXR/TusnwpoiY+fbERxnYI0e8CDfyfdUOuebUFavuWvkQHx/gQkTImPlbGyV5xTE+RIODc7cfleqVheQdiUrBod0yZx9vzqyN6n9slzFft/iwDan1JZ005O2geg9X766Po881RjcenbIrGwt0xiPjsX9SqWjwV31Ws2Mi7YngzrXEUGvicXR1L14i8Z2JH/P12O0TW9SnVfaSY+BCRMiY+Vuap0uyicevOYsB3xxG1zPS1oMzh1bWn8d3BOPx26rZR10tnJZnTO7+fV7u/hwH1tlo8KythbLeZvOy8QgxccVz2Wp78/4cRrYNUrj10XffYJHlupUx83B3t0bimZ6mq2Sv76PkwONnbYvs77OYiIiY+9MzNlGyFRe6MUd8M05E1WaRH95Q66mYllZUHGfp1NwHFg3CB4vEz5vRik5JFA6dE1VM5fivNsOUOdFV1t4Q3I0NxeU6UWWuKEVHFwcSHZEo7ILmae8l05P5N1a/CayqZRlSWjzXRDLYFA8JV9ulacVlKeXq4JqZs8dDG1kaEhPm9cP3z52WlI+T9cyXZoPuJRCKcn9kdHepULVf1sNjlRURSTHzKyJ20HAz54YRe05otxdfd+HVUJBIBB+W6RQxdYM9Q/kasxfP2b+q7oAylblXq81oWHpRX8GwqfbYBK0abm0gk0livLMGIBS69XByw7rVWrIdFROUSE58y8sHmaJyMT8PIZ2UPlM3fdQ3zd14zexyaBt4CJav7GiNHaezQAQPHhuhDvnvr5eYBBl//MFv9on+GcrCzQd1qrir7d19+oPPa/CIJdl9+gEaz96Drl4cUjkm7tyqSWX0aWDoEIiKDMPEpI1fulUyHVl4ULjUrDysPxWPl4Xizt5QEaVkU7sjNhyZ7H1N3LWw4dRvfHSxZOE9sxKJ56hbjk1p+IFbjMeWxT7Y2Iux5v6PKeeN/PQegJLms5i5WmTX1NL9Idp5yuZBWz8bHKI+T+XNCO42xWdqY9rUsHQIRkUGY+JQR+SEbkzddUDh253FJ8ccPlap6m5q23pVzamY8nb2VprN4pCAIGKg0aLhbfc0tS4ZKycxVKflgTOvNEy2Jzxd7rqPVPPWV2ufvUmyJO3wjFSKRCN3qV1N7vrRMw/BWQXBQStBy8osQ6uOi9jrpfxHlQeJNAjw1xm1Jc15oaOkQiIgMxsSnjMgPxt15SbGcg3wBy3+uGjaYVF6RRMCFO+koKJJoPCevUPOH/+1HOQrbh26kYuCKE2i3YL/W903PKcD1ZMWBw0UmXGen5bx9KvvWHEs02f2lkjPVJ1PKqx13b1Cc8Hw7rKna86VrDNmIgBAfxS6x+IdPEJda0tIjX9Vd+szs7SrGQFyWwSKiioiJTzmgqZK5IAhYcTAOx2L164KKXHwA/ZYf09pqlPgwR+OxU3JlEABg37XiJEx57R9lN9XMBsvR0rpSXvRp7K+yT90igx3qVFXaLl5cz9HeFlVcFEs6PMzOQ9GzZjUbGxF83BQHjCuvIxTxeUkrk/Q62wqSUTjam67ALRFRWWHiUw7Ij12Rt+9aChbujsFwueKUmhRJBNxJK06gtp6/p/E8Q8opZDzVb+q1ujV1biSX31pd2txIVp3yrlzvqZ6f5lIKzT//V9aCZysSqczcUl45WZ60xUfd+Ki2oWWzPs5MAwYr161W+pISRERljYlPOaBpyvD9DO2FJ+XpuwbP3mv6d6X9deG+7HWSlsX3ujVQHesSomEcS1mZs+0K+i0/ppB4eCu1ziiPvwGA3ALVbsL0p5rHOLVWk5BcuJMOoLgryMVBMdFx0LIgoXSafH6hagy/jW2Na58+r/FafTXTsd7QawYMVg7TkgASEZVXTHzKMflxMo91DDC+9Uj3eit5hUU6Bypr0nq+6jgbqcZqVsQ15RgfTZIz1SdjGTkFWHMsERfupOPSvZKCpJ7Oigv0vaKmRMOvJ2+p7Lt6v2RG3qKBjRWOdVTqBgNKEpi4lCewsRHhyIedZceaB2suCistUrrycLza404OtmhVy/xFZRvrOa3e3syrShMRmQN/c5Vj8l1NWTpWKlYegKuOfEuCMdPBNSlSM1XM0OKWmmhbbTk1S/1g5DfXn5W9fmnFCVn18vhUxeRQ3Qw35VpWAFBHrkunQXXFGVfK3WDy/jhzBwAQ4O2Mes/uIdEwrU5dK486i15qrPskLRrX9NR5zuqRzfW6F1dDJqKKiImPhfwdfV/nOcv+vSl7rWu86+azd3XeTz4XyVPzQTu+U6jOe6ijrnUn+k66SVp9rsi1tiizs1V8KGlP8tH8839xPO6RbF9+kQSTN15Qe726pCr6WTeVvJCqJd12DZSmmrfRc+yN9N9PXeLz+s+nUXfGLr3uE1SldF2IH6ipx6XM193wVbGJiCoKJj4W8q5cBW996lrpqm6gbcCtVGpWSdeQvVzS4Pts5tH3h+IQfScdyw/EauxGUkdTgjNgRekLgmqb0XZQaXXoTl8cULu+j7oSEwAQEaTabaSuBljusyUAmgZ6wkaplUM5+dJE2jqi7ln9ey1Fr3uYgouWwdVERNaAiU85oNyKoK6Gk6YuEqkueiwY6Cw30NZJbipye7lxKi8uP/ZsMT/NY3qU7b6cpHa/tPVEIhFw6W6G3t058tR1PUkt2KU4m0xT18vdx08Rl6o6+Ft5qrkm3x8sHnOjrh6XnY1+P0LSlqvoOxk6ziwf9k5UXZnajy1BRFQJMPEpB5THw9xXM4OqUKI9adA2W0h2D7laXC1rVcG/kzpi+zvtVdaiMVRVHQnEp9uvou+3RzHht3MG3/tJnv7rAWn7YH7hm6MK29vfaa+2Grk62haEtNfS4lPD00ll39J/b+h8v97hqusLlbU6aqaqJxnQCkhEVF4x8SkHFiqtg3Phdrps9V8pXV1d+oyniU1VHNNS29cNjWp4YOt53eONtFmlYRaS1M5LxcU79xqxKvUhPavZ30zOQkyS5oHQ8uUqgqs4o5GamWiaDGhW3BXZuZ6PyjE7LQnnjN719X4PeW93qW3UddqSsNKq7uGIOr6qhVmJiCqaCpP4zJ07F23btoWzszM8PT3VnnP79m307t0bzs7O8PX1xQcffIDCQu2zocojGxFQoNTCo27mlDz5Ab0AUKimlWLC+pJxRfLdaaWtWq5rBpexU+j18Xf0faw8FIfuSw/rfc173epoPPbR82Eq+0TPRiar6xpz0rJ6sbEtJOrWF9Jlz/sdseXNdiqrTMvTt2tP6uInPTC9V33M6F0ff77dDtveaY/3uhY/u1pVLbtOExGRsSpM4pOfn4+XX34Zb775ptrjRUVF6N27N/Lz83H8+HH8/PPPWLt2LWbNmlXGkapnyF/LwVVdFLqlAO2VxQFgzbEEhe3a01VnCcmXnjD/KjsljJnavuvSA4z86ZTO8979/Tzm71JdOVod6RigFnJr6QxtGahwztUHqrPIpEsJqFtSQNuUbm2LPmpjp2Oa+PtqEjdfNzHCa3pg3Wut8OtrrQAAP4yIUDhnzegWBsXh7miPsR1D8HqHEPi6OcLR3hYTu9fFxU964B81Y4CIiCqCCpP4zJkzBxMnTkR4eLja4//88w+uXr2KX3/9FU2aNEHPnj3x2WefYfny5cjPN1+Lg74WDNS8/oryB529rUhlwPCPRxUTG3n5hRK109MPa+kmah1S8uHvq2dLwJGb+nU7SfUK9zPofHlvrj+nNX5jqOsOVC7hsU3NMgM/PXv2uzQM4tZEZGTNLV3r47zXtQ6OTe2isE9+ccb2daoicUFv9Gio+PwN6d7Txt3RnosXElGFVWl+e504cQLh4eGoVq2kfEJUVBQyMzNx5coVjdfl5eUhMzNT4cscQrR0DTRUWhTvZnI2Zv+tGPP2iw/UXpuSmatxDRhtLSYjWgeX3EPDQoAq1/youwVGXrVyOgvo8ZOSxR5faR2oUnphzxXFBKd5sBcA7f+G6qhb1VkfuhImkUiEGp5O2PZ2ewBAvybVjU6yiIisTaVJfJKSkhSSHgCy7aQkzX+pz58/Hx4eHrKvgIAAs8Tn5eKAz/s1Unss+q7iFOc315/TOo1b3jo1JRbUuZ+uuJaNk4P5Kmv3flb1XN34F0PWB1I2vFWgrBunNORbVNwc7bH7fcVumwnrFWefSVc77hKme8kAecFGjoNRt5yBOuE1PZC4oDeWDWlq1PsQEVkjiyY+U6dOhUgk0voVE6Pf+A1jTZs2DRkZGbKvO3fumO291NWG0vdDTlmvr45g2KqTeidI6taxkTJl+QoAcH6W8FxTM15Gfn2gnPxCLNgVg4t30/W679z+4Whfpyrm9lefQOqrto7xVspjkoqeDTTX1AV16ZMe+OON1iqVzY0t6aDv2kCGYHUJIqJiFl3GdfLkyRg9erTWc0JCQvS6l5+fH06dUuyKSU5Olh3TRCwWQyw2bLaLKX24+aLB10TfSZcNwtU2q0ietgHGK0dEYPSa0wbHocmmZ+UzDlxPRaKGyvMAsHjPDfx0LAHfH4pD4oLeet9/aItATN962ej4DJ32LZ0gpymRcXO0R6uQKmgVUgWfbb8q229kTmtSf05oh7XHEvBRT9XZakRE1siiiY+Pjw98fFTXRjFGmzZtMHfuXKSkpMDXt7hLYu/evXB3d0eDBg10XG05m/SosaXsdlqO7PW+GO3lDgRBgEgk0li2AQDqqlmszlQiFx/UeOwnuZloOfmFKCgU4OFsj7uPczReA0ClbIShDB0Po6vFR56Lg61szSBjw6zqWroFJeU1CfBkVxgRkZwKM8bn9u3buHDhAm7fvo2ioiJcuHABFy5cQHZ2cRdOjx490KBBA4wYMQLR0dHYs2cPZsyYgQkTJli0RUcbY7u5fj6eqPe515OLF/XTNp3e38P4QcjqylCMaqPapafLc3P+wXOf/oMneYV6VZovS9I1lGz0SJiOT+0qey3fyBbgrbqKszqvtA7UuigiERGVToX5DTtr1iw0bdoUs2fPRnZ2Npo2bYqmTZvizJkzAABbW1ts374dtra2aNOmDV555RWMHDkSn376qYUj18zY4uVnbj3W+9zYlOLE8FRCmsZzSjMjSF0pjQZKs9T0UfBs3aK41GyddcnMTbpSs5R0Gryu9XUAQNPwnN9eb63Xe8vXUyMiItOrMInP2rVrIQiCyldkZKTsnKCgIOzcuRM5OTlITU3F4sWLYWdXfj9I9CkzUVo3kosTn6v3zTNNX51t0eqn3kspl+OQl/DwiUXGxpyb2V32ummAp8Ix6fR3fbrY3Bzt0b9pDfRu7I9q7iUtjQFK6wUBwFuRoSr7jG0FJCIi/VSYxKcyKouWja/33QQA9GtaQ8eZxlH3LegqgXH+WdV2daq5O+LgddMuXKgPbxcH9H2uOgAgLlVxQPbuZ+v6nEnU3Gomb+ngJlg+rJnOlrQ61VS7H9XN/CMiItNh4mNB5wzosgKAHRcfGN0iMP7Xs7LX1dWM6XG0N+6/grrkzU7HrKm4FM1T6/ddS1Zbl2zDG/p1FZXGjWdFTm+mqC92esDECVnfxtXxZmQovhveTLZPeSVpIiIyLSY+FvTziUSDzp/w2zn8raakgqHuq6khFfNZTyTM74VBzWsadC91ycCk7nW1XnPj2YBrdYUu98ekoMWzlZLltQ6porB95MPOhoSpF+lA8GOxJQVft543fNadvuxsbfDR82HoFe6PhPm9kDC/F1dgJiIyMyY+ZUy+ftWtR4rTtr8a0kTn9ZpKVxhiTLtaaveLRCIseuk5lf0fRNWTvf5HqZzDu7+fVz5dZ6vFoWc1uGp6qc50ikt9oldpDHXX6kO+dUWbt38rXr154h/RRr2PLs8pjSOSLthJRETmxcSnjC0fVvLBG5Ok2KXSLFC1pSPER7FV5Fjsw1LH8OHz9XSfJKd7g5JSIG+sO6vlzGLKY2SU+XsWJy1P9Fx1Wh1jkwTlumiabL/4AFm5itPqjV2JWSqyXsmaVfrGQUREpsXEp4xp+8BWN/PnVaXWmZz8IlR1NWxdIuXSEY56rvYsZcgHfr1qbgiqor3FR1p1/dztdIPiMIUansa1FAHAJy80LNV7fzO0ZCHBCxb43omIiIlPuRdZV3Vla12zppT1/OpIqWKw17N21Mw+DfDb2FZ6l9Eordfb11JbMT16dg+sebWFyv4Xnqtu0OKAyklqk2fFSo3l5mgve61rADgREZkHE59yxt2xZN2h/z7uqrYVqKwpf0hrKoz6WvtaqOIqhghl86E+o08DrHglQmHf8mHN4OFkj871fPFZv0b47fWSau730jWX7VBHeVVqU9YOfSnCsEHkRERkGuV3dT8r1ameL7Y9m7lVzV2/UhJVXR3wMDtfr3M1DWzW5J+JHVUSn2/3xyKoijPua0gkjB14bAx/z5JntHdiR9SRqzs2QmlNnLMGLh/ww+F4hW1TJnTlIaElIrJGbPEpJ7rVLx5ArK7rRpfP+4XLXg9tGYD1cq0cyn7975bO+8nPyqpbzQ0OSt1D1x5kYtqWS/hmf6za60tbRNQQ7o72ODglEsendlFIetQZ2MywVhblwc3JmarLABjq+1ci8G6X2mq7MImIyPzY4lNOSAe+vhkZiofZeejR0E/HFSVOxpesO1PTyxntalfVeK66oqLKWgR7K1SAV05kLt5NV7lGfsaSPob8cMKg87UJ1pEs/jKmJXZfScKM3vUNum+80uw0U4zLeb6RH55vpP+/LRERmRYTn3LCyaF4QLCjvS3m9g/Xcbai/k1roHWIN/65kmxwV5Y603vXR0GRBIOaBwAAxHaKLT7qZnl1CfPVed/uDaph79VkAMDJeP3KP5hCx7o+6GhEC8sJuYQSAO6kGTZGiIiIyh92dZUDfRr7l+p6R3tbPN/IH0sGN5ElUKXh7eKAr4c2Rfs6xS1HYjvFe6obT6TPujqNa3iUOjZzmthN+4rTbUOraD1ORETlHxMfC/B0tlfYblDKxewMmKGNAc3MU6xUW8V1KUMLpX7erxFah3jjt7GaxyyZ0rtda2s97ubIBlIiooqOiY8FtAj2Vti2K+VgYBsDVjGe0sOwVZul5FdvVufn44k672HoysevtA7ChjfaoG2o5jFLpqSr1crT2aFM4iAiIvNh4mMByh+vdqVcICZJzWyjbvXVj7mxN6R5SI6XUiuVsviH2stUAIYlaOVRaUtWEBGR5THxsYDeSmN67Es5WyjAS3VNGHXFRgHA3cm47hpnB+3X6ZPT6MobeoeXPBd/D/3WMCorn/RtYOkQiIjIBJj4WIByt9E2HRXX5aujq+Ngp/rP6O2ivltGeaCyvnTN2lKXfKnQkfgsl6uc3sC/fBXxjAjy1n0SERGVexytaQGC0jjgUwnap3b7uGkvSloWXTC6poO7ihX/KzXwd8dVueKovRv7Q6JhCaG/JrRTSdSOmKAKvSk1qlG+EjEiIjIOW3wsQLmIZ3Ud3TpRDfxUxtj87802steGVms31iutAzUeG902WGH7zwnt8N3wZjgxrQu+f6UZvnipMQSon/n1XICnSgmHjnXKZkCzsv4aZp7pM12fiIjKPyY+FmBjI8K4jiGybX9P7bWtPJztcWJaV4V9EUHeiPnseSTM72WWGNWpp6UkhPLYIQc7G/QK94e/hxOeb+QPZwc7lZYubWr7ai8/YS5LBzfBcwGeFnlvIiIyPyY+FhLmX/LBHlxFd30uR3tbLB1cPGC5w7PWEEd7W60tEe21lK4wxt3Hmlcu1mfskD4tU4teaox2tavgzU6hBsVmSsorVRMRUeXB3/AW0ktuBpO3i/ap4lL9m9bEfx93xc+vttTr/E9fbGhUbJpoSwiquOpe40bdIGxlg5oHYP3rreGhY/q8OemzGCMREVVMTHwsRGxni497hSHMzw1vRmpfMVheNXdHvaufuzmaNnnQ1rrUuKanSd/Lks7cemzpEIiIyEw4q8uC3ugYijc6mq9Lp6oerTCGKLKSlhAfNzFSs/IsHQYREZkBW3wqMVPPRNKnO0uX0pbnKAu+OpYPICKiiouJjxUZ2jKgVNdHNfQrdQyFSq1GLiaoJm9qVtKwRURklZj4WJH5AxqX6vrqOqbdG+qHERE4MCXSpPc0hZl96ls6BCIiMhMmPpVcjWfJyr+TOpnkfl11lK7QZdngJgCKFwrs0dAPvu7lqyYXALQNrYpRbYJk2x5OlpthRkREpsXBzZXckQ87Iyu30GTTw0N9XbEvJkW23TLYG5Fh2stZyOvXtAY61fWBpwWnq+tDLLe69tSeYRaMhIiITImJTyVnYyMy6Zo4zzfyww+H42XbG8e30XK2el4aCqiWJ25ytceGttRcqoOIiCoWdnWRQZoFelk6hDLxSusg1K3mig+i6lk6FCIiMiG2+JDR/HUUV63IvFwc8M9E04yLIiKi8oMtPmQ05anpRERE5R0THzIaVzcmIqKKhokPERERWQ0mPkRERGQ1mPgQERGR1WDiQwaTrnET4uNi4UiIiIgMw8SHDPb10KYI8HbC/P7hlg6FiIjIIFzHhwzWOcwXR8K6WDoMIiIig7HFh4iIiKwGEx8iIiKyGkx8iIiIyGow8SEiIiKrwcSHiIiIrAYTHyIiIrIaTHyIiIjIajDxISIiIqvBxIeIiIisBhMfIiIishpMfIiIiMhqMPEhIiIiq8HEh4iIiKwGEx8iIiKyGnaWDqC8EQQBAJCZmWnhSIiIiEhf0s9t6ee4Jkx8lGRlZQEAAgICLBwJERERGSorKwseHh4aj4sEXamRlZFIJLh//z7c3NwgEolMdt/MzEwEBATgzp07cHd3N9l9SRWfddnhsy47fNZlh8+6bJnqeQuCgKysLFSvXh02NppH8rDFR4mNjQ1q1qxptvu7u7vzB6mM8FmXHT7rssNnXXb4rMuWKZ63tpYeKQ5uJiIiIqvBxIeIiIisBhOfMiIWizF79myIxWJLh1Lp8VmXHT7rssNnXXb4rMtWWT9vDm4mIiIiq8EWHyIiIrIaTHyIiIjIajDxISIiIqvBxIeIiIisBhMfE1q+fDmCg4Ph6OiIVq1a4dSpU1rP37RpE8LCwuDo6Ijw8HDs3LmzjCKt+Ax51qtWrUKHDh3g5eUFLy8vdOvWTee/DZUw9P+11IYNGyASidCvXz/zBliJGPqs09PTMWHCBPj7+0MsFqNu3br8PaInQ5/1smXLUK9ePTg5OSEgIAATJ05Ebm5uGUVbcR0+fBh9+/ZF9erVIRKJ8Oeff+q85uDBg2jWrBnEYjFq166NtWvXmjYogUxiw4YNgoODg/DTTz8JV65cEcaOHSt4enoKycnJas8/duyYYGtrKyxatEi4evWqMGPGDMHe3l64dOlSGUde8Rj6rIcNGyYsX75cOH/+vHDt2jVh9OjRgoeHh3D37t0yjrziMfRZSyUkJAg1atQQOnToILz44otlE2wFZ+izzsvLE5o3by706tVLOHr0qJCQkCAcPHhQuHDhQhlHXvEY+qzXr18viMViYf369UJCQoKwZ88ewd/fX5g4cWIZR17x7Ny5U5g+fbqwZcsWAYCwdetWrefHx8cLzs7OwqRJk4SrV68K33zzjWBrayvs3r3bZDEx8TGRli1bChMmTJBtFxUVCdWrVxfmz5+v9vxBgwYJvXv3VtjXqlUrYdy4cWaNszIw9FkrKywsFNzc3ISff/7ZXCFWGsY868LCQqFt27bC6tWrhVGjRjHx0ZOhz3rFihVCSEiIkJ+fX1YhVhqGPusJEyYIXbp0Udg3adIkoV27dmaNs7LRJ/H58MMPhYYNGyrsGzx4sBAVFWWyONjVZQL5+fk4e/YsunXrJttnY2ODbt264cSJE2qvOXHihML5ABAVFaXxfCpmzLNWlpOTg4KCAnh7e5srzErB2Gf96aefwtfXF6+99lpZhFkpGPOs//77b7Rp0wYTJkxAtWrV0KhRI8ybNw9FRUVlFXaFZMyzbtu2Lc6ePSvrDouPj8fOnTvRq1evMonZmpTFZyOLlJrAw4cPUVRUhGrVqinsr1atGmJiYtRek5SUpPb8pKQks8VZGRjzrJV99NFHqF69usoPFyky5lkfPXoUP/74Iy5cuFAGEVYexjzr+Ph47N+/H8OHD8fOnTsRGxuLt956CwUFBZg9e3ZZhF0hGfOshw0bhocPH6J9+/YQBAGFhYUYP348Pv7447II2apo+mzMzMzE06dP4eTkVOr3YIsPWZUFCxZgw4YN2Lp1KxwdHS0dTqWSlZWFESNGYNWqVahataqlw6n0JBIJfH198cMPPyAiIgKDBw/G9OnT8f3331s6tErn4MGDmDdvHr777jucO3cOW7ZswY4dO/DZZ59ZOjQyAlt8TKBq1aqwtbVFcnKywv7k5GT4+fmpvcbPz8+g86mYMc9aavHixViwYAH+/fdfNG7c2JxhVgqGPuu4uDgkJiaib9++sn0SiQQAYGdnh+vXryM0NNS8QVdQxvy/9vf3h729PWxtbWX76tevj6SkJOTn58PBwcGsMVdUxjzrmTNnYsSIEXj99dcBAOHh4Xjy5AneeOMNTJ8+HTY2bEMwFU2fje7u7iZp7QHY4mMSDg4OiIiIwL59+2T7JBIJ9u3bhzZt2qi9pk2bNgrnA8DevXs1nk/FjHnWALBo0SJ89tln2L17N5o3b14WoVZ4hj7rsLAwXLp0CRcuXJB9vfDCC+jcuTMuXLiAgICAsgy/QjHm/3W7du0QGxsrSy4B4MaNG/D392fSo4UxzzonJ0cluZEmnALLXZpUmXw2mmyYtJXbsGGDIBaLhbVr1wpXr14V3njjDcHT01NISkoSBEEQRowYIUydOlV2/rFjxwQ7Ozth8eLFwrVr14TZs2dzOrueDH3WCxYsEBwcHITNmzcLDx48kH1lZWVZ6luoMAx91so4q0t/hj7r27dvC25ubsLbb78tXL9+Xdi+fbvg6+srfP7555b6FioMQ5/17NmzBTc3N+H3338X4uPjhX/++UcIDQ0VBg0aZKlvocLIysoSzp8/L5w/f14AICxZskQ4f/68cOvWLUEQBGHq1KnCiBEjZOdLp7N/8MEHwrVr14Tly5dzOnt59s033wiBgYGCg4OD0LJlS+HkyZOyY506dRJGjRqlcP7GjRuFunXrCg4ODkLDhg2FHTt2lHHEFZchzzooKEgAoPI1e/bssg+8AjL0/7U8Jj6GMfRZHz9+XGjVqpUgFouFkJAQYe7cuUJhYWEZR10xGfKsCwoKhE8++UQIDQ0VHB0dhYCAAOGtt94SHj9+XPaBVzAHDhxQ+/tX+nxHjRoldOrUSeWaJk2aCA4ODkJISIiwZs0ak8YkEgS20xEREZF14BgfIiIishpMfIiIiMhqMPEhIiIiq8HEh4iIiKwGEx8iIiKyGkx8iIiIyGow8SEiIiKrwcSHiMhEEhMTIRKJWJ2eSI3Dhw+jb9++qF69OkQiEf7880+D7yEIAhYvXoy6detCLBajRo0amDt3rkH3YOJDREZLTU3Fm2++icDAQIjFYvj5+SEqKgrHjh2TnWPsLzhjjB49GiKRCAsWLFDY/+eff0IkEpVJDESk3pMnT/Dcc89h+fLlRt/jvffew+rVq7F48WLExMTg77//RsuWLQ26B6uzE5HRBg4ciPz8fPz8888ICQlBcnIy9u3bh0ePHlksJkdHRyxcuBDjxo2Dl5eXxeIwJVZbp8qgZ8+e6Nmzp8bjeXl5mD59On7//Xekp6ejUaNGWLhwISIjIwEA165dw4oVK3D58mXUq1cPAFCrVi2D42CLDxEZJT09HUeOHMHChQvRuXNnBAUFoWXLlpg2bRpeeOEFAEBwcDAAoH///hCJRLJtAPjrr7/QrFkzODo6IiQkBHPmzEFhYaHsuEgkwooVK9CzZ084OTkhJCQEmzdv1hlXt27d4Ofnh/nz52s855NPPkGTJk0U9i1btkwhvtGjR6Nfv36YN28eqlWrBk9PT3z66acoLCzEBx98AG9vb9SsWRNr1qxRuX9MTAzatm0LR0dHNGrUCIcOHVI4fvnyZfTs2ROurq6oVq0aRowYgYcPH8qOR0ZG4u2338b777+PqlWrIioqSuf3TVTRvf322zhx4gQ2bNiAixcv4uWXX8bzzz+PmzdvAgC2bduGkJAQbN++HbVq1UJwcDBef/11pKWlGfQ+THyIyCiurq5wdXXFn3/+iby8PLXnnD59GgCwZs0aPHjwQLZ95MgRjBw5Eu+99x6uXr2KlStXYu3atSp99TNnzsTAgQMRHR2N4cOHY8iQIbh27ZrWuGxtbTFv3jx88803uHv3bqm+x/379+P+/fs4fPgwlixZgtmzZ6NPnz7w8vLCf//9h/Hjx2PcuHEq7/PBBx9g8uTJOH/+PNq0aYO+ffvKWsHS09PRpUsXNG3aFGfOnMHu3buRnJyMQYMGKdzj559/hoODA44dO4bvv/++VN8HUXl3+/ZtrFmzBps2bUKHDh0QGhqKKVOmoH379rI/LuLj43Hr1i1s2rQJv/zyC9auXYuzZ8/ipZdeMuzNTFrylIisyubNmwUvLy/B0dFRaNu2rTBt2jQhOjpa4RwAwtatWxX2de3aVZg3b57CvnXr1gn+/v4K140fP17hnFatWglvvvmmxnjkq8G3bt1aGDNmjCAIgrB161ZB/tfd7Nmzheeee07h2qVLlwpBQUEK9woKChKKiopk++rVqyd06NBBtl1YWCi4uLgIv//+uyAIgpCQkCAAEBYsWCA7p6CgQKhZs6awcOFCQRAE4bPPPhN69Oih8N537twRAAjXr18XBKG4OnjTpk01fp9EFZ3y74Xt27cLAAQXFxeFLzs7O2HQoEGCIAjC2LFjFX5OBEEQzp49KwAQYmJi9H5vjvEhIqMNHDgQvXv3xpEjR3Dy5Ens2rULixYtwurVqzF69GiN10VHR+PYsWMKLTxFRUXIzc1FTk4OnJ2dAQBt2rRRuK5NmzZ6z5hauHAhunTpgilTphj8fUk1bNgQNjYlDePVqlVDo0aNZNu2traoUqUKUlJSVOKUsrOzQ/PmzWUtVdHR0Thw4ABcXV1V3i8uLg5169YFAERERBgdN1FFk52dDVtbW5w9exa2trYKx6Q/K/7+/rCzs5P9jABA/fr1ARS3GEnH/ejCxIeISsXR0RHdu3dH9+7dMXPmTLz++uuYPXu21sQnOzsbc+bMwYABA9TezxQ6duyIqKgoTJs2TSUWGxsbFP/RWaKgoEDlHvb29grbIpFI7T6JRKJ3XNnZ2ejbty8WLlyocszf31/22sXFRe97ElV0TZs2RVFREVJSUtChQwe157Rr1w6FhYWIi4tDaGgoAODGjRsAgKCgIL3fi4kPEZlUgwYNFKav29vbo6ioSOGcZs2a4fr166hdu7bWe508eRIjR45U2G7atKnesSxYsABNmjRR+UvQx8cHSUlJEARBNs3dlGvvnDx5Eh07dgQAFBYW4uzZs3j77bcBFH/v//vf/xAcHAw7O/4KJuuRnZ2N2NhY2XZCQgIuXLgAb29v1K1bF8OHD8fIkSPx5ZdfomnTpkhNTcW+ffvQuHFj9O7dG926dUOzZs0wZswYLFu2DBKJBBMmTED37t0VWoF04eBmIjLKo0eP0KVLF/z666+4ePEiEhISsGnTJixatAgvvvii7Lzg4GDs27cPSUlJePz4MQBg1qxZ+OWXXzBnzhxcuXIF165dw4YNGzBjxgyF99i0aRN++ukn3LhxA7Nnz8apU6dkCYQ+wsPDMXz4cHz99dcK+yMjI5GamopFixYhLi4Oy5cvx65du0rxNBQtX74cW7duRUxMDCZMmIDHjx9jzJgxAIAJEyYgLS0NQ4cOxenTpxEXF4c9e/bg1VdfVUkQiSqTM2fOoGnTprI/XiZNmoSmTZti1qxZAIonQYwcORKTJ09GvXr10K9fP5w+fRqBgYEAiltqt23bhqpVq6Jjx47o3bs36tevjw0bNhgWiInGKRGRlcnNzRWmTp0qNGvWTPDw8BCcnZ2FevXqCTNmzBBycnJk5/39999C7dq1BTs7O4XBw7t37xbatm0rODk5Ce7u7kLLli2FH374QXYcgLB8+XKhe/fuglgsFoKDg4U//vhDa0zyg5ulEhISBAcHB0H5192KFSuEgIAAwcXFRRg5cqQwd+5clcHNyvfq1KmT8N577ynsCwoKEpYuXSp7LwDCb7/9JrRs2VJwcHAQGjRoIOzfv1/hmhs3bgj9+/cXPD09BScnJyEsLEx4//33BYlEovF9iMg0RIKg1NFNRFQOiEQibN26Ff369bN0KERUibCri4iIiKwGEx8iIiKyGpxSQETlEnvhicgc2OJDREREVoOJDxEREVkNJj5ERERkNZj4EBERkdVg4kNERERWg4kPERERWQ0mPkRERGQ1mPgQERGR1WDiQ0RERFbj/35KuGjhoyfeAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "def plot_running_average(rewards, window_size=10000):\n", + " running_avg = np.convolve(rewards, np.ones(window_size)/window_size, mode='valid')\n", + " plt.plot(running_avg)\n", + " plt.title('Running Average Reward Over Time')\n", + " plt.xlabel('Step Number')\n", + " plt.ylabel('Running Average Reward')\n", + " plt.show()\n", + "\n", + "# Assuming `rewards` is a list of rewards from each episode\n", + "for env in environment_results:\n", + " rewards = env[\"metrics\"][\"rewards\"]\n", + " print(f\"Plotting running average reward for {env['env_id']}\")\n", + " plot_running_average(rewards)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.10" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/evals/elsuite/incontext_rl/scripts/run_experiments.sh b/evals/elsuite/incontext_rl/scripts/run_experiments.sh new file mode 100755 index 0000000000..9d8765dc22 --- /dev/null +++ b/evals/elsuite/incontext_rl/scripts/run_experiments.sh @@ -0,0 +1,39 @@ +#!/bin/bash +logdir=./logs +outputdir=./outputs + +timestamp=$(date +%Y%m%d_%H%M%S) +logpathbase=$logdir/$timestamp/ + +mkdir -p ${logpathbase} + +echo Running experiments and logging to $logpathbase +read -p "Enter the number of runs: " num_runs + +set -x # Enable printing of each command before it's executed +# Random baselines +oaieval incontext_rl/random incontext_rl.v0 --record_path ${logpathbase}explanations/random.log +oaieval incontext_rl/random incontext_rl.raw.v0 --record_path ${logpathbase}raw/random.log + +for (( run=1; run<=num_runs; run++ )) +do + echo "Run #$run" + # Use explanations variant + # Direct + oaieval generation/direct/gpt-4-turbo-preview incontext_rl.v0 --record_path ${logpathbase}explanations/gpt-4-turbo-preview_${run}.log + oaieval generation/direct/gpt-3.5-turbo incontext_rl.v0 --record_path ${logpathbase}explanations/gpt-3.5-turbo_${run}.log + + # Raw variant + # Direct + oaieval generation/direct/gpt-4-turbo-preview incontext_rl.raw.v0 --record_path ${logpathbase}raw/gpt-4-turbo-preview_${run}.log + oaieval generation/direct/gpt-3.5-turbo incontext_rl.raw.v0 --record_path ${logpathbase}raw/gpt-3.5-turbo_${run}.log + +done + +echo Done running experiments, all logs in $logpathbase + +echo Producing plots for use_explanations variant, outputs to $outputdir +python plot_experiments.py --log_dir $logpathbase/explanations --out_dir $outputdir/explanations +echo Producing plots for raw variant, outputs to $outputdir +python plot_experiments.py --log_dir $logpathbase/raw --out_dir $outputdir/raw +set +x # Disable printing of each command after they've been executed diff --git a/evals/elsuite/skill_acquisition/eval.py b/evals/elsuite/skill_acquisition/eval.py new file mode 100644 index 0000000000..52c770db7d --- /dev/null +++ b/evals/elsuite/skill_acquisition/eval.py @@ -0,0 +1,428 @@ +import json +import logging +import os +import random +from collections import defaultdict +from typing import Any, Dict, List, Optional, Union + +import evals +import evals.metrics +from evals.api import CompletionFn +from evals.elsuite.skill_acquisition.task_description import TASK_DESCRIPTION +from evals.elsuite.skill_acquisition.utils import ( + PROMPTS, + answer_detected, + get_accuracy, + get_average_bleu_score, + get_average_invalid_retrieval_calls, + get_average_retrieval_calls, + get_average_retrieval_precision, + get_bleu_score, + get_bootstrap_accuracy_std, + get_question_type, + get_std_of_difference, + process_answer, + process_view_instruction, + render_intermediate_prompt, + view_instruction_detected, +) +from evals.eval import SolverEval +from evals.solvers.solver import Solver +from evals.task_state import Message, TaskState + +TARGET_LANGUAGES = ["miskito"] +LESSON_FILE_SUFFIX = "_lessons.jsonl" + +logger = logging.getLogger(__name__) + + +class SkillAcquisition(SolverEval): + def __init__( + self, + completion_fns: List[CompletionFn], + samples_jsonl: str, + target_language: str, + knowledge_base_directory: str, + max_replies: int, + seed: int = 6122023, + n_samples: Optional[int] = None, + *args, + **kwargs, + ): + super().__init__(completion_fns, seed=seed, *args, **kwargs) + + assert ( + target_language.lower() in TARGET_LANGUAGES + ), f"Error: target language must be one of {TARGET_LANGUAGES}" + + self.samples_jsonl = samples_jsonl + self.n_samples = n_samples + self.task_description = TASK_DESCRIPTION.format(target_language=target_language) + self.rng = random.Random(seed) + + # Retrieval-related attributes. + self.knowledge_base_directory = self._prefix_registry_path(knowledge_base_directory) + self.files_available = os.listdir(self.knowledge_base_directory) + self.content_by_file: dict[str, dict] = {} + self.max_replies = max_replies # Used as timeout. + + def eval_sample(self, solver: Solver, sample: Dict, rng: random.Random) -> Dict[str, Any]: + """Runs the appropriate private evaluation function depending on the eval phase: retrieval or non-retrieval. + + Args: + solver (Solver): per-sample solver instantiated in parent. + sample (Dict): input to evaluate on. + rng (random.Random): random number generator, used for reproducibility. + + Returns: + Dict[str, Any]: metrics collected during evaluation. + """ + # since we run two discrete experiments per sample, we have to copy the solver ahead of time + non_retrieval_solver = solver.copy() + retrieval_solver = solver.copy() + non_retrieval_out = self._eval_non_retrieval_sample(non_retrieval_solver, sample) + retrieval_out = self._eval_retrieval_sample(retrieval_solver, sample) + metrics_obj = { + "non_retrieval": non_retrieval_out, + "retrieval": retrieval_out, + } + + evals.record.record_metrics(**metrics_obj) + return metrics_obj + + def _eval_non_retrieval_sample(self, solver: Solver, sample: Dict, *_) -> Dict[str, Any]: + """Evaluates the given sample without using retrieval, ie. using the solver directly. + + Args: + solver (Solver): any compatible solver, instantiated just for this sample. + sample (Dict): input to evaluate on. + + Returns: + Dict[str, Any]: metrics collected during evaluation. + """ + task_state = TaskState( + task_description=self.task_description, + messages=[Message(**msg) for msg in sample["input"]], + ) + + result = solver(task_state) + output = result.output + if answer_detected(output): + answer = process_answer(output) + logger.debug(f"Model answered {answer}") + else: + answer = "NO ANSWER DETECTED" + + picked = evals.record_and_check_match( + prompt=sample["input"], + sampled=answer, + expected=[sample["ideal"]], + ) + + out_obj = { + "prompt": sample["input"], + "raw_output": result.output, + "parsed_output": answer, + "expected": [sample["ideal"]], + "correct": picked is not None, + "bleu": get_bleu_score(sample["ideal"], answer), + "question_type": get_question_type(sample["input"][-1]["content"]), + } + return out_obj + + def _eval_retrieval_sample(self, solver: Solver, sample: Dict, *_) -> Dict[str, Any]: + """Evaluates the given sample using retrieval. The retrieval logic is implemented in the _conversation_loop function. + + Args: + solver (Solver): any compatible solver, instantiated just for this sample. + sample (Dict): input to evaluate on. + + Returns: + Dict[str, Any]: metrics collected during evaluation. + """ + files_available_paths = [ + self.knowledge_base_directory / file for file in self.files_available + ] + assert all([file.exists() for file in files_available_paths]) + task_state = TaskState( + task_description=self.task_description, + messages=[Message(**msg) for msg in sample["input"]], + current_state={"files": files_available_paths}, + ) + + output, metrics = self._conversation_loop(solver, task_state) + + if answer_detected(output): + answer = process_answer(output) + logging.debug(f"Model answered {answer}") + elif output == "Context length exceeded.": + answer = "NO ANSWER DETECTED" + logger.warn("Current interaction exceeded model context length.") + else: + answer = "NO ANSWER DETECTED" + logging.debug(f"Model timed out after {metrics['current_replies']} replies.") + + picked = evals.record_and_check_match( + prompt=sample["input"], + sampled=answer, + expected=[sample["ideal"]], + ) + + out_obj = { + "prompt": sample["input"], + "raw_output": output, + "parsed_output": answer, + "expected": [sample["ideal"]], + "correct": picked is not None, + "bleu": get_bleu_score(sample["ideal"], answer), + "ctx_len_exceeded": output == "Context length exceeded.", + "interaction_timed_out": metrics["current_replies"] >= self.max_replies, + "question_type": get_question_type(sample["input"][-1]["content"]), + "lesson_retrieval_calls": metrics["lesson_retrieval_calls"], + "correct_retrieval_calls": metrics["correct_retrieval_calls"], + "invalid_retrieval_calls": metrics["total_retrieval_calls"] + - metrics["correct_retrieval_calls"], + "total_retrieval_calls": metrics["total_retrieval_calls"], + } + return out_obj + + def run(self, recorder: evals.record.Recorder) -> dict[str, Union[float, int]]: + samples = self.get_samples() + self.rng.shuffle(samples) + samples = samples[: self.n_samples] if self.n_samples is not None else samples + + results = self.eval_all_samples(recorder, samples) + non_retrieval_results = [result["non_retrieval"] for result in results] + retrieval_results = [result["retrieval"] for result in results] + + baseline_accuracy = get_accuracy(non_retrieval_results) + baseline_std = get_bootstrap_accuracy_std(non_retrieval_results) + + retrieval_accuracy = get_accuracy(retrieval_results) + retrieval_std = get_bootstrap_accuracy_std(retrieval_results) + + delta_accuracy = retrieval_accuracy - baseline_accuracy + + # TODO: decide which metric to report – propagated standard deviation + # from bootstrapping or standard error of the mean estimated from repeats + # of the eval experiments. + delta_std = get_std_of_difference(baseline_std, retrieval_std) + + ctx_len_exceeded_rate = sum( + 1 for result in retrieval_results if result["ctx_len_exceeded"] + ) / len(retrieval_results) + timeout_rate = sum( + 1 for result in retrieval_results if result["interaction_timed_out"] + ) / len(retrieval_results) + + num_translation_samples = len( + [result for result in retrieval_results if result["question_type"] == "translation"] + ) + num_non_translation_samples = len( + [result for result in retrieval_results if result["question_type"] == "non-translation"] + ) + + result = { + "baseline_accuracy": baseline_accuracy, + "baseline_std": baseline_std, + "retrieval_accuracy": retrieval_accuracy, + "retrieval_std": retrieval_std, + "delta_accuracy": delta_accuracy, + "delta_std": delta_std, + "average_retrieval_precision": get_average_retrieval_precision(retrieval_results), + "average_non_retrieval_bleu_score": get_average_bleu_score(non_retrieval_results), + "average_retrieval_bleu_score": get_average_bleu_score(retrieval_results), + "average_retrieval_calls": get_average_retrieval_calls(retrieval_results), + "average_invalid_retrieval_calls": get_average_invalid_retrieval_calls( + retrieval_results + ), + "ctx_len_exceeded_rate": ctx_len_exceeded_rate, + "timeout_rate": timeout_rate, + "num_samples": len(retrieval_results), + "num_translation_samples": num_translation_samples, + "num_non_translation_samples": num_non_translation_samples, + } + + return result + + def _view_content( + self, + file_name: str, + section_title: str = None, + sections_visible_to_model: dict[str, set] = defaultdict(set), + sections_viewed: dict[str, set] = defaultdict(set), + ) -> tuple[str, dict[str, set], dict[str, set]]: + """Views content from a JSONL file in the knowledge base. + If a section is provided, only the contents of that section are returned. + If no section is specified, the function returns the table of contents of the file. + + Args: + file_name (str): Name of the file. Full directory prefixed automatically. + section_title (str, optional): Name of the section to view. Defaults to None. + sections_visible_to_model (dict[str, set], optional): Dictionary of sections visible to the model. Defaults to {}. Updated in-place. + sections_viewed (dict[str, set], optional): Dictionary of sections viewed by the model. Defaults to {}. Updated in-place. + + Returns: + tuple(str, dict[str, set], dict[str, set]): A tuple of + the content of the section (if specified) and + the updated dictionaries of sections visible to and viewed by the model. + """ + # TODO: more general file format. + + if file_name in self.content_by_file: + file_content_by_section = self.content_by_file[file_name] + else: + # This should never occur, but if it does it should stop the eval from running. + if not os.path.exists(self.knowledge_base_directory / file_name): + raise ValueError( + f"File {self.knowledge_base_directory / file_name} does not exist." + ) + + file_content_by_section = {} + with open(self.knowledge_base_directory / file_name, "r") as f: + for line in f: + line_dict = json.loads(line) + file_content_by_section[line_dict["title"]] = line_dict["content"] + self.content_by_file[file_name] = file_content_by_section + + if section_title is None: + sections = set(file_content_by_section.keys()) + sections_visible_to_model[file_name] = sections + sections_viewed[file_name].add("Table of Contents") + + return ( + f"Table of contents for {file_name}: {sections}.", + sections_visible_to_model, + sections_viewed, + ) + + sections_viewed[file_name].add(section_title) + return file_content_by_section[section_title], sections_visible_to_model, sections_viewed + + def _conversation_loop( + self, solver: Solver, task_state: TaskState + ) -> tuple[str, Dict[str, int]]: + """Maintains a conversation with the model until it outputs an answer or times out. + The model may request to read a file or a section of a file from the knowledge base. + + Args: + solver (Solver): any compatible solver, instantiated just for this sample. + task_state (TaskState): current task_state, which additionally contains a list of knowledge base files in `current_state`. + + Returns: + tuple[str, Dict[str, int]]: a tuple of the model's output and a dictionary of metrics collected during the conversation. + """ + output = "" + + # Not all retrieval calls are valid, e.g. if the file doesn't exist. + # These two metrics are analogous to an instruction-following rate. + metrics = { + "lesson_retrieval_calls": 0, + "correct_retrieval_calls": 0, + "total_retrieval_calls": 0, + "current_replies": 0, + } + sections_visible_to_model: dict[str, set] = defaultdict(set) + sections_viewed: dict[str, set] = defaultdict(set) + consecutive_instruction_failures = 0 + + while not answer_detected(output) and metrics["current_replies"] < self.max_replies: + if metrics["current_replies"] == 0: + # Beginning of the conversation, prepare instructions. + task_state.task_description = ( + task_state.task_description + + "\n\n" + + PROMPTS["retrieval_instructions"].format(list_of_files=self.files_available) + ) + if len(sections_viewed.items()) > 0: + intermediate_prompt = render_intermediate_prompt(sections_viewed) + task_state.messages += [Message(role="system", content=intermediate_prompt)] + + output = solver(task_state).output + task_state.messages += [Message(role="assistant", content=output)] + metrics["current_replies"] += 1 + + if view_instruction_detected(output) or answer_detected(output): + consecutive_instruction_failures = 0 + + if view_instruction_detected(output): + file, section = process_view_instruction(output) + metrics["total_retrieval_calls"] += 1 + + if file.endswith(LESSON_FILE_SUFFIX): + metrics["lesson_retrieval_calls"] += 1 + + # Handle any errors by logging and re-prompting the model. + if file not in self.files_available: + task_state.messages += [ + Message( + role="system", + content=PROMPTS["wrong_file"].format( + file=file, knowledge_base=self.files_available + ), + ) + ] + logger.debug( + f"Model tried to view {file}, which does not exist in the knowledge base:\n{json.dumps(self.files_available, indent=4)}." + ) + continue + + if section is not None and section not in sections_visible_to_model[file]: + task_state.messages += [ + Message( + role="system", + content=PROMPTS["wrong_section"].format( + file=file, + section=section, + table_of_contents=sections_visible_to_model[file], + ), + ) + ] + logger.debug( + f"Model tried to view section {section} in file {file}, which does not exist.\nAvailable sections are {json.dumps(list(sections_visible_to_model[file]), indent=4)}." + ) + continue + + # If no errors, view the content and update the task state. + content, sections_visible_to_model, sections_viewed = self._view_content( + file, section, sections_visible_to_model, sections_viewed + ) + task_state.messages += [ + Message( + role="system", + content=PROMPTS["present_content"].format( + file=file, + section=section if section is not None else "Table of Contents", + content=content, + ), + ), + ] + metrics["correct_retrieval_calls"] += 1 + if section is None: + logger.debug(f"Model viewed table of contents for file {file}: {content}") + else: + logger.debug(f"Model viewed section {section} in file {file}.") + elif not answer_detected(output): + if consecutive_instruction_failures >= 3: + return "Model failed to follow instructions.", metrics + + consecutive_instruction_failures += 1 + logger.debug( + f"Model output did not contain a view instruction or an answer: {output}" + ) + + # Flag & move onto next sample if context length exceeded. + if ( + "'code': 'context_length_exceeded'" in output + or "Please reduce your prompt; or completion length" in output + ): + return "Context length exceeded.", metrics + + task_state.messages += [ + Message( + role="system", + content="Your output did not contain a view instruction or an answer. Please try again.", + ) + ] + + return output, metrics diff --git a/evals/elsuite/skill_acquisition/readme.md b/evals/elsuite/skill_acquisition/readme.md new file mode 100644 index 0000000000..2d5a8fafcb --- /dev/null +++ b/evals/elsuite/skill_acquisition/readme.md @@ -0,0 +1,64 @@ +# Skill acquisition + +This eval tests models' ability to learn a skill with minimal human involvement. In the initial release, models are evaluated on questions related to the [Miskito language](https://en.wikipedia.org/wiki/Miskito_language). Some samples are translation and others are language manipulation exercises. + +## Usage +Run with: +```bash +oaieval skill_acquisition.miskito +``` + +Where the solver can be any generation solver in `evals/registry/solvers/defaults.yaml`, eg. `generation/cot/gpt-3.5-turbo-16k`. + +## Evaluation process +Every time the eval is run, the model is evaluated twice. The first time, it answers the question directly using whatever prompting technique is executed by the solver you choose. The second time the model runs in a loop, interacting with an interface which gives it access to a knowledge base. The knowledge base contains text files, some of which are relevant for answering the question, while others are unrelated. If models can use this interface to increase their performance on the task, we can say that they've improved or acquired their language translation and manipulation skills. + +## Prompts +See `skill_acquisition/utils.py` to review/adjust the prompts used in this eval. + +## Datasets + +The dataset is generated from [this language course](https://en.wikibooks.org/wiki/Miskito), which comprises 229 questions. We further split this into manipulation-only (`miskito_test_manipulation.jsonl`) and translation-only (`miskito_test_translation.jsonl`) subsets. + +## Variants + +We test zero-shot and few-shot prompting techniques on the dataset: + +| Dataset | Zero-shot | Few-shot | +| --------- | -------- | -------- | +| Miskito | `skill_acquisition.miskito.zero-shot.full`|`skill_acquisition.miskito.few-shot.full`| + +The `full` in this case refers to the size of the dataset – there are also variants for testing where only 5 examples are considered, called `dev5`. For full details, look at `evals/registry/skill_acquisition/skill_acquisition.yaml`. + +For the few-shot setting, use the eval-specific solvers in `evals/registry/solvers/skill_acquisition.yaml` to avoid train/test leakage. + +## Token Usage Estimates + +Below is a rough estimate of the total number of tokens consumed by some variations the eval, including both input and output tokens: + +| Model | Solver | Prompt tokens | Completion tokens | Total tokens +| --- | --- | --- | --- | --- | +| gpt-3.5-turbo | direct | 1,000,000 | 23,000 | 1,050,000 | +| gpt-3.5-turbo | cot | 930,000 | 120,000 | 1,050,000 | +| gpt-3.5-turbo | fewshot | 450,000 | 9,600 | 460,000 | +| gpt-3.5-turbo-16k | direct | 1,400,000 | 24,000 | 1,500,000 | +| gpt-3.5-turbo-16k | cot | 2,000,000 | 120,000 | 2,100,000 | +| gpt-3.5-turbo-16k | fewshot | 610,000 | 10,000 | 620,000 | +| gpt-4-base | direct | 1,800,000 | 420,000 | 2,200,000 | +| gpt-4-base | cot | 4,700,000 | 890,000 | 5,600,000 | +| gpt-4-base | fewshot | 1,400,000 | 320,000 | 1,700,000 | +| gpt-4-1106-preview | direct | 1,700,000 | 100,000 | 1,800,000 | +| gpt-4-1106-preview | cot | 1,600,000 | 99,000 | 1,700,000 | +| gpt-4-1106-preview | fewshot | 1,700,000 | 95,000 | 1,800,000 | +| gpt-4-32k | direct | 1,800,000 | 80,000 | 1,900,000 | +| gpt-4-32k | cot | 2,700,000 | 180,000 | 2,900,000 | +| gpt-4-32k | fewshot | 190,000 | 6,000 | 190,000 | + +## Version History +v0: Initial version released + + +## Contribution statement + +Eval design, implementation, and results evaluation were primarily conducted by Andrei Alexandru. Giulio Starace was responsible for code reviews throughout the implementation process, along with fine-grained feedback on the project in general. Additional guidance was provided by (alphabetically by last-name) Steven Adler, James Aung and Chan Jun Shern, who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation. + diff --git a/evals/elsuite/skill_acquisition/scraping/human_rights.html b/evals/elsuite/skill_acquisition/scraping/human_rights.html new file mode 100644 index 0000000000..c6d49a320c --- /dev/null +++ b/evals/elsuite/skill_acquisition/scraping/human_rights.html @@ -0,0 +1,2839 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + OHCHR | Universal Declaration of Human Rights - Miskito + + + + + + + + + + + + + +
+
+ +
+ + + +
+ +
+ +
+
+ + + + + + +
+ +
+
+ + + +
+
+
+ + + + +
+ +

Universal Declaration of Human Rights - Miskito

+
+

SOURCE

+

Comité para la Defensa de los Derechos Humanos, Honduras

+
+
+ +
+ + + + + Download + Other Languages +
+
+
+
+ +
+
+
+
+ + +
Miskito
+ +
+
+
Language Profile
+ + +

TOTAL SPEAKERS

160,000 (1982)

USAGE BY COUNTRY (OFFICIAL LANGUAGE)

Home Speakers: Nicaragua, Honduras

BACKGROUND

It belongs to the Misumalpan family (Macro-Chibchan subgroup) and is spoken by 11,000 people in Honduras and over 150,000 people in Nicaragua. It is a language of trade in Honduras, whereas it is widely used in Nicaragua, both in primary schools and among older people.

+ +
+
+
+
+
+
+ + +

Upla sut Raitka nani ba Tasba aiska laka ba Bapuia

+

Asla Takanka tara ba Naha Upla sut Raitka nani ba Tasba aiska laka ba Bapuia,

+

Sut lukanka baku, upla sut, kantri nani sut, trai kaikaia; baku, kKumi bani, dakni nani bani, nahara kat luki, tabaikaia, Smalkanka bak, kul nani bak, naha Raitka nani ba, Bara, Prika laka naniba, pramis kum Dauki, kantri laka nani bilkak, at apia, tasba, aiska, laka, nani bilkak, atsa, yaka kakaira takaia bara kulkaia wan kantri nani bui.

+

[Preamble]

+

Kulkanka 1

+

Upla sut ba kulkanka lakara, airaitka nanira bara pri, sin, aikuki, baku takisa. Bamna sins laka bri baku, lukanka bain pri baku aimuihni lakara, pana pana tabaikan kaiasa.

+

Kulkanka 2

+

Naha lakara pas taurá wisa, upla baniba airaitka brisa, bara sin, pri san: nisanka kulkras, taya maplika kulkras, mairin sapa waikna sapa kulkras, bila aisanka kulkras, ani gadkara mayuni sapa kulkras, aipulitikka lukanka ba, apia sa kaka, dia dia dukya kabia sin kulkras, wan tasbaya wina sapa, yuya kira sapa, wan tasbayara baikan sapa, apia kaka, dia dia walanatkara sin kulkras kira.

+

Baku sin wan kantri pulitik ka laka bui sin, wan kantri laka nani bui sin, apia kaka, tasba aiska laka mita sin, apia laka, ani tasbayara iwi ba bui sin upla kumira sin mayara kulkan kaia apia sa. Bamna kantri wala nani natkara iwi bara, kankri wala laka munhtara iwiba, alba laka natkara nanira iwiba sin saura baku kulkan kaia apia sa.

+

Kulkanka 3

+

Upla sut ba airaitka brisa airayaka kum brieia pri lakara iwaia upla baku, aimain kira kaia.

+

Kulkanka 4

+

Sip apia upla kum alba lakara kaia, bamna, baha natkara yus munan kaia sin, kan baha laka apu sa.

+

Kulkanka 5

+

Upla kumi sinra, sip apia sa uba saura munan kaia, silak mankan kaia, swira pask an, an upla apia baku munaia.

+

Kulkanka 6

+

Upla bani ai raitka brisa anira kabia sin lâkat upla baku kulkan kaia.

+

Kulkanka 7

+

La mawanra sut ba kumi kulkan sa, bara kumi bani la ba bui aikuki baku main kaikisa, upla sut ba airaitka brisa aikuki baku main kaikan kaia, wala nani bui mayara kulkan kabiara sin; naha laka tara bapanna kulkras baku.

+

Kulkanka 8

+

Upla sut la airaitka brisa tabaikanka uplika pain kum brikaia, wan kantri laka mawanra, baku mika sipsa airaitka nani kulkras munbia sin, la kulkan ka ba brih wabia.

+

Kulkanka 9

+

Upla kumi sin, sip apia sa ban kakalhni silak ra mangki saura munaia.

+

Kulkanka 10

+

Upla sutba, airaitka brisa, wala nani baku, upla sut mawanra, an la kum taibanka apu kira bui aiturka ba walan kaia. Baku mika airaitka nani, bara, witin daukaia dukia nani ba marikan kabia,m apia kaka, dia dia saurka dukiara munansa kapa sip kabia laki kaikaia.

+

Kulkanka 11

+
    +
  1. Upla bani ba, dia dia saurka dudiara laura lulkansa bara, airaitkabrisapas taura aiturkaba aisaia sip kabia, kau taibi munras bara, la tankaba kat, baku mika, bilka nani yâban kaiasa, upla sut mawanra, bapi buaia sip kaia.
    2. +
  2. Upla kumira sin, saura munan kaia apiasa pât kum dukiara, daukanka ba puyara, saura pali kulkan apia sa kaka, wan kantry raitka nani bui sin, ba wisi sin, saurka uba tara kulkan kaia apia sa, baha pât kaba dukiara.
+

Kulkanka 12

+

Upla bani Rayakaba, Wala bui turban kaia apiasa, Tâika nanira kabia sinm, watla bilara kabia sin, dukya nanira kabia sin ki, apia kaka, nina sauhkaia, rispik ka alahbaia; upla bani ba, airaitka brisa, baha nani saurka mapara la bui main kaikaia.

+

Kulkanka 13

+
    +
  1. Upla bani ba airaitka brisa, pri pali taukaia bara, kantri bilara tasba kum bri kaia, iwaia lahma.
    2. +
  2. Upla bani ba, airaitka brisa aitasbaya wina taki waia, bara, kli balaia, dimaia sin.
+

Kulkanka 14

+
    +
  1. Bankra dia dia patka dukiara nina blikisa kaka, upla bani ba, airaitka brisa, natka kumpliki, tasba wala kum distika makabaia, wala nani baku auya pah iwikaia dukiara.
    2. +
  2. La nani bui pat bahki nani kulkanba dukiara ban sin Tasba Aiska Asla Takanka brinka nani bara lukanka ta nani ba kulkras kira, naha Raitkana makabaia.
+

Kulkanka 15

+

Upla baniba, airaitka brisa kantri kumra iwikaia apia kaka, kantri walara iwaia lukankaba yabalka prakan kaiasa.

+

Kulkanka 16

+
    +
  1. Waikna bani, mairin bani ba, airaitka brisa pyua alkansa bara, nisanka kulkras, ani kantrikara iwiba kulkras, ani Gadkara mayuniba kulkras kira, sipsa marittakaia; sahwaia sin, baku mika, marit takansa bara, apia, mahka wal swibia sinki wal baku iwaiasa.
    2. +
  2. Marit laka daukan, kabia marit uplika naniba, aikupya wilinkira sakaka.
    3. +
  3. Panli laka, upia sut wina kau yamnika bak sakan dukia kumsa, bamna, upla sut bui, Gabament bui sin main pali kaikan kaiasa.
+

Kulkanka 17

+
    +
  1. Upla bani ba airairka brisa aidukia pawaia lahma brikaia, yakan lakara, bamna, upla wala nani aikuki asla lakara sin.
    2. +
  2. Upla kumi ra sin, Aidukia Pawaia lahma pat taki ba, yabalka prakaia.
+

Kulkanka 18

+

Upla bani ba, ai raitka brisa pri lakara dia dia lukaia, lukanka pain nani brikaia, bara, ani ani Gadkara lukaia, naha raitkana ra luki sipsa Gad Wala nani ra lukaia, upla nanira marikaia sipkabia; yakan kabia, upla wala sin, Aikulkanka aikuki kabia sin; upla nani mawanra, prakan ra dauki kaia, lahma, kulkaia, lahma, bara, laki kaikaia mata kabia sin.

+

Kulkanka 19

+

Upla sutba, airaitka brisa prikaia ailukankara aisankara; naha raitkara aisisa, dia dia lukanka dukiara, upla kumira sin warbras kaia sa tanka plikaia sip kaia, dia dia turiba nu kaia, bara, wala nanira maisapakaia, kantry ka kulkras, dia dia bilkak kat kabia sin.

+

Kulkanka 20

+
    +
  1. Upla bani ba, airaitka brisa pri lakara aslatakanka kum daukaia, bara, aslatakanka lamni laka kat brikaia.
    2. +
  2. Upla kumi sin, sip apia sa, taibi munankaia asla takanka kumra tilara kaia.
+

Kulkanka 21

+
    +
  1. Upla sutba, airaitka brisa, gabament dukia tilara kaia, ban sin, wala nanira tabaikaia baha nani tilara kabia.
    2. +
  2. Upla sutba airaitka brisa, wala nani aikuki baku Gabament Warkka nani tilara kaia.
    3. +
  3. Tawan aiska brinka ba, upla sut karhnika sa gabament tanira; naha brinka ba, gabament bani mangkisa bara, klir lakisa, lulkaia laka ba kat kulki, aikuki baku, bara, upla bani aikupya laka kat, ban natka wal nani ni daukkbia sin.
+

Kulkanka 22

+

Upla bani ba, upla baku airaitka brisa, main kaikankaia, baku sin, gabament tabaikanka baku, tasba aiska buisin asla takanka nani bilka brisa bara, gabament dukia nani sut kulki, pawaia natka nani, asla takaia nani, bara, aikulkanka nani sutba brin kaiasa, baku mika, upla baku ailukanka kat, bara, pri pali pawaia sip kabia.

+

Kulkanka 23

+
    +
  1. Upla bani ba airaitka brisa warkka kum brikaia, bara, aikupya pahkira wark plikaia, wala nani baku kaia, wark pain brikaia, bara, warka apu pyuara tabaikan kaiasa.
    2. +
  2. Upla sutba airaitka brisa wala mita mayara kulkankaia apiasa, aiwarkka daukiba baku mâna sin baku kaia su.
    3. +
  3. Upla wark taki naniba airaitka aprisa aimana kum brikaia, mana sin painkaia, baku sip kabia aitaya nani main kaikaia, upla baku iwaia, ban sin bilka sa kaka natka wala nani pliki mainka kaikan kaiasa.
    4. +
  4. Upla bani ba airaitka risa aslatakanka dakni paskaia bara tilara dimaia sin, aibrinka nani dukyara aiklabaia mata.
+

Kulkanka 24

+

Upla bani ba ai raitka brisa ris briaia, riska lilika briaia ai wark ka pyua kum brikaia, bara, baku sin ris pyua yari nani sin, ai mana wal.

+

Kulkanka 25

+
    +
  1. Upla bani ba, airaitka brisa iwaia natka pain kum brikaia, baku, sip kabia witin, bara, aitaika nani sin, siknis nani luhakaia; kau purara ban kulkan kaiasa: plun ba, praka, utla ba, sika nani yabaiaba, upla baku mainka kaikaia ba; baku sin, airaitka brisa wark apu sa pyuara, mainka kaikan kaia, siknis sa pyuara saua sakan sa bara, pyarka takansa bara, almuk takan sa bara, ban sin dia dia bui kra aidukia nani sut sauhki tikan sa bara, tabaikan karia.
    2. +
  2. Mairin ba, kwihra sa bara, baikan pyuara sin airaitka brisa main kaikan kaia; bara, dia dia brinka nani sut yâban kaia, ani tuktika, marit laka kat kulki baikan kabia, apia, tnayara baikan kabia sin airaitka brisa wal baku main kaikan kaia.
+

Kulkanka 26

+
    +
  1. Upla bani ba, airaitka brisa aaisinska kwakaia, smalkanka ba pri natkara kaiasa, ulbaia ba pan, aisikaikaiaba pan. Baku sin, karhna munan kaiasa ulbaia ba, bara aisikaikaiaba, lan takaia; lila kulka naniba, sut lahma kaiasa baku sin, kul nanira dimaia ba sip takan kaiasa sut lahma, kumi bani daukan kaba kaiki.
    2. +
  2. Kul smalkanka brinka kabia; upla ba; upla baku lukanka brikaia dukiara, smalkaia, baku sin, upla bani airaitka ba kulkaia, bara, upla bani aiprika laka kum kum bri nmaniba kulkaia dukiara smalkan kaia; tanka pain briaia bra, aidahra pain walaia, bara, pana laka tasba wala nani aikuki bara, indian nani sut aikuki kau taura kulkan kaiasa, baku si kupya kumi laka, upla sut mata, tasba aiska asla takanka daukiba ta baikan kaiasa.
    3. +
  3. Tuktan nani aisika bani pa, sip kabia ailuhpya dia a dia lan takaia ba, witin pali pliki yabaia.
+

Kulkanka 27

+
    +
  1. Upla bani ba, pri lakara aitasbaya lukanka laka nani tilara, kaia, baku sin paskanka nani tilara, bara, sins laka tara nani pawanka dilara kaia, ban sin baha lilika briaia.
    2. +
  2. Upla bani ba, airaitka brisa airispik ka laka ba, bara, aidukia nani ba sin, main kaikan kaia, witin aisinska tihukani, aiulbanka nani bak kra, apia, aipaskanka nani bak brisa kaka.
+

Kulkanka 28

+

Upla bani ba, airaitka brisa, tasba aiskara, bara, aitasbayara sin la kat, bara, wapni laka kata iwaia; naha laka bapan na; upla nani raitka ba kulkaia, bara, pri laka ba kulkaia nani ba, sut alkaia mata.

+

Kulkanka 29

+
    +
  1. Upla sut bui ai tawan kara, rispik ka ba yaban kaiasa bara baman upla baku ai auya pah pawisa.
    2. +
  2. Ai raitka nani ba , kulki, bara, ai prika lakaba wal, ai auya pah kaiasa kaka, upla bani ba la nani bapanba yabalka kat wapaia sa, baku mika, upla wala nani raitka, bara, prika laka aniba kaikaia, kulkaia sin, baku rispik ka yaia la kat iwaia, bara pana pana kupya pliki natka nani iwaia sa, kaka.
    3. +
  3. Naha raitka naniba, bara prika laka nani ba kulkan kaia apiasa, Tasba Aiska Asla Takanka lukanka mapara sa kaka.
+

Kulkanka 30

+

Naha laka bapanna, gavament ra kabia, dakni kumra kabia, upla kumra kabia, bilka ya bansa lukan kaia apia sa, raitka nani, bara, prika laka nani naha lakara aisan na, alki taibi munaia upla wala nanira.

+ +
+
+
+ + + + + + + + +
+
+ +
+ +
+
+
+ +
+ + +
+
+ + + + + + + + + + + + + diff --git a/evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py b/evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py new file mode 100644 index 0000000000..93e248b107 --- /dev/null +++ b/evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py @@ -0,0 +1,96 @@ +# %% +import json +import re + +import requests +from bs4 import BeautifulSoup +from markdownify import markdownify as md + +articles_to_scrape = [ + "https://en.wikipedia.org/wiki/Mosquito", + "https://en.wikipedia.org/wiki/Mosquito_Coast", + "https://en.wikipedia.org/wiki/Nicaragua", + "https://en.wikipedia.org/wiki/Honduras", + "https://en.wikipedia.org/wiki/Miskito_language", + "https://en.wikipedia.org/wiki/Miskito_people", +] +dirpath = "evals/registry/data/skill_acquisition/distractor_articles/" + + +def clean_soup(content): + for infobox_tag in content.find_all("table", class_="infobox"): + infobox_tag.decompose() + for figure_tag in content.find_all("figure"): + figure_tag.decompose() + for style_tags in content.find_all("style"): + style_tags.decompose() + reflist_div = '
") + + sections = {} + for heading_text in headings: + if "" not in heading_text: + sections["Introduction"] = clean_heading_text(heading_text) + continue + span = heading_text[: heading_text.index("")] + heading_title = BeautifulSoup(span, "html.parser").contents[0].contents[0] + text = heading_text[heading_text.index("") + 5 :] + if heading_title not in ["References", "See also", "External links", "Footnotes"]: + sections[heading_title] = clean_heading_text(text) + + article_title = article.split("/")[-1] + + print(f"Scraped {article_title} successfully. Headings: {sections.keys()}\n") + filename = f"{article_title.lower()}.jsonl" + + with open(dirpath + filename, "w") as f: + for k, v in sections.items(): + f.write(json.dumps({"title": k, "content": v}, ensure_ascii=False) + "\n") + +# Separate code to scrape human rights article, as it's in a different format. +with open("human_rights.html", "r") as f: + html = f.read() + +soup = BeautifulSoup(html, "html.parser") +content = soup.find("div", class_="migrated-content") +md_content = md(str(content)).replace("\xa0", " ").replace("\u3000", " ") + +with open(dirpath + "human_rights_miskito.jsonl", "w") as f: + f.write( + json.dumps( + {"title": "Declaration of Human Rights in Miskito", "content": md_content}, + ensure_ascii=False, + ) + + "\n" + ) diff --git a/evals/elsuite/skill_acquisition/scraping/scrape_miskito.py b/evals/elsuite/skill_acquisition/scraping/scrape_miskito.py new file mode 100644 index 0000000000..697b5667cd --- /dev/null +++ b/evals/elsuite/skill_acquisition/scraping/scrape_miskito.py @@ -0,0 +1,135 @@ +# %% +import json + +import bs4 +import requests +from bs4 import BeautifulSoup +from markdownify import markdownify as md + +# TODO: make sure italicised text is crawled properly and that hints are excluded from answers. +# TODO: Split any multi-part questions into individual questions. + +miskito_base_url = "https://en.wikibooks.org/wiki/Miskito/Lesson_{idx}" + + +def process_practice_section_div(practice_div: bs4.element.Tag): + tds = practice_div.find_all("td") + instructions = ( + md(str(tds[1])) + .replace("*", "") + .replace("|", "") + .strip() + .replace("What do these mean?", "Translate to English:") + .replace("What do these sentences mean?", "Translate to English:") + ) + question_text = tds[2] + questions = question_text.find_all("li") + questions = [str(q.contents[0]) for q in questions] + answer_text = tds[3] + answers = answer_text.find_all("li") + answers = [str(a.contents[0]) for a in answers] + return instructions, questions, answers + + +def extract_toc_sections(content: bs4.element.Tag): + toc = content.find_all("div", class_="toc")[0] + lis = toc.find_all("li", class_="toclevel-1") + lis = [li.find_all("span", class_="toctext")[0].contents[0] for li in lis] + + lis = [md(str(li)).strip().replace("*", "") for li in lis] + return lis + + +def process_miskito_page(): + qa_pairs_by_lesson = {} + articles_without_qa_pairs = [] + for idx in range(1, 11): + response = requests.get(miskito_base_url.format(idx=idx)) + soup = BeautifulSoup(response.text, "html.parser") + content = soup.find("div", class_="mw-content-ltr mw-parser-output") + + # Extract the question-answer pairs. + divs_with_specific_style = content.find_all( + "div", style=lambda value: value and "width:300px; float:right;" in value + ) + lesson_qa_pairs = [] + for i, div in enumerate(divs_with_specific_style): + if i == 0 and idx == 1: # First section of first lesson is not in the same format. + instructions = "Translate to English:" + questions = div.find_all("ul")[0].find_all("li") + questions = [str(q.contents[0]) for q in questions] + answers = div.find_all("ul")[1].find_all("li") + answers = [str(a.contents[0]) for a in answers] + lesson_qa_pairs += [ + {"question": q, "answer": a, "instructions": instructions} + for q, a in zip(questions, answers) + ] + continue + instructions, questions, answers = process_practice_section_div(div) + for q, a in zip(questions, answers): + lesson_qa_pairs += [{"question": q, "answer": a, "instructions": instructions}] + qa_pairs_by_lesson[f"lesson_{idx}"] = lesson_qa_pairs + + # Remove them from the page and store the page contents. + for div in divs_with_specific_style: + div.decompose() + + articles_without_qa_pairs += [content] + + return qa_pairs_by_lesson, articles_without_qa_pairs + + +# %% +# Write to file: all questions by lesson, and all questions in evallib format. +qa_pairs_by_lesson, clean_articles = process_miskito_page() +qa_by_lesson_file = "miskito_qa_pairs_by_lesson.jsonl" + +with open(qa_by_lesson_file, "w") as f: + for lesson, qa_pairs in qa_pairs_by_lesson.items(): + f.write(json.dumps({"lesson": lesson, "qa_pairs": qa_pairs}) + "\n") + +miskito_qa = "miskito_qa.jsonl" +with open(miskito_qa, "w") as f: + for lesson, qa_list in qa_pairs_by_lesson.items(): + for qa_dict in qa_list: + instructions = qa_dict["instructions"][:-1] + ": " + f.write( + json.dumps( + { + "input": [{"role": "user", "content": instructions + qa_dict["question"]}], + "ideal": qa_dict["answer"], + }, + ensure_ascii=False, + ) + + "\n" + ) +# %% +as_text = [str(a).split("

")[1:] for a in clean_articles] +sections_by_heading = {} +for article in as_text: + for heading in article: + hsoup = BeautifulSoup(heading, "html.parser") + heading_name = ( + md(str(hsoup.find("span", class_="mw-headline").contents[0])).replace("*", "").strip() + ) + hsoup.find("span", class_="mw-editsection").decompose() + content = ( + md(str(hsoup)) + .strip() + .replace("*", "") + .replace("|", "") + .replace("What do they mean?", "") + .replace(" --- ", "") + .replace("\u2003", " ") + .replace(" ", " ") + ) + content = content.split(" Study ")[1] if "Study " in content else content + sections_by_heading[heading_name] = content.strip() + +sections_by_heading +# %% +file = "lessons_no_exercises.jsonl" +with open(file, "w") as f: + for heading, content in sections_by_heading.items(): + f.write(json.dumps({"title": heading, "content": content}, ensure_ascii=False) + "\n") +# %% diff --git a/evals/elsuite/skill_acquisition/scripts/make_plots.py b/evals/elsuite/skill_acquisition/scripts/make_plots.py new file mode 100644 index 0000000000..01eab83412 --- /dev/null +++ b/evals/elsuite/skill_acquisition/scripts/make_plots.py @@ -0,0 +1,204 @@ +import argparse +import os +from pathlib import Path + +import matplotlib.pyplot as plt +import pandas as pd +import seaborn as sns + +from evals.utils import log_utils + +PLOT_TITLES_BY_METRIC = { + "overall_accuracy": "Accuracy", # ie. both retrieval and non-retrieval in one plot + "baseline_accuracy": "Baseline accuracy (non-retrieval)", + "retrieval_accuracy": "Retrieval accuracy", + "average_retrieval_precision": "Average retrieval precision", + "average_non_retrieval_bleu_score": "Average non-retrieval BLEU score", + "average_retrieval_bleu_score": "Average retrieval BLEU score", + "average_retrieval_calls": "Average retrieval calls", + "average_invalid_retrieval_calls": "Average invalid retrieval calls", + "bleu_score": "BLEU score", + "correct_call_rate": "Correct call rate", + "invalid_call_rate": "Invalid call rate", + "timeout_rate": "Timeout rate", + "ctx_len_exceeded_rate": "Context length exceeded rate", +} + +UNIT_METRICS = set( + ["correct_call_rate", "invalid_call_rate", "timeout_rate", "ctx_len_exceeded_rate"] +) + + +def extract_metrics(datadir: Path) -> pd.DataFrame: + df_rows = [] + for path, results in sorted(list(log_utils.get_final_results_from_dir(datadir).items())): + spec = log_utils.extract_spec(path) + solver_path = Path(spec["completion_fns"][0]) + model = solver_path.name + solver = solver_path.parent.name + # Remove root section of path, which is the eval name + solver_path = solver_path.relative_to(solver_path.parts[0]) + df_rows.append({"solver": solver, "model": model, **results}) + df = pd.DataFrame(df_rows) + + return df + + +def make_plot( + df: pd.DataFrame, + outpath: Path, + metric="baseline_accuracy", + min_ylim=0, + max_ylim=0.08, + dataset="miskito", +): + plt.figure() + sns.set_theme(style="whitegrid") + # Calculating mean and SEM + grouped = df.groupby(["model", "solver"])[metric].agg(["mean", "sem"]).reset_index() + + def compute_sem(x): + sem = x.std() / (len(x) ** 0.5) + sem2 = sem * 2 # 95% confidence interval + return (x.mean() - sem2, x.mean() + sem2) + + # Plotting + sns.set(style="whitegrid") + sns.barplot(x="model", y="mean", hue="solver", data=grouped, errorbar=compute_sem, capsize=0.1) + plt.xticks(rotation=30, ha="right") + plt.ylim(min_ylim, max_ylim) + + # Some of the metrics are in [0, 1]. + if metric in UNIT_METRICS: + plt.ylim(0, 1) + + plt.title(PLOT_TITLES_BY_METRIC[metric] + f" on {dataset.capitalize()} Q&A dataset") + plt.xlabel("Model") + plt.tight_layout() + plt.savefig(outpath) + plt.close() + + +def make_side_bar_plot( + df: pd.DataFrame, + outpath: Path, + metric="overall_accuracy", + min_ylim=0, + max_ylim=0.1, + dataset="miskito", +): + if metric == "overall_accuracy": + df_clean = df[["model", "solver", "baseline_accuracy", "retrieval_accuracy"]] + elif metric == "bleu_score": + df_clean = df[ + ["model", "solver", "average_non_retrieval_bleu_score", "average_retrieval_bleu_score"] + ] + + fig, ax = plt.subplots(figsize=(10, 5)) + # df_clean = df_clean.drop(columns=["solver"]) + df_clean.set_index(["model", "solver"], inplace=True) + + # Group by 'model' and calculate mean and SEM + grouped = df_clean.groupby(["model", "solver"]).agg(["mean", "sem"]) + xlabels = [f"{model}/{solver}" for model, solver in grouped.index] + + # Prepare data for plotting + means = grouped.xs("mean", axis=1, level=1) + errors = grouped.xs("sem", axis=1, level=1) + + # Plotting + means.plot(kind="bar", yerr=errors, capsize=4, ax=ax) # Removed 'stacked=True' + + ax.set_ylabel(metric) + ax.set_xticklabels(xlabels, rotation=30, ha="right") + ax.set_xlabel("model/solver") + ax.set_ylim(min_ylim, max_ylim) + + fig.tight_layout(pad=3.0) + fig.suptitle(PLOT_TITLES_BY_METRIC[metric] + f" on {dataset.capitalize()} dataset") + fig.savefig(outpath) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--log-dir", "-d", type=str, required=True) + parser.add_argument("--out-dir", "-o", type=str, default="./outputs") + args = parser.parse_args() + log_dir = Path(args.log_dir) + out_dir = Path(args.out_dir) + + out_dir.mkdir(exist_ok=True, parents=True) + + datasets = os.listdir(log_dir) + + for dataset in datasets: + print(f"Extracting data for eval dataset {dataset}...") + df = extract_metrics(log_dir / dataset) + + # Rename some of the solver values so they can be represented in the same plot. + df.loc[df["solver"] == "cot_hhh", "solver"] = "cot" + df.loc[df["solver"] == "hhh", "solver"] = "direct" + df.loc[df["solver"] == "fewshot_direct", "solver"] = "fewshot" + + # TODO: report directly as 'average_correct_calls' in future and remove this rename. + df.rename(columns={"average_retrieval_precision": "average_correct_calls"}, inplace=True) + df["correct_call_rate"] = df["average_correct_calls"] / df["average_retrieval_calls"] + df["invalid_call_rate"] = ( + df["average_invalid_retrieval_calls"] / df["average_retrieval_calls"] + ) + + print(f"Plotting other metrics for eval dataset {dataset}...") + + # Generate bar plots for all other metrics. + core_metrics = ( + [] + ) # ["baseline_accuracy", "retrieval_accuracy", "average_non_retrieval_bleu_score", "average_retrieval_bleu_score"] + auxiliary_metrics = [ + "correct_call_rate", + "invalid_call_rate", + "timeout_rate", + "ctx_len_exceeded_rate", + ] + for metric in core_metrics + auxiliary_metrics: + make_plot( + df[["model", "solver", metric]].copy(), + out_dir / f"{dataset}_{metric}.png", + metric, + dataset=dataset, + ) + + print(f"Plotting headline metrics for eval dataset {dataset}...") + + # Generate stacked bar plots for the two headline metrics. + for metric in ["overall_accuracy", "bleu_score"]: + make_side_bar_plot(df, out_dir / f"{dataset}_{metric}.png", metric, dataset=dataset) + + # Print numerical results (and compute % improvement metrics) + grouped = df.groupby(["model", "solver"]).agg(["mean", "sem"]) + for type, closedbook, openbook in [ + ( + "Translation (BLEU)", + "average_non_retrieval_bleu_score", + "average_retrieval_bleu_score", + ), + ("Non-translation (%)", "baseline_accuracy", "retrieval_accuracy"), + ]: + print(f"Improvement Metrics for {type} on {dataset.capitalize()} dataset") + improvement_rows = [] + for idx, row in grouped.iterrows(): + openbook_score = row[openbook]["mean"] + closedbook_score = row[closedbook]["mean"] + rel_improvement_score = (openbook_score - closedbook_score) / (1 - closedbook_score) + improvement_rows.append( + { + "model": idx[0], + "solver": idx[1], + "closedbook": closedbook_score, + "openbook": openbook_score, + "improvement": rel_improvement_score, + } + ) + improvement_df = pd.DataFrame(improvement_rows) + print(improvement_df) + # print to stdout as csv + print(improvement_df.to_csv(index=False)) diff --git a/evals/elsuite/skill_acquisition/scripts/run_experiments.sh b/evals/elsuite/skill_acquisition/scripts/run_experiments.sh new file mode 100755 index 0000000000..aaf81e0745 --- /dev/null +++ b/evals/elsuite/skill_acquisition/scripts/run_experiments.sh @@ -0,0 +1,76 @@ +logdir=./logs +outputdir=./outputs + +timestamp=$(date +%Y%m%d_%H%M%S) +logpathbase=$logdir/$timestamp/ + +size=full +num_repeats=1 +eval_variants_zero_shot=("skill_acquisition.miskito.zero_shot.$size") + +# Check for --num_repeats argument +for arg in "$@" +do + if [[ $arg == --num_repeats=* ]]; then + num_repeats="${arg#*=}" + fi +done + + +echo Running experiments and logging to $logpathbase + +declare -a ZEROSHOT_SOLVERS=( + # Solvers for gpt-3.5-turbo + "generation/direct/gpt-3.5-turbo" + "skill_acquisition/cot/gpt-3.5-turbo" + + + # Solvers for gpt-4-turbo-preview + "generation/direct/gpt-4-turbo-preview" + "skill_acquisition/cot/gpt-4-turbo-preview" +) + +declare -a FEWSHOT_SOLVERS=( + "miskito_all/fewshot_direct/gpt-3.5-turbo" + "miskito_all/fewshot_direct/gpt-4-turbo-preview" +) + +if [ ! -d "$logpathbase/miskito" ]; then + mkdir -p "$logpathbase/miskito" +fi + + +# Run zero-shot experiments. +for eval_variant in "${eval_variants_zero_shot[@]}" +do + if [[ $eval_variant == *"miskito"* ]]; then + record_path="$logpathbase/miskito" + fi + + for solver in "${ZEROSHOT_SOLVERS[@]}" + do + for ((i=1;i<=num_repeats;i++)); do + echo "Running $solver, iteration $i" + oaieval $solver $eval_variant --record_path "$record_path/$solver-$i.log" + done + done +done + +# Run few-shot experiments. +# Miskito +for solver in "${FEWSHOT_SOLVERS[@]}" +do + if [[ $solver == *"miskito"* ]]; then + for ((i=1;i<=num_repeats;i++)); do + echo "Running $solver, iteration $i" + oaieval $solver skill_acquisition.miskito.few_shot.$size --record_path "$logpathbase/miskito/$solver-$i.log" + done + fi +done + +echo Done running experiments, all logs in $logpathbase + +echo Producing plots, outputs to $outputdir + +# Produce results +python make_plots.py --log-dir $logpathbase --out-dir $outputdir \ No newline at end of file diff --git a/evals/elsuite/skill_acquisition/solvers.py b/evals/elsuite/skill_acquisition/solvers.py new file mode 100644 index 0000000000..16bd72f653 --- /dev/null +++ b/evals/elsuite/skill_acquisition/solvers.py @@ -0,0 +1,22 @@ +from evals.solvers.openai_assistants_solver import OpenAIAssistantsSolver +from evals.solvers.solver import SolverResult +from evals.task_state import TaskState + + +class SkillAcquisitionAssistantsSolver(OpenAIAssistantsSolver): + def _solve( + self, + task_state: TaskState, + **kwargs, + ) -> SolverResult: + + if task_state.current_state is not None: + task_state.task_description = """ +You are about to take a Q&A test in a language called {target_language}. You should do your best to answer each question correctly. Your score will be your number of correct answers divided by the number of questions. + +This is an open book test, and you have been provided with relevant materials to help you answer the questions via your file browser. + +To provide your final answer, output [ANSWER X], where X is the answer. For example, if you answer is 'this is a bear', output [ANSWER this is a bear] +""".strip() + + return super()._solve(task_state, **kwargs) diff --git a/evals/elsuite/skill_acquisition/task_description.py b/evals/elsuite/skill_acquisition/task_description.py new file mode 100644 index 0000000000..5a356179d0 --- /dev/null +++ b/evals/elsuite/skill_acquisition/task_description.py @@ -0,0 +1,3 @@ +TASK_DESCRIPTION = """You are about to take a Q&A test in a language called {target_language}. You should do your best to answer each question correctly. Your score will be your number of correct answers divided by the number of questions. + +To provide your final answer, output [ANSWER X], where X is the answer. For example, if you answer is 'this is a bear', output [ANSWER this is a bear]""" diff --git a/evals/elsuite/skill_acquisition/test_skill_acquisition.py b/evals/elsuite/skill_acquisition/test_skill_acquisition.py new file mode 100644 index 0000000000..079fdcb435 --- /dev/null +++ b/evals/elsuite/skill_acquisition/test_skill_acquisition.py @@ -0,0 +1,118 @@ +import json +import os +from pathlib import Path + +from evals.elsuite.skill_acquisition.eval import SkillAcquisition +from evals.elsuite.skill_acquisition.utils import ( + answer_detected, + process_answer, + process_view_instruction, + view_instruction_detected, +) +from evals.registry import Registry + +registry = Registry() + +dummy_eval_spec = { + "eval_registry_path": Path("evals/registry"), + "completion_fns": [registry.make_completion_fn("gpt-4")], + "samples_jsonl": "skill_acquisition/miskito/miskito_test.jsonl", + "target_language": "miskito", + "n_samples": 5, + "knowledge_base_directory": "skill_acquisition/miskito/knowledge_base/", + "max_replies": 50, +} + + +def test_answer_detected(): + assert answer_detected("[ANSWER foo]") is True + assert answer_detected("[ANSWER: foo]") is True + assert answer_detected("ANSWER foo") is False + assert answer_detected("[ANSWER foo") is False + assert answer_detected("ANSWER foo]") is False + assert answer_detected("[ANSWER foo][ANSWER bar]") is True + + +def test_view_instruction_detected(): + SkillAcquisition(**dummy_eval_spec) + assert view_instruction_detected("[VIEW file1]") is True + assert view_instruction_detected("[VIEW: file1]") is True + assert view_instruction_detected("[VIEW file1 section1]") is True + assert view_instruction_detected("[VIEW: file1 section1]") is True + assert view_instruction_detected("VIEW file1") is False + assert view_instruction_detected("[VIEW file1") is False + assert view_instruction_detected("VIEW file1]") is False + assert view_instruction_detected("[VIEW file1][VIEW file2]") is True + assert view_instruction_detected("[VIEW: file1][VIEW: file2]") is True + + +def test_process_answer(): + SkillAcquisition(**dummy_eval_spec) + assert process_answer("[ANSWER foo]") == "foo" + assert process_answer("[ANSWER: foo]") == "foo" + assert process_answer("[ANSWER foo bar baz]") == "foo bar baz" + assert process_answer("[ANSWER: foo bar baz]") == "foo bar baz" + assert process_answer("[ANSWER foo][ANSWER bar]") == "bar" + assert process_answer("[ANSWER foo][ANSWER bar") == "foo" + + +def test_process_view_instruction(): + SkillAcquisition(**dummy_eval_spec) + assert process_view_instruction("[VIEW file1]") == ("file1", None) + assert process_view_instruction("[VIEW: file1]") == ("file1", None) + assert process_view_instruction("[VIEW file1 section1]") == ( + "file1", + "section1", + ) + assert process_view_instruction("[VIEW: file1 section1]") == ( + "file1", + "section1", + ) + assert process_view_instruction("[VIEW file1][VIEW file2]") == ( + "file2", + None, + ) + assert process_view_instruction("[VIEW: file1][VIEW: file2]") == ( + "file2", + None, + ) + assert process_view_instruction("[VIEW file1 section1][VIEW file2 section2]") == ( + "file2", + "section2", + ) + + +def test_process_view_instruction_spaces_and_quotes(): + assert process_view_instruction("[VIEW file1 sectionpart1 sectionpart2]") == ( + "file1", + "sectionpart1 sectionpart2", + ) + assert process_view_instruction("[VIEW file1 sectionpart1 'sectionpart2']") == ( + "file1", + "sectionpart1 'sectionpart2'", + ) + + +def test_view_content(): + skill_acquisition_eval = SkillAcquisition(**dummy_eval_spec) + + # Create a file to view first. + filepath = skill_acquisition_eval.knowledge_base_directory / "test_file.jsonl" + with open(filepath, "w") as f: + f.write(json.dumps({"title": "foo", "content": "Test file contents."}) + "\n") + + content, sections_visible_to_model, sections_viewed = skill_acquisition_eval._view_content( + "test_file.jsonl" + ) + assert content == "Table of contents for test_file.jsonl: {'foo'}." + assert sections_visible_to_model == {"test_file.jsonl": {"foo"}} + assert sections_viewed == {"test_file.jsonl": {"Table of Contents"}} + + content, sections_visible_to_model, sections_viewed = skill_acquisition_eval._view_content( + "test_file.jsonl", "foo" + ) + assert content == "Test file contents." + assert sections_visible_to_model == {"test_file.jsonl": {"foo"}} + assert sections_viewed == {"test_file.jsonl": {"Table of Contents", "foo"}} + + os.remove(filepath) diff --git a/evals/elsuite/skill_acquisition/utils.py b/evals/elsuite/skill_acquisition/utils.py new file mode 100644 index 0000000000..ef0ef09349 --- /dev/null +++ b/evals/elsuite/skill_acquisition/utils.py @@ -0,0 +1,179 @@ +import json +import logging +import random +import re +from math import sqrt +from typing import Callable, Dict, List, Set, Union + +import numpy as np +from nltk.translate.bleu_score import SmoothingFunction, sentence_bleu + +ANSWER_PATTERN = r"\[ANSWER:? .+?\]" +VIEW_PATTERN = r"\[VIEW:? .+?\]" +PROMPTS = { + "retrieval_instructions": ( + "This is an open-book test." + " If you are unsure about a question or unable to answer, you may read through a number of" + " knowledge sources that are made available to you in the form of text files." + " To open a file, output [VIEW X] where X is the name of the source you would like." + " For example, to view a file called help.jsonl, output [VIEW help.jsonl]." + " When you first open a file I will tell you its table of contents." + " You may view a particular section by outputting [VIEW X Y] where X is the name of" + " the file and Y is the name of the section you would like to read." + " The section titles won't necessarily match your exact task." + " You should explore them for any useful information, such as key words or phrases," + " before giving your answer. You should always try to provide an answer," + " even if you are not confident. Failure to provide an answer is penalised" + " more strongly than incorrect answers." + "\nHere are the sources available to you: {list_of_files}." + ), + "intermediate_prompt": "You've already viewed the following files and sections: {sections}.\nYou can view another file or section by outputting [VIEW X] or [VIEW X Y], or you can answer the question by outputting [ANSWER X].", + "present_content": "You asked to view file {file}, section {section}. Here is the content: {content}", + "wrong_file": "You tried to view {file}, which does not exist in the knowledge base. Choose another file from {knowledge_base}.", + "wrong_section": "You tried to view section {section} in file {file}, which does not exist. The table of contents for that file contains: {table_of_contents}.", +} + +logger = logging.getLogger(__name__) + + +def answer_detected(output: str) -> bool: + return len(re.findall(ANSWER_PATTERN, output)) > 0 + + +def view_instruction_detected(output: str) -> bool: + return len(re.findall(VIEW_PATTERN, output)) > 0 + + +def process_answer(output: str) -> str: + """Extracts the answer from model output. + The answer looks like [ANSWER X], where X is the answer. + + Args: + output (str): model output + + Returns: + str: answer provided by the model + """ + maybe_multiple_answers = re.findall(ANSWER_PATTERN, output) + + # Sanity check – this should never happen. + assert len(maybe_multiple_answers) > 0, f"No answer detected in {output}." + + if len(maybe_multiple_answers) > 1: + logger.debug( + f"Multiple answers detected, using only the final answer: {maybe_multiple_answers}" + ) + + final_answer_instruction = maybe_multiple_answers[-1] + final_answer = " ".join(final_answer_instruction.split(" ")[1:])[:-1] + + return final_answer + + +def process_view_instruction(output: str) -> Union[tuple[str, str], tuple[str, None]]: + """Extracts the target of a view instruction from model output. + The view instruction looks like [VIEW X Y], where X is a file name and Y is a section name. + This function extracts X and Y. + + Args: + output (str): model output + + Returns: + Union[tuple[str, str], tuple[str, None]]: tuple of file name and if applicable section name to view + """ + maybe_multiple_views = re.findall(VIEW_PATTERN, output) + + # Sanity check – this should never happen. + assert len(maybe_multiple_views) > 0, f"No view instruction detected in {output}." + + if len(maybe_multiple_views) > 1: + logger.debug( + f"Multiple view instructions detected, using only the final instruction: {maybe_multiple_views}" + ) + + final_view_instruction = maybe_multiple_views[-1][1:-1].split(" ")[1:] + file = final_view_instruction[0].strip() + + section = ( + None if len(final_view_instruction) == 1 else " ".join(final_view_instruction[1:]).strip() + ) + + return (file, section) + + +def _get_average_metric( + results: List[Dict[str, str]], metric_fn: Callable[List[Dict[str, str]], List[float]] +) -> float: + total_metric = sum(metric_fn(results)) + num_total = len(results) + if num_total == 0: + return float("nan") + else: + return total_metric / num_total + + +def get_bootstrap_accuracy_std(results: List[Dict[str, str]], num_samples: int = 1000) -> float: + results = [sample for sample in results if sample["question_type"] != "translation"] + vals = [result["correct"] for result in results] + return np.std([np.mean(random.sample(vals, len(vals) // 2)) for _ in range(1000)]) + + +def render_intermediate_prompt(sections_viewed: Dict[str, Set]) -> str: + return PROMPTS["intermediate_prompt"].format( + sections=json.dumps( + {k: list(v) for k, v in sections_viewed.items()}, indent=4 + ) # Cannot serialise sets directly. + ) + + +def get_question_type(question: str) -> str: + return "translation" if question.strip().startswith("Translate") else "non-translation" + + +def get_average_bleu_score(results: List[Dict[str, str]]) -> float: + results = [sample for sample in results if sample["question_type"] == "translation"] + return _get_average_metric( + results, + lambda samples: [ + get_bleu_score(sample["expected"][0], sample["parsed_output"]) for sample in samples + ], + ) + + +def get_bleu_score(expected: str, sampled: str) -> float: + punctuation = r"[^\w\s]" + + return sentence_bleu( + [re.sub(punctuation, "", expected).split()], + re.sub(punctuation, "", sampled).split(), + smoothing_function=SmoothingFunction().method1, + ) + + +def get_accuracy(results: List[Dict[str, str]]) -> float: + results = [sample for sample in results if sample["question_type"] != "translation"] + return _get_average_metric( + results, lambda samples: [int(sample["correct"]) for sample in samples] + ) + + +def get_average_retrieval_calls(results: List[Dict[str, str]]) -> float: + return _get_average_metric( + results, lambda samples: [sample["total_retrieval_calls"] for sample in samples] + ) + + +def get_average_invalid_retrieval_calls(results: List[Dict[str, str]]) -> float: + return _get_average_metric( + results, lambda samples: [sample["invalid_retrieval_calls"] for sample in samples] + ) + + +def get_average_retrieval_precision(results: List[Dict[str, str]]) -> float: + return _get_average_metric( + results, lambda samples: [sample["lesson_retrieval_calls"] for sample in samples] + ) + + +def get_std_of_difference(baseline_std: float, retrieval_std: float) -> float: + return sqrt(baseline_std**2 + retrieval_std**2) diff --git a/evals/elsuite/solver_tools_convo.py b/evals/elsuite/solver_tools_convo.py new file mode 100644 index 0000000000..8a13adf80b --- /dev/null +++ b/evals/elsuite/solver_tools_convo.py @@ -0,0 +1,240 @@ +import copy +import logging +import re +from dataclasses import dataclass +from typing import Any, Optional + +from evals.elsuite.bugged_tools.tools import Tool, ToolTaskState +from evals.solvers.solver import Solver, SolverResult +from evals.task_state import Message, TaskState + +logger = logging.getLogger(__name__) + + +@dataclass +class ToolCall: + tool_name: str + input: str + output: Any + + +@dataclass +class ParsedSolverResult: + tool_calls: list[ToolCall] + final_answer: Optional[str] + + +@dataclass +class RunnerResult: + final_task_state: ToolTaskState + final_solver_result: SolverResult + metrics: dict + + +class Runner: + def __init__( + self, + solver: Solver, + sample: Any, + name_to_tool: dict, + max_turns: int, + default_task_description: str, + default_reminder_message: str, + ): + self.solver = solver + self.sample = sample + self.name_to_tool = name_to_tool + self.max_turns = max_turns + self.default_task_description = default_task_description + self.default_reminder_message = default_reminder_message + + def run(self) -> RunnerResult: + # Prepare initial task state + tools = self.name_to_tool.values() + tool_names_and_descriptions = self._get_tool_names_and_descriptions(tools) + task_description = self.default_task_description.format( + tool_names_and_descriptions=tool_names_and_descriptions + ) + task_message = self.sample["task"] + messages = [ + Message(role="user", content=task_message), + ] + task_state = TaskState( + task_description=task_description, + messages=messages, + current_state=None, + ) + + # Loops until solver completes task or hits turn limit + turn = 0 + final_answer = None + while turn < self.max_turns: + # Get result from solver + solver_result = self.solver(task_state) + parsed_solver_result = self._parse_solver_result(solver_result) + final_answer = parsed_solver_result.final_answer + + # If solver failed to call tool or give final answer, prompt them to try again + if parsed_solver_result.tool_calls == [] and final_answer is None: + content = self.default_reminder_message + task_state = self._add_eval_message(task_state, solver_result, content=content) + turn += 1 + continue + + if final_answer is not None: + return self._finish_run(task_state, solver_result, final_answer, turn) + + # Run tools. If solver gave tool incorrect input, prompt them to try again. + assert parsed_solver_result.tool_calls != [] + tool_outputs = [self._run_tool_call(i) for i in parsed_solver_result.tool_calls] + if any([i is None for i in tool_outputs]): + content = self.default_reminder_message + task_state = self._add_eval_message(task_state, solver_result, content=content) + turn += 1 + continue + + # Add user message containing tool outputs + task_state = self._add_tool_outputs(task_state, solver_result, tool_outputs) + turn += 1 + + return self._finish_run(task_state, solver_result, None, turn) + + def _get_tool_names_and_descriptions(self, tools: list[Tool]): + """ + Given sequence of tools, creates a string of each tools name + and description, each tool's info separated by a newline + """ + s = "" + for tool in tools: + s += f"{tool._name}: {tool._desc}\n" + return s + + def _parse_solver_result(self, solver_result: SolverResult) -> ParsedSolverResult: + output = solver_result.output + tool_calls = self._parse_tool_calls(output) + final_answer = self._parse_final_answer(output) + return ParsedSolverResult(tool_calls=tool_calls, final_answer=final_answer) + + def _parse_tool_calls(self, output: str) -> Optional[list[ToolCall]]: + tool_message_matches = self._find_tool_messages(output) + if tool_message_matches == []: + return [] + + tool_calls = [] + for tool_name, tool_message in tool_message_matches: + # Log warning if solver calls a tool that doesn't exist + try: + self.name_to_tool[tool_name] + except KeyError: + logger.warn(f"Solver tried to call '{tool_name}' tool which doesn't exist!") + continue + + tool_call = ToolCall(tool_name=tool_name, input=tool_message, output=None) + tool_calls.append(tool_call) + return tool_calls + + def _find_tool_messages(self, text: str) -> list[tuple[str, str]]: + """ + Finds all tool calls, which are formatted [NAME: INPUT], + where NAME != "Answer" and NAME != "Bugged" + """ + pattern = r"\(@(?!Answer|Bugged)(\w+): (.+?)\)" + matches = re.findall(pattern, text, re.DOTALL) + return matches + + def _parse_final_answer(self, output: str) -> Optional[str]: + """ + If a final answer exists of form [Answer: OUTPUT], returns the output, + otherwise returns None + """ + match = re.search(r"\(@Answer: (.*?)\)", output, re.DOTALL) + return match.group(1) if match else None + + def _run_tool_call(self, tool_call: ToolCall) -> ToolCall: + # Prepare task state + tool_name = tool_call.tool_name + tool = self.name_to_tool[tool_name] + tool_input = tool_call.input + tool_desc = self.name_to_tool[tool_name]._desc + + # Remove quotes if solver wrapped input + if tool_input.startswith(("'", '"')) and tool_input.endswith(("'", '"')): + tool_input = tool_input[1:-1] + + task_description = ( + f"Your name is {tool_name}. A description of your purpose is shown below:\n{tool_desc}" + ) + messages = [Message(role="user", content=tool_input)] + task_state = ToolTaskState( + task_description=task_description, messages=messages, current_state=None + ) + try: + out = tool(task_state) + except (TypeError, ValueError, IndexError): + out = None + + if out is None: + return None + + tool_call.output = out.output + return tool_call + + def _add_eval_message( + self, + task_state: TaskState, + solver_output: SolverResult, + content: str, + ) -> TaskState: + messages = copy.deepcopy(task_state.messages) + messages.append(Message(role="assistant", content=solver_output.output)) + # NOTE: we assume that the order of tool_outputs is the same as the order of tool_calls + + messages.append(Message(role="user", content=content)) + new_task_state = TaskState( + task_description=task_state.task_description, + messages=messages, + current_state=None, + ) + return new_task_state + + def _add_tool_outputs( + self, + task_state: TaskState, + solver_output: SolverResult, + tool_outputs: list[ToolCall], + ) -> TaskState: + content = "" + for tool_output in tool_outputs: + name = tool_output.tool_name + input = tool_output.input + output = tool_output.output + content += f"{name} output on input {input}: {output}\n" + + return self._add_eval_message(task_state, solver_output, content) + + def _finish_run( + self, + final_task_state: TaskState, + solver_result: SolverResult, + final_answer: Optional[str], + turn: int, + ) -> RunnerResult: + expected_answer = self.sample["answer"] + is_correct = False + if final_answer is not None: + final_answer = final_answer.lower().strip() + # Remove quotes if solver wrapped input + if final_answer.startswith(("'", '"')) and final_answer.endswith(("'", '"')): + final_answer = final_answer[1:-1] + is_correct = final_answer == expected_answer.lower().strip() + + metrics = { + "is_correct": is_correct, + "num_turns": turn + 1, # zero-indexed, + } + + return RunnerResult( + final_task_state, + solver_result, + metrics, + ) diff --git a/evals/elsuite/track_the_stat/README.md b/evals/elsuite/track_the_stat/README.md new file mode 100644 index 0000000000..20c1580b2f --- /dev/null +++ b/evals/elsuite/track_the_stat/README.md @@ -0,0 +1,134 @@ +# Track the Stat + +This eval measures how well models can implicitly keep track of task state, by +asking models to compute the rolling median or the rolling mode over a sequence +of integers. + +## Usage + +Run with: + +```bash +oaieval track_the_stat +``` + +We have found that `generation/direct/gpt-4-0125-preview` works well on this +eval. For more examples of tested solvers, see +[`./scripts/run_experiments.sh`](./scripts/run_experiments.sh). + +## Evaluation Process + +The evaluation process is as follows for a given sample from our dataset: + +1. The `TASK_DESCRIPTION` prompt is shown to the solver. +2. The sample contains an integer to use as a seed for a random number + generator. +3. The random number generator generates 300 random integers between 0 and 100, + with replacement. +4. The integers are shown one by one to the solver. +5. At each turn (i.e., after each integer is shown), the solver needs to respond + with the current rolling median or the current rolling mode of the integers + seen so far. +6. The solver's response is parsed and compared to the correct rolling median or + rolling mode. +7. If the solver's response is incorrect or a violation is raised (answered in + the incorrect format), the evaluation stops and we measure how many turns the + solver lasted for. If the solver's response is correct, we move on to the + next integer. + +## Prompts + +We refer readers to the [`./prompts/`](./prompts/) folder for the +`TASK_DESCRIPTION` used in the eval. + +## Metrics + +Below are the metrics returned by the eval: + + +| **Metric** | **Notes** | +|------------------- |-------------------------------------------------------------------------------------------------------------------------------------------- | +| avg_max_length | The maximum sequence length the model can handle before failing, averaged across the samples. Higher is better. Best possible is 300. | +| stddev_max_length | The standard deviation on the above. | +| median_max_length | The median of the maximum sequence length the model can handle before failing, across the samples. Higher is better. Best possible is 300. | +| max_max_length | The maximum sequence length the model handled before failing across all samples. | +| min_max_length | The minimum sequence length the model handled before failing across all samples. | +| violation_rate | how often the model responds in an invalid format. i.e. not using the `[: ]` format. | + + +## Variants + +The eval has two variants: median and mode. In the median variant, the solver +needs to track the rolling median. In the mode variant, the solver needs to +track the rolling mode. + +```bash +oaieval track_the_stat. +``` + +## Custom Solvers + +We implement 3 custom solvers for this eval in [./solvers.py](./solvers.py) + +1. `ExplicitStateSolver`: A nested solver that injects an explicit + representation of the task state after each number is seen. For example, for + the median task we inject the sorted list of numbers seen so far. For the + mode task, we inject a dictionary that maps each number seen so far to its + count. We view this solver as a baseline for the task, providing the + performance of the models on _explicit_ state tracking, rather than the + default _implicit_ state tracking. +2. `RandomBaselineSolver`: A solver that randomly chooses a number from the + numbers seen so far as the rolling median or mode. In case of even length + lists in the median variant, it chooses two random numbers and returns their + arithmetic mean. We view this baseline as equivalent to randomly guessing. +3. `TrackTheStatHuman`: A helper solver class that wraps the `HumanCliSolver` + class such that users do not have to wrap their answer in the + `[median: ]` or `[mode: ]` format and can instead just + directly type the number. + +## Token Usage Estimates + +Below are token usage estimates for a given run (one run = all samples) of the +eval. + +For the mode task: + +| Model (state tracking) | Input | Output | Total | +| ----------------------------- | --------- | --------- | ---------- | +| gpt-3.5-turbo-0125 (implicit) | 670,000 | 10,000 | 680,000 | +| gpt-3.5-turbo-0125 (explicit) | 2,710,000 | 30,000 | 2,740,000 | +| gpt-4-base (implicit) | 9,030,000 | 2,110,000 | 11,150,000 | +| gpt-4-base (explicit) | 3,720,000 | 960,000 | 4,680,000 | +| gpt-4-0125-preview (implicit) | 3,050,000 | 30,000 | 3,080,000 | +| gpt-4-0125-preview (explicit) | 8,580,000 | 50,000 | 8,630,000 | + +For the median task: + +| Model (state tracking) | Input | Output | Total | +| ----------------------------- | --------- | ------- | --------- | +| gpt-3.5-turbo-0125 (implicit) | 430,000 | 10,000 | 440,000 | +| gpt-3.5-turbo-0125 (explicit) | 880,000 | 10,000 | 890,000 | +| gpt-4-base (implicit) | 2,900,000 | 760,000 | 3,660,000 | +| gpt-4-base (explicit) | 3,250,000 | 810,000 | 4,060,000 | +| gpt-4-0125-preview (implicit) | 690,000 | 10,000 | 700,000 | +| gpt-4-0125-preview (explicit) | 1,430,000 | 20,000 | 1,450,000 | + +## Future modifications + +- Identify new variants of the task beyond median or mode, where the explicit + state is either impossible to represent or not useful for the task. This would + allow us to more comfortably measure the implicit state tracking, even on CoT + solvers. +- Identify more realistic and/or complex tasks. +- Introduce distractors. + +## Version History + +- v0: Initial version released + +## Contribution Statement + +Eval design, implementation, and results evaluation were primarily conducted by +Giulio Starace, under the guidance of (alphabetically by last-name) Steven +Adler, Andrei Alexandru, James Aung, and Chan Jun Shern who provided research +input, report revisions, and project management support. diff --git a/evals/elsuite/track_the_stat/eval.py b/evals/elsuite/track_the_stat/eval.py new file mode 100644 index 0000000000..d1ca65d719 --- /dev/null +++ b/evals/elsuite/track_the_stat/eval.py @@ -0,0 +1,96 @@ +import logging +import random +from typing import Any, Optional + +import numpy as np + +from evals.elsuite.track_the_stat import prompts, utils +from evals.eval import SolverEval +from evals.record import RecorderBase, record_metrics +from evals.solvers.solver import Solver +from evals.task_state import Message, TaskState + +logging.getLogger("httpx").setLevel(logging.WARNING) +logger = logging.getLogger(__name__) + + +class TrackTheStat(SolverEval): + def __init__(self, task: str, n_samples: Optional[int] = 250, *args, **kwargs): + super().__init__(*args, **kwargs) + assert task in [ + "median", + "mode", + ], f"task must be either 'median' or 'mode', but got {task}" + self.task = task + # warn, color in yellow + logger.warning( + utils.yellow_string( + "By nature of what is being evaluated, this eval assumes that the " + "solver cannot make use of external scratchpads or similar solutions " + "to explicitly write down the task state at every step. Using solvers " + "that allow for this functionality will likely produce invalid results." + ) + ) + self.task_desc = prompts.TASK_DESCRIPTION.format( + task=task, + task_further_details=prompts.task_to_further_details[task], + task_example=prompts.task_to_example[task], + ) + self.task_fn = utils.task_to_fn[task] + self.n_samples = n_samples + self.rng = random.Random(self.seed) + + def eval_sample(self, solver: Solver, sample: Any, rng: random.Random) -> None: + capped_inf_list = np.random.default_rng(sample["seed"]).integers(0, 100, size=300) + metrics = self._eval_sample(solver, capped_inf_list) + + record_metrics(**metrics) + + def _eval_sample(self, solver: Solver, capped_inf_list: list[int]) -> dict: + violation = False + task_state = TaskState(task_description=self.task_desc, messages=[]) + for i, num in enumerate(capped_inf_list): + curr_list = capped_inf_list[: i + 1] + task_state.messages.append(Message(role="user", content=str(num))) + task_state.current_state = utils.compute_state(curr_list, self.task) + solver_output = solver(task_state).output + solver_response = utils.parse_solver_output(solver_output, self.task) + if solver_response is None: + violation = True + break + if round(solver_response, 1) != round(self.task_fn(curr_list), 1): + break + task_state.messages.append(Message(role="assistant", content=solver_output)) + + return { + "max_length": len(curr_list) - 1, + "violation": violation, + } + + def run(self, recorder: RecorderBase): + samples = self._get_samples() + self.eval_all_samples(recorder, samples) + logged_metrics: list[dict] = recorder.get_metrics() + + agg_metrics = self._compute_agg_metrics(logged_metrics) + return agg_metrics + + def _compute_agg_metrics(self, logged_metrics: list[dict]) -> dict: + max_lengths = np.array([x["max_length"] for x in logged_metrics]) + + agg_metrics = { + "avg_max_length": np.mean(max_lengths), + "stddev_max_length": np.std(max_lengths), + "median_max_length": np.median(max_lengths), + "max_max_length": np.max(max_lengths), + "min_max_length": np.min(max_lengths), + "violation_rate": np.mean([x["violation"] for x in logged_metrics]), + } + # necessary for serialization, json doesn't like np floats + agg_metrics = {k: float(v) for k, v in agg_metrics.items()} + return agg_metrics + + def _get_samples(self) -> list[dict]: + samples = [{"seed": num} for num in range(self.n_samples)] + samples = self.rng.sample(samples, self.n_samples) # shuffling + return samples diff --git a/evals/elsuite/track_the_stat/prompts/__init__.py b/evals/elsuite/track_the_stat/prompts/__init__.py new file mode 100644 index 0000000000..31d1b2de39 --- /dev/null +++ b/evals/elsuite/track_the_stat/prompts/__init__.py @@ -0,0 +1,27 @@ +from . import mode +from . import median + + +TASK_DESCRIPTION = """\ +In the following messages, you will be given a sequence of numbers.\ + At each turn, you will be shown a number as input, and you should respond with the\ + {task} of all the input numbers shown to you so far. + +{task_further_details} + +Here is an example of what this may look like. +{task_example} + +Format your response as [{task}: ] (square brackets included), as shown in\ +the transcript above. The task will begin now. +""" + +task_to_example = { + "median": median.MEDIAN_EXAMPLE, + "mode": mode.MODE_EXAMPLE, +} + +task_to_further_details = { + "median": median.MEDIAN_FURTHER_DETAILS, + "mode": mode.MODE_FURTHER_DETAILS, +} diff --git a/evals/elsuite/track_the_stat/prompts/median.py b/evals/elsuite/track_the_stat/prompts/median.py new file mode 100644 index 0000000000..aae3c0ecc8 --- /dev/null +++ b/evals/elsuite/track_the_stat/prompts/median.py @@ -0,0 +1,33 @@ +MEDIAN_EXAMPLE = """\ +```example +input: 1 +ideal_response: [median: 1]\ + # your response; 1 is the only number shown so far +--- +input: 2 +ideal_response: [median: 1.5]\ + # even number of numbers, so median = mean(1,2) = 1.5 +--- +input: 1 +ideal_response: [median: 1]\ + # 1 is now the middle number when sorting the numbers +--- +input: 3 +ideal_response: [median: 1.5]\ + # middle numbers are now 1 and 2, so once again median = mean(1,2) = 1.5 +--- +input: 3 +ideal_response: [median: 2]\ +# the sorted list is [1 1 2 3 3]; odd length, so median is the middle number, 2 +--- +input: 0 +ideal_response: [median: 1.5]\ +# the sorted list is [0 1 1 2 3 3]; even length, so median is mean(1,2) = 1.5 +```\ +""" + + +MEDIAN_FURTHER_DETAILS = """\ +NOTE: In case of lists containing an even number of elements, you should respond with the\ + arithmetic mean of the middle two numbers of the sorted list.\ +""" diff --git a/evals/elsuite/track_the_stat/prompts/mode.py b/evals/elsuite/track_the_stat/prompts/mode.py new file mode 100644 index 0000000000..5756e7e55c --- /dev/null +++ b/evals/elsuite/track_the_stat/prompts/mode.py @@ -0,0 +1,29 @@ +MODE_EXAMPLE = """\ +```example +input: 1 +ideal_response: [mode: 1]\ + # your response; 1 is the only number shown so far +--- +input: 2 +ideal_response: [mode: 2]\ + # 1 and 2 are tied modes (both appeared once), 2 > 1 +--- +input: 1 +ideal_response: [mode: 1]\ + # 1 now has appeared more than any other number +--- +input: 3 +ideal_response: [mode: 1] +--- +input: 3 +ideal_response: [mode: 3]\ + # 3 is tied with 1 in terms of appearances, 3 > 1 +--- +input: 0 +ideal_response: [mode: 3] +```\ +""" + +MODE_FURTHER_DETAILS = """\ +NOTE: In case of ties, you should respond with the largest number that is part of the tie.\ +""" diff --git a/evals/elsuite/track_the_stat/scripts/make_plots.py b/evals/elsuite/track_the_stat/scripts/make_plots.py new file mode 100644 index 0000000000..b40e4a3586 --- /dev/null +++ b/evals/elsuite/track_the_stat/scripts/make_plots.py @@ -0,0 +1,296 @@ +from pathlib import Path +import argparse +import json + +from tqdm.auto import tqdm +import numpy as np +import matplotlib.pyplot as plt +import seaborn as sns + +from evals.utils import log_utils + + +def zero_if_none(input_num): + if input_num is None: + return 0 + else: + return input_num + + +MODELS = [ + "gpt-4-0125-preview", + "gpt-4-base", + "gpt-3.5-turbo-0125", + "gemini-pro-1.0", + "mixtral-8x7b-instruct", + "llama-2-70b-chat", + "random_baseline", + "human_baseline", +] +# separate list for OAI models for token counting, not supported in others. +OAI_MODELS = [ + "gpt-4-0125-preview", + "gpt-3.5-turbo-0125", + "gpt-4-base", +] + +STAT_TO_LABEL = { + "avg_max_length": "Average maximum sequence length achieved [no. of turns]", + "violation_rate": "Violation rate", +} + + +def make_results_dict(log_dir: Path) -> dict: + results_dict = prepare_results_dict() + results_dict = fill_results_dict(results_dict, log_dir) + return results_dict + + +def get_model(spec): + # this is hilariously ugly but it works for now (sorry) + if "gpt-4-turbo-preview" in spec["completion_fns"][0]: + return "gpt-4-0125-preview" + elif "gpt-3.5-turbo" in spec["completion_fns"][0]: + return "gpt-3.5-turbo-0125" + elif "gpt-4-base" in spec["completion_fns"][0]: + return "gpt-4-base" + elif "gemini-pro" in spec["completion_fns"][0]: + return "gemini-pro-1.0" + elif "mixtral-8x7b-instruct" in spec["completion_fns"][0]: + return "mixtral-8x7b-instruct" + elif "llama-2-70b-chat" in spec["completion_fns"][0]: + return "llama-2-70b-chat" + elif "random_baseline" in spec["completion_fns"][0]: + return "random_baseline" + elif "human" in spec["completion_fns"][0]: + return "human_baseline" + + +def get_state_tracking(spec): + if "explicit" in spec["completion_fns"][0]: + return "explicit" + else: + return "implicit" + + +def fill_results_dict(results_dict, log_dir): + print("Parsing logs...") + final_results = log_utils.get_final_results_from_dir(log_dir) + specs = log_utils.get_specs_from_dir(log_dir) + files = list(final_results.keys()) + + for file in tqdm(files): + final_result = final_results[file] + spec = specs[file] + task = spec["split"] + model = get_model(spec) + state_tracking = get_state_tracking(spec) + for stat in results_dict: + results_dict[stat][task][model][state_tracking]["raw"].append( + final_result[stat] + ) + # compute means/std_errs + for file in tqdm(files): + spec = specs[file] + task = spec["split"] + model = get_model(spec) + state_tracking = get_state_tracking(spec) + for stat in results_dict: + data_points = results_dict[stat][task][model][state_tracking]["raw"] + results_dict[stat][task][model][state_tracking]["mean"] = np.mean( + data_points + ) + results_dict[stat][task][model][state_tracking]["std_err"] = np.std( + data_points + ) / np.sqrt(len(data_points) if len(data_points) > 1 else 1) + return results_dict + + +def prepare_results_dict(): + results_dict = { + stat: { + task: { + model: { + state_tracking: {"raw": []} + for state_tracking in ["implicit", "explicit"] + } + for model in MODELS + } + for task in ["mode", "median"] + } + for stat in ["avg_max_length", "violation_rate"] + } + return results_dict + + +def make_bar_plot(results_dict: dict, task: str, stat: str, save_path: Path): + sns.set_context("paper") + sns.set_style("whitegrid") + + data = results_dict[stat][task] + + # the random baseline and human baseline aren't plotted as bars + models = MODELS[:-2] + + state_tracking_kinds = ["explicit", "implicit"] + + means = [ + [data[model][cat]["mean"] for cat in state_tracking_kinds] for model in models + ] + std_errs = [ + [data[model][cat]["std_err"] for cat in state_tracking_kinds] + for model in models + ] + cmap = plt.get_cmap("Paired") + colors = np.array([cmap(i) for i in range(len(state_tracking_kinds))]) + + # Plotting + x = np.arange(len(models)) # the label locations + + width = 0.4 + + fig, ax = plt.subplots(1, 1, figsize=(8, 6), dpi=300) + + explicit_bars = ax.barh( + x + width / 2, + [mean[0] for mean in means], + width, + xerr=[err[0] for err in std_errs], + label="Explicitly tracked state baseline", + color=colors[0], + ) + implicit_bars = ax.barh( + x - width / 2, + [mean[1] for mean in means], + width, + xerr=[err[1] for err in std_errs], + label="Implicitly tracked state", + color=colors[1], + ) + + ax.set_xlabel(STAT_TO_LABEL[stat]) + # maximum x + xerr value times 1.2 + x_max = ( + max([m for mean in means for m in mean]) + + max([e for err in std_errs for e in err]) + ) * 1.2 + ax.set_xlim([0, x_max]) + ax.set_yticks(x) + ax.set_yticklabels(models) + + ax.bar_label(implicit_bars, padding=3, fmt="%.2f") + ax.bar_label(explicit_bars, padding=3, fmt="%.2f") + + # plot random and human baselines + random_baseline = data["random_baseline"]["implicit"]["mean"] + random_err = data["random_baseline"]["implicit"]["std_err"] + ax.axvline(random_baseline, color="red", linestyle="--", label="Random baseline") + ax.axvspan( + random_baseline - random_err, + random_baseline + random_err, + color="red", + alpha=0.05, + ) + + human_baseline = data["human_baseline"]["implicit"]["mean"] + human_err = data["human_baseline"]["implicit"]["std_err"] + ax.axvline( + human_baseline, + color="#366a9d", + linestyle=":", + label="Human baseline (implicit)", + ) + + ax.axvspan( + human_baseline - human_err, + human_baseline + human_err, + color="#366a9d", + alpha=0.05, + ) + + # get rid of horizontal grid lines + ax.grid(axis="y", which="both") + + ax.legend() + + fig.tight_layout() + + plt.savefig(save_path, bbox_inches="tight", dpi=300) + + +def count_tokens(log_dir) -> dict[str, dict[str, dict[str, int]]]: + """ + model -> task -> input, output, total tokens + """ + token_counts = { + model: { + task: { + state_tracking: {kind: 0 for kind in ["input", "output", "total"]} + for state_tracking in ["implicit", "explicit"] + } + for task in ["mode", "median"] + } + for model in OAI_MODELS + } + globbed_logs = list(log_dir.glob("*.log")) + already_examined = set() + for log in tqdm(globbed_logs, total=len(globbed_logs), desc="Counting tokens"): + spec = log_utils.extract_spec(log) + task = spec["split"] + model = get_model(spec) + state_tracking = get_state_tracking(spec) + + if model not in OAI_MODELS: + continue + + # dont care about repeats, this is a rough estimate anyway + if (model, task, state_tracking) in already_examined: + continue + already_examined.add((model, task, state_tracking)) + + samplings = log_utils.extract_individual_results(log, "sampling") + for sampling in samplings: + usage = sampling["usage"] + token_counts[model][task][state_tracking]["input"] += zero_if_none( + usage["prompt_tokens"] + ) + token_counts[model][task][state_tracking]["output"] += zero_if_none( + usage["completion_tokens"] + ) + token_counts[model][task][state_tracking]["total"] += zero_if_none( + usage["total_tokens"] + ) + return token_counts + + +def main(args: argparse.Namespace): + log_dir = Path(args.log_dir) + save_dir = Path(args.save_dir) + save_dir.mkdir(exist_ok=True, parents=True) + + results_dict = make_results_dict(log_dir) + + for stat in tqdm(results_dict.keys(), desc=f"Plotting..."): + for task in tqdm(["mode", "median"], desc=f"Plotting {stat}"): + save_path = save_dir / f"{task}_{stat}.png" + make_bar_plot(results_dict, task, stat, save_path) + save_path = save_dir / f"{stat}.json" + with open(save_path, "w") as f: + json.dump(results_dict[stat], f, indent=2) + + token_counts = count_tokens(log_dir) + save_path = save_dir / "token_counts.json" + with open(save_path, "w") as f: + json.dump(token_counts, f, indent=2) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--log_dir", type=str, required=True, help="Where the logs are stored" + ) + parser.add_argument( + "--save_dir", type=str, required=True, help="Where to save the plots" + ) + args = parser.parse_args() + main(args) diff --git a/evals/elsuite/track_the_stat/scripts/run_experiments.sh b/evals/elsuite/track_the_stat/scripts/run_experiments.sh new file mode 100644 index 0000000000..8307866418 --- /dev/null +++ b/evals/elsuite/track_the_stat/scripts/run_experiments.sh @@ -0,0 +1,92 @@ +#!/bin/bash + +usage() { + echo "Usage: $0 -l logdir" + echo " -l logdir Specify the directory for log files" + exit 1 +} + +# Check if no arguments were provided +if [ $# -eq 0 ]; then + usage + exit 1 +fi + +# Parse command-line options +while getopts 's:l:' flag; do + case "${flag}" in + l) logdir=${OPTARG} ;; + *) usage ;; + esac +done + +# Check if mandatory arguments were provided +if [ -z "$logdir" ]; then + usage + exit 1 +fi + +NUM_REPEATS=3 + +export EVALS_THREADS=10 +export EVALS_THREADS_TIMEOUT=5 + +declare -a SOLVERS=( + # 4-turbo-preview + "generation/direct/gpt-4-turbo-preview" + "track_the_stat/explicit_state/gpt-4-turbo-preview" + # 3.5-turbo + "generation/direct/gpt-3.5-turbo" + "track_the_stat/explicit_state/gpt-3.5-turbo" + # 4-base + "generation/hhh/gpt-4-base" + "track_the_stat/explicit_state/hhh/gpt-4-base" + # gemini pro + "generation/direct/gemini-pro" + "track_the_stat/explicit_state/gemini-pro" + # mixtral-8x7b-instruct + "generation/direct/mixtral-8x7b-instruct" + "track_the_stat/explicit_state/mixtral-8x7b-instruct" + # llama chat 70b + "generation/direct/llama-2-70b-chat" + "track_the_stat/explicit_state/llama-2-70b-chat" + # random baseline + "track_the_stat/random_baseline" +) +declare -a TASKS=( + "mode" + "median" +) + +# Check if GEMINI_API_KEY is set +if [ -z "$GEMINI_API_KEY" ]; then + echo "Enter your Gemini API Key:" + read -s GEMINI_API_KEY + export GEMINI_API_KEY +fi + +# Check if TOGETHER_API_KEY is set +if [ -z "$TOGETHER_API_KEY" ]; then + echo "Enter your Together API Key:" + read -s TOGETHER_API_KEY + export TOGETHER_API_KEY +fi + +start_time=$SECONDS +for ((i = 1; i <= NUM_REPEATS; i++)); do + for task in "${TASKS[@]}"; do + for solver in "${SOLVERS[@]}"; do + if [[ $solver == *"gemini"* ]]; then + export EVALS_SEQUENTIAL=1 + else + export EVALS_SEQUENTIAL=0 + fi + solver_dotted=${solver//\//.} + record_path="${logdir}/${solver_dotted}_${task}_${i}" + echo "Running $solver on $task (repeat $i)" + oaieval $solver "track_the_stat.${task}" \ + --record_path "$record_path.log" --seed $i + done + done +done +echo "Total time: $((SECONDS - start_time)) seconds" diff --git a/evals/elsuite/track_the_stat/solvers.py b/evals/elsuite/track_the_stat/solvers.py new file mode 100644 index 0000000000..65721002cc --- /dev/null +++ b/evals/elsuite/track_the_stat/solvers.py @@ -0,0 +1,98 @@ +import random +from typing import Any + +from evals.elsuite.track_the_stat import utils +from evals.solvers.solver import NestedSolver, Solver, SolverResult, SolverSpec +from evals.task_state import Message, TaskState + + +class ExplicitStateSolver(NestedSolver): + def __init__( + self, + underlying_solver: SolverSpec, + state_role: str = "assistant", + *args, + **kwargs, + ): + super().__init__(underlying_solver=underlying_solver, *args, **kwargs) + self.state_role = state_role + + @property + def underlying_solver(self) -> Solver: + return self.get_solver("underlying_solver") + + def _render_state(self, current_state: dict) -> str: + rendered_state_string = f"{current_state['state_label']}\n{current_state['state_data']}" + return rendered_state_string + + def _build_message(self, task_state: TaskState) -> str: + message_string = "The current state, useful for solving the task\n" + self._render_state( + task_state.current_state + ) + return Message(role=self.state_role, content=message_string) + + def _solve(self, task_state: TaskState) -> SolverResult: + precomputed_state_message = self._build_message(task_state) + task_state.messages.append(precomputed_state_message) + + solver_result = self.underlying_solver(task_state=task_state) + return solver_result + + +class RandomBaselineSolver(Solver): + def __init__(self, registry: Any = None, *args, **kwargs): + super().__init__() + + def _solve(self, task_state: TaskState) -> SolverResult: + task = task_state.current_state["task_name"] + random_output = self._task_solve(task, task_state) + solver_result = SolverResult(output=f"[{task}: {random_output}]") + return solver_result + + def _task_solve(self, task: str, task_state: TaskState) -> str: + if task == "mode": + return self._mode_solve(task_state) + elif task == "median": + return self._median_solve(task_state) + + def _mode_solve(self, task_state: TaskState) -> str: + """ + Picks a random number from the numbers seen so far + """ + numbers = list(task_state.current_state["state_data"].keys()) + random_mode = random.choice(numbers) + return str(random_mode) + + def _median_solve(self, task_state: TaskState) -> str: + """ + Picks a random number from the numbers seen so far + (in case of even number of numbers, picks the average of two random numbers) + """ + numbers = task_state.current_state["state_data"] + if len(numbers) % 2 == 0: + random_1, random_2 = random.choices(numbers, k=2) + random_median = (random_1 + random_2) / 2 + else: + random_median = random.choice(numbers) + return str(round(random_median, 1)) + + +class TrackTheStatHuman(NestedSolver): + def __init__(self, human_cli_solver: SolverSpec, *args, **kwargs): + super().__init__(human_cli_solver=human_cli_solver, *args, **kwargs) + + @property + def human_cli_solver(self) -> Solver: + return self.get_solver("human_cli_solver") + + def _solve(self, task_state: TaskState) -> SolverResult: + human_result = self.human_cli_solver(task_state=task_state) + task = task_state.current_state["task_name"] + # wrap the result in [: ] if not already wrapped + output = utils.parse_solver_output(human_result.output, task) + if output is None: # there is a violation -- output is not wrapped + return SolverResult( + output=f"[{task}: {human_result.output}]", + ) + else: # no violation -- output is already wrapped + return human_result diff --git a/evals/elsuite/track_the_stat/utils.py b/evals/elsuite/track_the_stat/utils.py new file mode 100644 index 0000000000..55467c5100 --- /dev/null +++ b/evals/elsuite/track_the_stat/utils.py @@ -0,0 +1,78 @@ +import re +from collections import Counter +from typing import Union + +import numpy as np + + +def yellow_string(str: str) -> str: + return f"\033[1;33m{str}\033[0m" + + +def median(numbers: list[int]) -> int: + """ + Returns the median of the given list of numbers. If the list has an even + number of elements, the arithmetic mean of the two middle elements of the + sorted list is returned. + """ + return np.median(numbers) + + +def mode(numbers: list[int]) -> int: + """ + Returns the mode of the given list of numbers. If there are multiple modes, + the largest mode is returned. + """ + frequency = {} + for number in numbers: + frequency[number] = frequency.get(number, 0) + 1 + + max_frequency = max(frequency.values()) + candidates = [number for number, freq in frequency.items() if freq == max_frequency] + + return max(candidates) + + +task_to_fn = {"median": median, "mode": mode} + + +def parse_solver_output(solver_output: str, task: str) -> Union[int, None]: + solver_string = solver_output.strip().lower() + pattern = rf"\[{task}: (\d+(?:\.\d+)?)\]" + + match = re.search(pattern, solver_string) + + if match: + try: + output = float(match.group(1)) + except ValueError: + output = None + else: + output = None + + return output + + +def compute_mode_state(curr_list: list[int]) -> dict: + counter = Counter(curr_list) + return dict(counter) + + +def compute_median_state(curr_list: list[int]) -> dict: + sorted_list = sorted(curr_list) + return sorted_list + + +def compute_state(curr_list: list[int], task) -> dict: + if task == "mode": + return { + "task_name": task, + "state_label": "number to count", + "state_data": compute_mode_state(curr_list), + } + else: + return { + "task_name": task, + "state_label": "sorted list of shown numbers", + "state_data": compute_median_state(curr_list), + } diff --git a/evals/elsuite/twenty_questions/eval.py b/evals/elsuite/twenty_questions/eval.py new file mode 100644 index 0000000000..3cb0d5c857 --- /dev/null +++ b/evals/elsuite/twenty_questions/eval.py @@ -0,0 +1,204 @@ +import logging +import random +import re +from typing import Any, Dict, List, Optional, Union + +import evals +import evals.metrics +from evals.api import CompletionFn +from evals.elsuite.twenty_questions.utils import PROMPTS, generate_task_state_for +from evals.eval import SolverEval +from evals.record import Recorder +from evals.registry import registry +from evals.solvers.human_cli_solver import HumanCliSolver +from evals.solvers.solver import Solver +from evals.solvers.utils import maybe_wrap_with_solver +from evals.task_state import Message + +logger = logging.getLogger(__name__) +WORD_PATTERN = r"\[GUESS (.*?)\]" + + +class TwentyQuestions(SolverEval): + def __init__( + self, + completion_fns: List[CompletionFn], + samples_jsonl: str, + gamemaster_spec: str, + max_questions: int = 20, + max_replies: int = 40, + num_shortlist_items: int = 20, + shortlist_variant: bool = False, + seed: int = 222024, + n_samples: Optional[int] = None, + *args, + **kwargs, + ): + super().__init__(completion_fns, seed=seed, *args, **kwargs) + + self.samples_jsonl = samples_jsonl + self.gamemaster_solver = maybe_wrap_with_solver( + registry.make_completion_fn(gamemaster_spec) + ) + self.max_questions = max_questions + + if max_replies < max_questions: + logger.warn( + f"max_replies ({max_replies}) is less than max_questions ({max_questions}). Setting max_replies to {max_questions + 20}" + ) + self.max_replies = max_replies if max_replies > max_questions else max_questions + 20 + self.num_shortlist_items = num_shortlist_items + self.shortlist_variant = shortlist_variant + + self.n_samples = n_samples + self.rng = random.Random(seed) + + def eval_sample(self, solver: Solver, sample: Dict, rng: random.Random) -> Dict[str, Any]: + assert "word" in sample, "Sample must contain 'word' field" + assert "difficulty" in sample, "Sample must contain 'difficulty' field" + + if not isinstance(solver, HumanCliSolver): + logging.info(f"Running sample: {sample['word']}") + + # Generate the shortlist for the current sample if applicable. + if self.shortlist_variant: + assert self.num_shortlist_items <= len( + self.shortlist + ), "Number of shortlist items must be less than or equal to the total number of samples." + shortlist_for_sample = rng.sample(self.shortlist, self.num_shortlist_items) + if sample["word"] not in shortlist_for_sample: + random_index = rng.randint(0, len(shortlist_for_sample) - 1) + shortlist_for_sample[random_index] = sample["word"] + else: + shortlist_for_sample = None + response = self._conversation_loop(solver, sample, shortlist_for_sample) + + return response + + def run(self, recorder: Recorder) -> Dict[str, Union[float, int]]: + samples = self.get_samples() + self.rng.shuffle(samples) + samples = samples[: self.n_samples] if self.n_samples else samples + + if self.shortlist_variant: + self.shortlist = [sample["word"] for sample in samples] + + self.eval_all_samples(recorder, samples) + events = recorder.get_events("match") + + scores = [event.data["score"] for event in events] + num_guesses = [event.data["num_guesses"] for event in events] + num_questions = [event.data["num_questions"] for event in events] + num_violations = [event.data["num_violations"] for event in events] + num_gamemaster_refusals = [event.data["num_gamemaster_refusals"] for event in events] + incorrect_guesses = [event.data["incorrect_guesses"] for event in events] + word_difficulties = [event.data["word_difficulty"] for event in events] + + return { + "score": sum(scores) / len(scores), + "accuracy": evals.metrics.get_accuracy(events), + "bootstrap_std": evals.metrics.get_bootstrap_accuracy_std(events), + "average_num_guesses": sum(num_guesses) / len(num_guesses), + "average_num_questions": sum(num_questions) / len(num_questions), + "average_num_violations": sum(num_violations) / len(num_violations), + "average_num_gamemaster_refusals": sum(num_gamemaster_refusals) + / len(num_gamemaster_refusals), + "average_num_incorrect_guesses": sum((len(ig) for ig in incorrect_guesses)) + / len(incorrect_guesses), + "average_word_difficulty": sum(word_difficulties) / len(word_difficulties), + } + + def _conversation_loop( + self, solver: Solver, sample: Dict, shortlist: Optional[List[str]] = None + ) -> Dict[str, Any]: + """Maintains a conversation between the guesser and the gamemaster until the maximum number of questions is reached, or until a correct guess is made. + + Args: + solver (Solver): any compatible solver, instantiated for the current sample. + sample (Dict): current sample – one word to guess, and its associated difficulty. + + Returns: + Dict[str, Any]: a dictionary containing the final result and metrics of the conversation. + """ + + metrics = { + "num_guesses": 0, + "num_questions": 0, + "num_violations": 0, + "num_guesser_replies": 0, # num_guesses + num_questions + num_violations + "num_gamemaster_refusals": 0, + "incorrect_guesses": [], + } + conversation = [] + + # Contains fall-back condition to avoid infinite loops for solvers which never output questions. + while ( + metrics["num_questions"] < self.max_questions + and metrics["num_guesser_replies"] < self.max_replies + ): + task_state = generate_task_state_for( + "guesser", conversation, max_questions=self.max_questions, shortlist=shortlist + ) + guesser_response = solver(task_state) + conversation += [Message(content=guesser_response.output, role="guesser")] + metrics["num_guesser_replies"] += 1 + + # Check if guess made: + match = re.search(WORD_PATTERN, guesser_response.output) + if match is not None: + metrics["num_guesses"] += 1 + guess = match.group(1) + if guess.lower() == sample["word"].lower(): + response = { + "correct": True, + "score": self.max_questions - metrics["num_questions"], + "expected": sample["word"], + "word_difficulty": sample["difficulty"], + "picked": guess, + "num_guesses": metrics["num_guesses"], + "num_questions": metrics["num_questions"], + "num_violations": metrics["num_violations"], + "num_gamemaster_refusals": metrics["num_gamemaster_refusals"], + "incorrect_guesses": metrics["incorrect_guesses"], + } + evals.record.record_match(**response) + return response + else: + metrics["incorrect_guesses"] += [guess] + conversation += [ + Message( + content=PROMPTS["incorrect_guess"].format(guess=guess), role="system" + ) + ] + continue + elif "?" in guesser_response.output.strip(): + metrics["num_questions"] += 1 + else: # Neither guess nor question. + # TODO: Maybe make the guesser retry here? + logger.warn( + f"Rule violation, no guess or question in output: {guesser_response.output}" + ) + metrics["num_violations"] += 1 + conversation += [Message(content=PROMPTS["rule_violation"], role="system")] + continue + + task_state = generate_task_state_for("gamemaster", conversation, sample["word"]) + gamemaster_response = self.gamemaster_solver(task_state) + conversation += [Message(content=gamemaster_response.output, role="gamemaster")] + if gamemaster_response.output.lower() == "skip": + metrics["num_gamemaster_refusals"] += 1 + + logger.info(f"Ran out of questions for word: {sample['word']}") + response = { + "correct": False, + "score": 0, + "expected": sample["word"], + "word_difficulty": sample["difficulty"], + "num_guesses": metrics["num_guesses"], + "num_questions": metrics["num_questions"], + "num_violations": metrics["num_violations"], + "num_gamemaster_refusals": metrics["num_gamemaster_refusals"], + "incorrect_guesses": metrics["incorrect_guesses"], + } + evals.record.record_match(**response) + return response diff --git a/evals/elsuite/twenty_questions/readme.md b/evals/elsuite/twenty_questions/readme.md new file mode 100644 index 0000000000..89f2e3ce6c --- /dev/null +++ b/evals/elsuite/twenty_questions/readme.md @@ -0,0 +1,82 @@ +# 20 Questions + +This eval tests models' ability to generate and iterate over hypotheses by playing the game of "20 questions". In 20 questions, one of the players – the "gamemaster" – thinks of a word (in our case a noun) and the other player needs to guess it. To help them guess, the player can ask up to 20 yes-or-no questions, which the gamemaster must answer. + +## Usage +Run with: +```bash +# Standard variant. +oaieval twenty_questions.full + +# Shortlist variant. +oaieval twenty_questions.shortlist.full +``` + +Where the solver can be any generation solver in `evals/registry/solvers/defaults.yaml`, e.g. `generation/cot/gpt-3.5-turbo-16k`, or the chain-of-thought solvers in `evals/registry/solvers/twenty_questions.yaml`. + +## Evaluation process +We run a dialogue loop between two models for each sample: the evaluated model and the "gamemaster". By default, the gamemaster is gpt-4-turbo-preview – but this can be updated by specifying a different solver in `evals/registry/evals/twenty_questions.yaml`. + +The dialogue continues until the word is guessed correctly, or until 20 questions have been asked, whichever comes first. We also terminate conversations that last longer than 40 replies, to ensure that models which do not ask questions don't have infinite conversations. Both the maximum questions and the maximum replies can be controlled from the eval YAML file. + +## Task State +The task state can be found in `twenty_questions/utils.py`; it reads: +``` +You are about to play the game '20 questions'. The other player has thought of a noun which you should try to guess. You can ask 20 yes/no questions, to which they will answer 'yes', 'no', or 'skip' (if they cannot answer your question). You should try to guess the word as soon as possible, using the least amount of questions. To guess a word, type [GUESS ] – for example to guess the word 'sibling', output [GUESS sibling]. Your score will be 0 if you do not guess correctly, and {max_questions} minus the number of questions you asked if you guess correctly. Start by asking your first question. +``` + +## Prompts +See `twenty_questions/utils.py` to review/adjust the prompts used in this eval. + +## Datasets + +We use a dataset of 207 words, 177 of which were from [this lexicon](https://github.com/mounicam/lexical_simplification), annotated by our team with a difficulty category. This dataset comprises: +- 47 words rated “easy”, e.g. ear, father, potato; +- 91 words rated “medium”, e.g. cloth, hike, discount; +- 69 words rated “hard”, e.g. prosperity, gland, philosopher; + +In addition to these common nouns, we include 30 proper nouns such as “Sherlock Holmes,” “The Beatles,” “Titanic,” and “Starbucks”, which span the easy and medium difficulties. + +## Metrics +We measure the score each model achieves, defined as `score = max_questions - questions_asked`. We also track the win-rate, i.e. the % of samples the model guesses correctly. Auxiliary metrics such as average number of average number of questions asked, average number of incorrect guesses, and average number of gamemaster refusals (i.e. situations where the gamemaster says 'skip') are also tracked. + + +## Variants + +We run two main variants of this evaluation: +- **standard**: the main variant +- **shortlist**: an easier variant where the evaluated model sees a shortlist of words in its system prompt. The word the gamemaster has selected is part of the list. In this variant, the evaluated model effectively has to narrow down the pool of candidate words until it finds the answer. + +## Token Usage Estimates + +Below is a rough estimate of the total number of tokens consumed by some variations the eval, including both input and output tokens: + +Variant | Model | Solver | Prompt tokens | Completion tokens | Total tokens +| --- | --- | --- | --- | --- | --- | +standard | direct | gpt-4-turbo-preview | 2,502,067 | 52,879 | 2,554,946 +standard | direct | gpt-4-base | 13,197,212 | 2,814,623 | 16,011,835 +standard | direct | gpt-3.5-turbo | 2,670,866 | 57,917 | 2,728,783 +standard | cot | gpt-4-turbo-preview | 73,765,861 | 1,881,455 | 75,647,316 +standard | cot | gpt-4-base | 51,777,817 | 6,397,472 | 58,175,289 +standard | cot | gpt-3.5-turbo | 38,236,500 | 199,831 | 38,436,331 +standard | cot | llama-2-70b | 6,785,634 | 581,421 | 7,367,055 +standard | cot | mixtral-8x7b-instruct | 175,956,903 | 5,327,393 | 181,284,296 +shortlist | direct | gpt-4-turbo-preview | 1,237,172 | 28,351 | 1,265,523 +shortlist | direct | gpt-4-base | 11,034,903 | 2,133,487 | 13,168,390 +shortlist | direct | gpt-3.5-turbo | 1,704,154 | 36,356 | 1,740,510 +shortlist | cot | gpt-4-turbo-preview | 10,951,215 | 545,945 | 11,497,160 +shortlist | cot | gpt-4-base | 45,591,363 | 596,429 | 46,187,792 +shortlist | cot | gpt-3.5-turbo | 19,798,263 | 165,731 | 19,963,994 +shortlist | cot | llama-2-70b | 5,980,667 | 528,879 | 6,509,546 +shortlist | cot | mixtral-8x7b-instruct | 143,646,924 | 4,315,806 | 147,962,730 + + +## Version History +v0: Initial version released + + +## Contribution statement + +Eval design, implementation, and results evaluation were primarily conducted by Andrei Alexandru with contributions from Dane Sherburn, under the guidance of (alphabetically by last-name) Steven Adler, James Aung, and Chan Jun Shern who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation. + + diff --git a/evals/elsuite/twenty_questions/scripts/make_plots.py b/evals/elsuite/twenty_questions/scripts/make_plots.py new file mode 100644 index 0000000000..f07b76da5a --- /dev/null +++ b/evals/elsuite/twenty_questions/scripts/make_plots.py @@ -0,0 +1,142 @@ +import argparse +from pathlib import Path +import os + +import matplotlib.pyplot as plt +import pandas as pd +import seaborn as sns + +from evals.utils import log_utils + +PLOT_TITLES_BY_METRIC = { + "score": "Score", + "winrate": "Win-rate", + "bootstrap_std": "Bootstrapped standard deviation", + "average_num_guesses": "Average guesses per sample", + "average_num_questions": "Average questions per sample", + "average_num_violations": "Average rule violations per sample", + "average_num_gamemaster_refusals": "Average gamemaster refusals per sample", + "average_num_incorrect_guesses": "Average incorrect guesses per sample", + "average_word_difficulty": "Average word difficulty", +} + +HUMAN_BASELINE = { + "standard": { + "winrate": 0.0333, + "score": 0.1333, + "average_num_guesses": 0.3666, + "average_num_questions": 19.8666, + "average_num_violations": 0.62, + "average_num_gamemaster_refusals": 0.28, + "average_num_incorrect_guesses": 0.3333, + "average_word_difficulty": 2.2333, + }, + "shortlist": { + "winrate": 1, + "score": 14.1388, + "average_num_guesses": 1.8611, + "average_num_questions": 5.8611, + "average_num_violations": 0.1944, + "average_num_gamemaster_refusals": 0.1111, + "average_num_incorrect_guesses": 0.8611, + "average_word_difficulty": 2.2777, + } +} + +UNIT_METRICS = ["winrate"] + +def extract_metrics(datadir: Path) -> pd.DataFrame: + df_rows = [] + # There are two eval variants: standard and shortlist. + for variant in os.listdir(datadir): + for path, results in sorted(list(log_utils.get_final_results_from_dir(f"{datadir}/{variant}").items())): + spec = log_utils.extract_spec(path) + solver_path = Path(spec["completion_fns"][0]) + model = solver_path.name + solver = solver_path.parent.name + # Remove root section of path, which is the eval name + solver_path = solver_path.relative_to(solver_path.parts[0]) + df_rows.append({"solver": solver, "model": model, "variant": variant, **results}) + df = pd.DataFrame(df_rows) + df.rename(columns={"accuracy": "winrate"}, inplace=True) + df.sort_values(by=["variant", "model", "solver"], inplace=True) + df.to_csv(datadir / "results.csv", index=False) + + return df + +def make_plot(df: pd.DataFrame, outpath: Path, metric="score", variant="standard"): + df = df.round(2) + plt.figure() + sns.set_theme(style="whitegrid") + + def compute_sem(x): + sem = x.std() / (len(x) ** 0.5) + sem2 = sem * 2 # 95% confidence interval + lower = max(0, (x.mean() - sem2).round(2)) + upper = (x.mean() + sem2).round(2) + return lower, upper + + + # Plotting + sns.set(style="whitegrid") + ax = sns.barplot(x=metric, y="model", hue="solver", data=df, errorbar=compute_sem, capsize=0.1) + for container in ax.containers: + ax.bar_label(container, fmt="{:.2f}", label_type="edge", padding=15) + + ax.axvline(HUMAN_BASELINE[variant][metric], color="red", linestyle="--") + + # A bunch of tweaks to make individual plots look nice. + if variant == "shortlist" and metric == "winrate": + plt.text(HUMAN_BASELINE[variant][metric] - 0.35, .5, "Human baseline", color="red", fontsize=12, ha="left") + elif variant == "standard" and metric == "average_num_questions": + plt.text(HUMAN_BASELINE[variant][metric] - 7, .5, "Human baseline", color="red", fontsize=12, ha="left") + else: + plt.text(HUMAN_BASELINE[variant][metric] + 0.05, .5, "Human baseline", color="red", fontsize=12, ha="left") + + # Some of the metrics are in [0, 1]. + if metric in UNIT_METRICS: + plt.xlim(0, 1.1) + + if metric in ("score", "average_num_questions"): + plt.xlim(0, 20.1) + + if metric == "average_word_difficulty": + plt.xlim(0, 3.1) # 6 is the maximum word difficulty in the dataset. + + if metric in ("score", "winrate"): + plt.legend(loc="lower right") + + plt.title(PLOT_TITLES_BY_METRIC[metric] + f" ({variant} variant)") + plt.xlabel(metric) + plt.tight_layout() + plt.savefig(outpath) + plt.close() + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--log-dir", "-d", type=str, required=True) + parser.add_argument("--out-dir", "-o", type=str, default="./outputs") + args = parser.parse_args() + log_dir = Path(args.log_dir) + out_dir = Path(args.out_dir) + + out_dir.mkdir(exist_ok=True, parents=True) + + df = extract_metrics(log_dir) + + # Rename some of the solver values so they can be represented in the same plot. + df.loc[df['solver'] == 'cot_hhh', 'solver'] = 'cot' + df.loc[df['solver'] == 'hhh', 'solver'] = 'direct' + + for variant in df['variant'].unique(): + df_per_variant = df[df['variant'] == variant] + + print(f"Plotting all metrics for {variant} variant...") + + core_metrics = ["score", "winrate"] + auxiliary_metrics = ["average_num_guesses", "average_num_questions", "average_num_violations", "average_num_gamemaster_refusals", "average_num_incorrect_guesses", "average_word_difficulty"] + for metric in core_metrics + auxiliary_metrics: + make_plot(df_per_variant[["model", "solver", metric]].copy(), + out_dir / f"{variant}_{metric}.png", + metric, + variant) \ No newline at end of file diff --git a/evals/elsuite/twenty_questions/scripts/run_experiments.sh b/evals/elsuite/twenty_questions/scripts/run_experiments.sh new file mode 100644 index 0000000000..4b8718d607 --- /dev/null +++ b/evals/elsuite/twenty_questions/scripts/run_experiments.sh @@ -0,0 +1,60 @@ +logdir=./logs +outputdir=./outputs + +timestamp=$(date +%Y%m%d_%H%M%S) +logpathbase=$logdir/$timestamp + +num_repeats=1 + +# Check for --num_repeats argument +for arg in "$@" +do + if [[ $arg == --num_repeats=* ]]; then + num_repeats="${arg#*=}" + fi +done + +echo Num repeats is: $num_repeats +echo Running experiments and logging to $logpathbase + +declare -a SOLVERS=( + # Solvers for gpt-3.5-turbo + "generation/direct/gpt-3.5-turbo" + "twenty_questions/cot/gpt-3.5-turbo" + + # # Solvers for gpt-4-turbo-preview + "generation/direct/gpt-4-turbo-preview" + "twenty_questions/cot/gpt-4-turbo-preview" + + # # Solvers for gpt-4-base + "generation/hhh/gpt-4-base" + "twenty_questions/cot_hhh/gpt-4-base" +) + +if [ ! -d "$logpathbase/standard" ]; then + mkdir -p "$logpathbase/standard" +fi + +if [ ! -d "$logpathbase/standard" ]; then + mkdir -p "$logpathbase/shortlist" +fi + + for solver in "${SOLVERS[@]}" + do + for ((i=1;i<=num_repeats;i++)) + do + echo "Running $solver, iteration $i, standard variant." + oaieval $solver twenty_questions.full --record_path "$logpathbase/standard/$solver-$i.log" + + echo "Running $solver, iteration $i, shortlist variant." + oaieval $solver twenty_questions.shortlist.full --record_path "$logpathbase/shortlist/$solver-$i.log" + done + done +done + +echo Done running experiments, all logs in $logpathbase + +echo Producing plots, outputs to $outputdir + +# Produce results +python scripts/make_plots.py --log-dir $logpathbase --out-dir $outputdir \ No newline at end of file diff --git a/evals/elsuite/twenty_questions/test_utils.py b/evals/elsuite/twenty_questions/test_utils.py new file mode 100644 index 0000000000..069b86c8ad --- /dev/null +++ b/evals/elsuite/twenty_questions/test_utils.py @@ -0,0 +1,27 @@ +from utils import format_msg, format_msgs +from evals.task_state import Message + +def test_format_msg(): + msg = Message(content="I'm a message", role="guesser") + + assert format_msg(msg, "guesser") == Message(content="I'm a message", role="assistant") + assert format_msg(msg, "gamemaster") == Message(content="I'm a message", role="user") + +def test_format_msgs(): + msgs = [ + Message(content="I'm a guesser message", role="guesser"), + Message(content="I'm a gamemaster message", role="gamemaster"), + Message(content="I'm another guesser message", role="guesser"), + ] + + assert format_msgs(msgs, "guesser") == [ + Message(content="I'm a guesser message", role="assistant"), + Message(content="I'm a gamemaster message", role="user"), + Message(content="I'm another guesser message", role="assistant"), + ] + + assert format_msgs(msgs, "gamemaster") == [ + Message(content="I'm a guesser message", role="user"), + Message(content="I'm a gamemaster message", role="assistant"), + Message(content="I'm another guesser message", role="user"), + ] \ No newline at end of file diff --git a/evals/elsuite/twenty_questions/utils.py b/evals/elsuite/twenty_questions/utils.py new file mode 100644 index 0000000000..169dd0a977 --- /dev/null +++ b/evals/elsuite/twenty_questions/utils.py @@ -0,0 +1,69 @@ +from typing import Literal, List, Optional +from evals.task_state import Message, TaskState + +PROMPTS = { + "guesser": "You are about to play the game '20 questions'. The other player has thought of a noun which you should try to guess. You can ask 20 yes/no questions, to which they will answer 'yes', 'no', or 'skip' (if they cannot answer your question). You should try to guess the word as soon as possible, using the least amount of questions. To guess a word, type [GUESS ] – for example to guess the word 'sibling', output [GUESS sibling]. Your score will be 0 if you do not guess correctly, and {max_questions} minus the number of questions you asked if you guess correctly. Start by asking your first question.", + "guesser_shortlist": "You are about to play the game '20 questions'. The other player has thought of a noun which you should try to guess. The noun is in this list: {shortlist}.\nYou can ask 20 yes/no questions, to which they will answer 'yes', 'no', or 'skip' (if they cannot answer your question). You should try to guess the word as soon as possible, using the least amount of questions. To guess a word, type [GUESS ] – for example to guess the word 'sibling', output [GUESS sibling]. Your score will be 0 if you do not guess correctly, and {max_questions} minus the number of questions you asked if you guess correctly. Start by asking your first question.", + "gamemaster": "You are about to play the game '20 questions'. You'll be given a noun, which the other player will try to guess. They will ask 20 yes/no questions to try to guess the word. You will answer their questions with 'yes', 'no', or 'skip'. The word is: {word}.", + "incorrect_guess": "Guess {guess} is incorrect. Ask more questions, or make another guess!", + "rule_violation": "Your output was neither a guess nor a question. Try again! You can ask a yes/no question, or make a guess by outputting [GUESS ]." +} + +def generate_task_state_for(role: Literal["guesser", "gamemaster"], conversation: list[Message], word: Optional[str] = None, max_questions: int = 20, shortlist: Optional[List[str]] = None) -> TaskState: + """Generates a TaskState for the given role and conversation.""" + if role == "guesser": + prompt = PROMPTS["guesser"].format(max_questions=max_questions) if shortlist is None else PROMPTS["guesser_shortlist"].format(max_questions=max_questions, shortlist=shortlist) + elif role == "gamemaster": + prompt = PROMPTS[role].format(word=word) + else: + raise ValueError(f"Invalid role: {role}") + + formatted_conversation = format_msgs(conversation, role) + + return TaskState( + task_description=prompt, + messages=formatted_conversation, + ) + + +def format_msgs( + messages: list[Message], + role: Literal["guesser", "gamemaster"], +) -> list[Message]: + """Format messages from the perspective of the `role`.""" + new_messages = [format_msg(msg, role) for msg in messages] + + # post-conditions + for m in new_messages: + assert m.role in ["user", "assistant", "system"] + + return new_messages + +def format_msg(msg: Message, role: Literal["guesser", "gamemaster"]) -> Message: + """Formats a single message from the perspective of the `role`.""" + + # body + is_others_msg = role not in msg.role + new_content = msg.content + + if is_others_msg: + new_role = "user" + elif is_system_msg(msg): + new_role = "system" + else: + new_role = "assistant" + + new_message = Message(content=new_content, role=new_role) + + # post-conditions + assert isinstance(new_message.content, str) + assert new_message.role in ["user", "assistant", "system"] + + return new_message + +def is_system_msg(m: Message) -> bool: + assert isinstance(m, Message), "Message must be a Message type." + assert hasattr(m, "role"), "Message must have a role." + assert isinstance(m.role, str), "Message role must be a string." + + return m.role == "system" \ No newline at end of file diff --git a/evals/registry/data/already_said_that/500_100.jsonl b/evals/registry/data/already_said_that/500_100.jsonl new file mode 100644 index 0000000000..1804855f51 --- /dev/null +++ b/evals/registry/data/already_said_that/500_100.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ce6eeadef5aa33a57abcd0b42bbdb02d1a4eb3c61a35bcc1da9b1b29766f3469 +size 633725 diff --git a/evals/registry/data/already_said_that/LICENSE b/evals/registry/data/already_said_that/LICENSE new file mode 100644 index 0000000000..5a6c138028 --- /dev/null +++ b/evals/registry/data/already_said_that/LICENSE @@ -0,0 +1,3 @@ +WordNet: +WordNet License: https://wordnet.princeton.edu/license-and-commercial-use +Source: https://wordnet.princeton.edu/ diff --git a/evals/registry/data/bugged_tools/main.jsonl b/evals/registry/data/bugged_tools/main.jsonl new file mode 100644 index 0000000000..64bc2aa725 --- /dev/null +++ b/evals/registry/data/bugged_tools/main.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:418a355616ca1749a425d2ebce92a6441d62c22858d391887e1aae554db9b724 +size 257364 diff --git a/evals/registry/data/bugged_tools/main_small.jsonl b/evals/registry/data/bugged_tools/main_small.jsonl new file mode 100644 index 0000000000..c2cd4596ca --- /dev/null +++ b/evals/registry/data/bugged_tools/main_small.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d3ef5a5fadf027b15af803dca5d4c61e9abfe27c29f984a76895467de79b30a1 +size 25661 diff --git a/evals/registry/data/cant_do_that_anymore/diagonal_moves_dataset.jsonl b/evals/registry/data/cant_do_that_anymore/diagonal_moves_dataset.jsonl new file mode 100644 index 0000000000..7cce7ab588 --- /dev/null +++ b/evals/registry/data/cant_do_that_anymore/diagonal_moves_dataset.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:345340a9c74ae6d3ad73393b43986c37fa30ad2df8e94d147d9f63cf519e703e +size 540964 diff --git a/evals/registry/data/cant_do_that_anymore/gpt-3.5-turbo-0125_dataset.jsonl b/evals/registry/data/cant_do_that_anymore/gpt-3.5-turbo-0125_dataset.jsonl new file mode 100644 index 0000000000..d63a762d37 --- /dev/null +++ b/evals/registry/data/cant_do_that_anymore/gpt-3.5-turbo-0125_dataset.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:08d0cbf162d7b46e8931c74816f597085d5d365895e7f8c9f9b20d98be0566c8 +size 170427 diff --git a/evals/registry/data/cant_do_that_anymore/gpt-3.5-turbo-instruct_dataset.jsonl b/evals/registry/data/cant_do_that_anymore/gpt-3.5-turbo-instruct_dataset.jsonl new file mode 100644 index 0000000000..43161bec40 --- /dev/null +++ b/evals/registry/data/cant_do_that_anymore/gpt-3.5-turbo-instruct_dataset.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d3d9927244f61a7e00d7b4d9e5521b8ad3249be08cbf8afd3c75b30fe8f4e9a5 +size 223466 diff --git a/evals/registry/data/cant_do_that_anymore/gpt-4-0125-preview_dataset.jsonl b/evals/registry/data/cant_do_that_anymore/gpt-4-0125-preview_dataset.jsonl new file mode 100644 index 0000000000..1c693f76de --- /dev/null +++ b/evals/registry/data/cant_do_that_anymore/gpt-4-0125-preview_dataset.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:80a41ce88bab1d6b9315835fa2845bb754ed52d0d7983857f255f5de0fd2fbdb +size 283930 diff --git a/evals/registry/data/cant_do_that_anymore/gpt-4-0314_dataset.jsonl b/evals/registry/data/cant_do_that_anymore/gpt-4-0314_dataset.jsonl new file mode 100644 index 0000000000..e6dffa7d4d --- /dev/null +++ b/evals/registry/data/cant_do_that_anymore/gpt-4-0314_dataset.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5df2376c0805ea323dddec11a01d5d843edce069f86550f2a9e91efcad4f51cc +size 549365 diff --git a/evals/registry/data/cant_do_that_anymore/special_moves_dataset.jsonl b/evals/registry/data/cant_do_that_anymore/special_moves_dataset.jsonl new file mode 100644 index 0000000000..6f5e89e691 --- /dev/null +++ b/evals/registry/data/cant_do_that_anymore/special_moves_dataset.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:baea567fbd18be57a6fba31a8e7d05a670bfd86799397269aa9b47ab6d2f2a5b +size 3381675 diff --git a/evals/registry/data/error_recovery/main.jsonl b/evals/registry/data/error_recovery/main.jsonl new file mode 100644 index 0000000000..77835457c7 --- /dev/null +++ b/evals/registry/data/error_recovery/main.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8fda8fddd6a63d6b84ee4b6a8934bcedcada67e3fcd5df64041f14c04d774be3 +size 1543818 diff --git a/evals/registry/data/error_recovery/medium.jsonl b/evals/registry/data/error_recovery/medium.jsonl new file mode 100644 index 0000000000..77b989dee3 --- /dev/null +++ b/evals/registry/data/error_recovery/medium.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d5c591504d282ca7763d7abe407958da1ea06d6dc62be4808ba4fa97ff5f3cb2 +size 280075 diff --git a/evals/registry/data/error_recovery/small.jsonl b/evals/registry/data/error_recovery/small.jsonl new file mode 100644 index 0000000000..64172d3d10 --- /dev/null +++ b/evals/registry/data/error_recovery/small.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3e55b1af640b26eff5661c83c7ff6bf52040ea062c9a71ba16069e2305fdb362 +size 10191 diff --git a/evals/registry/data/function_deduction/data.jsonl b/evals/registry/data/function_deduction/data.jsonl new file mode 100644 index 0000000000..bded32c52b --- /dev/null +++ b/evals/registry/data/function_deduction/data.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cb7cd13c1f67a7be8d153de26c7436a805035053f5497b77296e3f3615023e86 +size 50468 diff --git a/evals/registry/data/identifying_variables/balanced_ctrl_vars.jsonl b/evals/registry/data/identifying_variables/balanced_ctrl_vars.jsonl new file mode 100644 index 0000000000..c29a8ee65d --- /dev/null +++ b/evals/registry/data/identifying_variables/balanced_ctrl_vars.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e9429fe712578ae4298e012cc374198bf83cf968115004dc00d24e42ebdc4f1d +size 12525123 diff --git a/evals/registry/data/identifying_variables/balanced_hypotheses.jsonl b/evals/registry/data/identifying_variables/balanced_hypotheses.jsonl new file mode 100644 index 0000000000..cb05f29c53 --- /dev/null +++ b/evals/registry/data/identifying_variables/balanced_hypotheses.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e92ee79ee832d7f6f40e55cad82fe26100ea3c1ca1faac2f606a046ef4a09b79 +size 7554989 diff --git a/evals/registry/data/incontext_rl/samples.jsonl b/evals/registry/data/incontext_rl/samples.jsonl new file mode 100644 index 0000000000..acacbee595 --- /dev/null +++ b/evals/registry/data/incontext_rl/samples.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4a675930c0b31dcee9dca9f653085f9eb2b856c1284c289ed5501d44bd94fec5 +size 4138 diff --git a/evals/registry/data/incontext_rl/samples_dev.jsonl b/evals/registry/data/incontext_rl/samples_dev.jsonl new file mode 100644 index 0000000000..110af6c8cf --- /dev/null +++ b/evals/registry/data/incontext_rl/samples_dev.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:863664b313c3c8e77e3da6ad6f0ef695e8f86ff9d1ecdd7d5fcf0d408bf464da +size 1617 diff --git a/evals/registry/data/incontext_rl/samples_gymnasium_only.jsonl b/evals/registry/data/incontext_rl/samples_gymnasium_only.jsonl new file mode 100644 index 0000000000..d0448241d0 --- /dev/null +++ b/evals/registry/data/incontext_rl/samples_gymnasium_only.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7314053ae7203d627611fadb2d5f04f2aa6b001def00047bca206d0db43cb62b +size 3455 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/honduras.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/honduras.jsonl new file mode 100644 index 0000000000..cc818363b7 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/honduras.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:50b995b959aa7308a0be6413d005c0407984cf6f57a953c1fdde745f17df0db4 +size 72360 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/human_rights_miskito.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/human_rights_miskito.jsonl new file mode 100644 index 0000000000..fe48f48eef --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/human_rights_miskito.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a3baae4eade2acc21395c8b29a1f82cc05da00b7f7bc4cd458cc8ee2f7d032cb +size 10298 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_language.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_language.jsonl new file mode 100644 index 0000000000..7b118988e6 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_language.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2972b14f1f6aa0fb4246a3d4a964cf07c0dfc3e717b6036ccff7d1f6284e7812 +size 7399 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_lessons.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_lessons.jsonl new file mode 100644 index 0000000000..fd42093260 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_lessons.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f657754efc73614292b53c313583cd0013a9f7bde1e6018220d0bd15a546838c +size 43506 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_people.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_people.jsonl new file mode 100644 index 0000000000..eb18d39508 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/miskito_people.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2be3a3684c1586cc0779ae4cf47866d0e88bd8f67c5256438fe59aaa2e8a81b7 +size 53928 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/mosquito.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/mosquito.jsonl new file mode 100644 index 0000000000..f3abc68f3e --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/mosquito.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a69fde31a05e3f95e34bcbbc7e9986e3bf107513658a6e002ae8bb303d69d7d8 +size 28786 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/mosquito_coast.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/mosquito_coast.jsonl new file mode 100644 index 0000000000..29e7de5a3f --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/mosquito_coast.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1539eceeb715376db2db9eb17dcc7c5e43d0e71df710c65b71a7d6276c23dc44 +size 34533 diff --git a/evals/registry/data/skill_acquisition/miskito/knowledge_base/nicaragua.jsonl b/evals/registry/data/skill_acquisition/miskito/knowledge_base/nicaragua.jsonl new file mode 100644 index 0000000000..c71d1603ef --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/knowledge_base/nicaragua.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b446d17a582e1d8bdf2c6c46a742716c7290dc441559817c841361c3e33c39fd +size 80204 diff --git a/evals/registry/data/skill_acquisition/miskito/qa_pairs_by_lesson.jsonl b/evals/registry/data/skill_acquisition/miskito/qa_pairs_by_lesson.jsonl new file mode 100644 index 0000000000..226af11348 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/qa_pairs_by_lesson.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:92c631af79044257aea396b250a93eb466d404d637c6c0fc764a30763576f5ea +size 32651 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_all.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_all.jsonl new file mode 100644 index 0000000000..d72e8b83fa --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_all.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5c9540f646ea2610874b3e33286e300cd92b70d91f6c00f5b0275f1be918b74a +size 38464 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_all_fewshot.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_all_fewshot.jsonl new file mode 100644 index 0000000000..8114c4e111 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_all_fewshot.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2af150986f257a3e358d76b5a878d17116b583332eee303f0792fcffd1eee6d1 +size 37930 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_manipulation.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_manipulation.jsonl new file mode 100644 index 0000000000..151136d565 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_manipulation.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:51ec99e36e05dd2ee0f87f9177c4c4fc0155c744b1ed30d26bf4464ff7985e4f +size 28627 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_manipulation_fewshot.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_manipulation_fewshot.jsonl new file mode 100644 index 0000000000..6a5e655d82 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_manipulation_fewshot.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:62b665285283e232bbd670c32900078c77380b5c3c612d3fa11b4369e007edd5 +size 28201 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_translation.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_translation.jsonl new file mode 100644 index 0000000000..491624d22c --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_translation.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:19d8e9bbe4868479d2df0f6c7e72740399db5943dde1d3109c66affe878a62d8 +size 9836 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_translation_fewshot.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_translation_fewshot.jsonl new file mode 100644 index 0000000000..7e91554f6f --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_test_translation_fewshot.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2a76625921e1810e4ce22ba76f4881ee9327e1522555cc9eccc6beb854b7a129 +size 9236 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_all.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_all.jsonl new file mode 100644 index 0000000000..17d04c41d2 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_all.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f099000b117f1d5d46778263674998e9785f3e993b4c32ca8afb5f82065e1afb +size 560 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_manipulation.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_manipulation.jsonl new file mode 100644 index 0000000000..480e645ad3 --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_manipulation.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:625f3fa618e52688f6593774b7ba5691879882dbe9e3a8508a8aed43327f7d86 +size 425 diff --git a/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_translation.jsonl b/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_translation.jsonl new file mode 100644 index 0000000000..4f83ef3e2d --- /dev/null +++ b/evals/registry/data/skill_acquisition/miskito/variants/miskito_train_translation.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8c2f3a4699303b85d49641ec17ec77bff600c75940078d735406db1539da90c4 +size 599 diff --git a/evals/registry/data/twenty_questions/LICENSE b/evals/registry/data/twenty_questions/LICENSE new file mode 100644 index 0000000000..7b971d365d --- /dev/null +++ b/evals/registry/data/twenty_questions/LICENSE @@ -0,0 +1,3 @@ +lexical_simplification: +MIT License: https://opensource.org/licenses/MIT +Source: https://github.com/mounicam/lexical_simplification \ No newline at end of file diff --git a/evals/registry/data/twenty_questions/dataset.jsonl b/evals/registry/data/twenty_questions/dataset.jsonl new file mode 100644 index 0000000000..ea11e6a68e --- /dev/null +++ b/evals/registry/data/twenty_questions/dataset.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8a8358c42ef70c2c48c6bb2e214787e968cd1b092daeb1dd572f942bd7146bff +size 7664 diff --git a/evals/registry/data/twenty_questions/lexicon_nouns.jsonl b/evals/registry/data/twenty_questions/lexicon_nouns.jsonl new file mode 100644 index 0000000000..869a13feb1 --- /dev/null +++ b/evals/registry/data/twenty_questions/lexicon_nouns.jsonl @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:754d1f85637de87dac8aadfa5163f073d65289f27677031e765334c786742171 +size 112218 diff --git a/evals/registry/evals/already_said_that.yaml b/evals/registry/evals/already_said_that.yaml new file mode 100644 index 0000000000..2895544ffe --- /dev/null +++ b/evals/registry/evals/already_said_that.yaml @@ -0,0 +1,50 @@ +already_said_that: + id: already_said_that.reverse-sort-words-eng + metrics: + [ + "avg_num_turns", + "stddev_num_turns", + "median_num_turns", + "max_num_turns", + "min_num_turns", + "false_positive_rate", + "false_negative_rate", + "avg_distractor_accuracy", + "violation_rate", + "avg_num_distractors", + "stddev_num_distractors", + "median_num_distractors", + "max_num_distractors", + "min_num_distractors", + ] + description: "Sustain performance in the presence of distractors" + +already_said_that.which-is-heavier: + class: evals.elsuite.already_said_that.eval:AlreadySaidThat + args: + samples_jsonl: already_said_that/500_100.jsonl + distractor_variant: which-is-heavier + +already_said_that.first-letters: + class: evals.elsuite.already_said_that.eval:AlreadySaidThat + args: + samples_jsonl: already_said_that/500_100.jsonl + distractor_variant: first-letters + +already_said_that.ambiguous-sentences: + class: evals.elsuite.already_said_that.eval:AlreadySaidThat + args: + samples_jsonl: already_said_that/500_100.jsonl + distractor_variant: ambiguous-sentences + +already_said_that.reverse-sort-words-eng: + class: evals.elsuite.already_said_that.eval:AlreadySaidThat + args: + samples_jsonl: already_said_that/500_100.jsonl + distractor_variant: reverse-sort-words-eng + +already_said_that.distractorless: + class: evals.elsuite.already_said_that.eval:AlreadySaidThat + args: + samples_jsonl: already_said_that/500_100.jsonl + distractor_variant: distractorless diff --git a/evals/registry/evals/bugged_tools.yaml b/evals/registry/evals/bugged_tools.yaml new file mode 100644 index 0000000000..ff63e87321 --- /dev/null +++ b/evals/registry/evals/bugged_tools.yaml @@ -0,0 +1,31 @@ +bugged_tools: + id: bugged_tools.all + metrics: [f1, precision, recall, accuracy] + description: Evaluates ability to identify bugs in tools + +bugged_tools.all: + class: evals.elsuite.bugged_tools.eval:BuggedTools + args: + samples_jsonl: bugged_tools/main.jsonl + max_turns: 10 + log_all_metrics: False + use_judge: True + bug_instructions_type: simple_warning + +bugged_tools.all_log: + class: evals.elsuite.bugged_tools.eval:BuggedTools + args: + samples_jsonl: bugged_tools/main.jsonl + max_turns: 10 + log_all_metrics: True + use_judge: True + bug_instructions_type: simple_warning + +bugged_tools.all_small: + class: evals.elsuite.bugged_tools.eval:BuggedTools + args: + samples_jsonl: bugged_tools/main_small.jsonl + max_turns: 10 + log_all_metrics: False + use_judge: True + bug_instructions_type: simple_warning diff --git a/evals/registry/evals/cant_do_that_anymore.yaml b/evals/registry/evals/cant_do_that_anymore.yaml new file mode 100644 index 0000000000..d7254a9545 --- /dev/null +++ b/evals/registry/evals/cant_do_that_anymore.yaml @@ -0,0 +1,23 @@ +cant_do_that_anymore: + id: cant_do_that_anymore.all + metrics: [variant_impact_factor, delta, predicted_move_proportion, predicted_move_in_variant_proportion, avg_num_previous_moves, std_num_previous_moves] + description: Evaluates how well models can adapt to new rules of an environment (chess) + +cant_do_that_anymore.all: + class: evals.elsuite.cant_do_that_anymore.eval:CantDoThatAnymore + args: + default_model_dataset: "gpt-3.5-turbo-0125" + n_samples: 1000 + +cant_do_that_anymore.all_small: + class: evals.elsuite.cant_do_that_anymore.eval:CantDoThatAnymore + args: + default_model_dataset: "gpt-3.5-turbo-0125" + n_samples: 100 + +cant_do_that_anymore.all_diagonal: + class: evals.elsuite.cant_do_that_anymore.eval:CantDoThatAnymore + args: + default_model_dataset: "gpt-3.5-turbo-0125" + n_samples: 1000 + diagonal_variation: True diff --git a/evals/registry/evals/error_recovery.yaml b/evals/registry/evals/error_recovery.yaml new file mode 100644 index 0000000000..f42e0e9243 --- /dev/null +++ b/evals/registry/evals/error_recovery.yaml @@ -0,0 +1,36 @@ +error-recovery: + id: error-recovery.main + metrics: [accuracy] + description: TODO + +error-recovery.main: + class: evals.elsuite.error_recovery.eval:ErrorRecovery + args: + samples_jsonl: error_recovery/main.jsonl + +error-recovery.small: + class: evals.elsuite.error_recovery.eval:ErrorRecovery + args: + samples_jsonl: error_recovery/small.jsonl + +error-recovery.medium: + class: evals.elsuite.error_recovery.eval:ErrorRecovery + args: + samples_jsonl: error_recovery/medium.jsonl + +# --- mark reasoning as 'user' variant --- +error-recovery.main.other-reasoning: + class: evals.elsuite.error_recovery.eval:ErrorRecovery + args: + samples_jsonl: error_recovery/main.jsonl + mark_as_own_reasoning: False +error-recovery.small.other-reasoning: + class: evals.elsuite.error_recovery.eval:ErrorRecovery + args: + samples_jsonl: error_recovery/small.jsonl + mark_as_own_reasoning: False +error-recovery.medium.other-reasoning: + class: evals.elsuite.error_recovery.eval:ErrorRecovery + args: + samples_jsonl: error_recovery/medium.jsonl + mark_as_own_reasoning: False diff --git a/evals/registry/evals/function-deduction.yaml b/evals/registry/evals/function-deduction.yaml new file mode 100644 index 0000000000..337856cd72 --- /dev/null +++ b/evals/registry/evals/function-deduction.yaml @@ -0,0 +1,37 @@ +function_deduction: + id: function_deduction.easy + metrics: [adjusted_avg_rounds, solved_ratio, solved, samples, avg_success_rounds, avg_sample_rounds_std_adjusted, avg_sample_rounds_std_no_failed, solved_ratio_if_any_solved, avg_ask_rounds, avg_guess_rounds, avg_incorrect_format_rounds, solved_avg_complexity, not_solved_avg_complexity, solved_or_not_mann_whitney_u_p_value, sem_adjusted_avg_rounds, sem_avg_success_rounds, sem_avg_guess_rounds, sem_avg_incorrect_format_rounds] + description: Test a model's ability to deduce unknown functions + +function_deduction.easy: + class: evals.elsuite.function_deduction.eval:FunctionDeductionEval + args: + mode: easy + n_rounds: 20 + +function_deduction.easy.long: + class: evals.elsuite.function_deduction.eval:FunctionDeductionEval + args: + mode: easy + n_rounds: 20 + n_repeat: 10 + +function_deduction.easy.dev5: + class: evals.elsuite.function_deduction.eval:FunctionDeductionEval + args: + mode: easy + n_rounds: 20 + n_samples: 5 + +function_deduction.hard: + class: evals.elsuite.function_deduction.eval:FunctionDeductionEval + args: + mode: hard + n_rounds: 20 + +function_deduction.hard.dev5: + class: evals.elsuite.function_deduction.eval:FunctionDeductionEval + args: + mode: hard + n_rounds: 20 + n_samples: 5 diff --git a/evals/registry/evals/identifying_variables.yaml b/evals/registry/evals/identifying_variables.yaml new file mode 100644 index 0000000000..32f4ecbafa --- /dev/null +++ b/evals/registry/evals/identifying_variables.yaml @@ -0,0 +1,136 @@ +identifying_variables: + id: identifying_variables.language-corrset.balanced-ctrl + metrics: + [ + "ctrl_nDCG", + "ctrl_recall", + "ctrl_fallout", + "hyp_valid_acc", + "ind_acc", + "dep_acc", + "violation_rate", + ] + description: + "Evaluate the model's ability of identifying the right experimental + variables for testing a given hypothesis." + +# Balanced-hypotheses datasets + +identifying_variables.markdown.balanced-hypotheses: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + n_samples: 500 + renderer: markdown +identifying_variables.markdown.balanced-hypotheses-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + renderer: markdown + group_metrics: true + +identifying_variables.csv.balanced-hypotheses: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + n_samples: 500 + renderer: csv +identifying_variables.csv.balanced-hypotheses-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + renderer: csv + group_metrics: true + +identifying_variables.json.balanced-hypotheses: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + n_samples: 500 + renderer: json +identifying_variables.json.balanced-hypotheses-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + renderer: json + group_metrics: true + +identifying_variables.language-tabular.balanced-hypotheses: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + n_samples: 500 + renderer: language-tabular +identifying_variables.language-tabular.balanced-hypotheses-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + renderer: language-tabular + group_metrics: true + +identifying_variables.language-corrset.balanced-hypotheses: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + n_samples: 500 + renderer: language-corrset +identifying_variables.language-corrset.balanced-hypotheses-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + renderer: language-corrset + group_metrics: true + +identifying_variables.corrset.balanced-hypotheses: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + n_samples: 500 + renderer: corrset +identifying_variables.corrset.balanced-hypotheses-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_hypotheses.jsonl + renderer: corrset + group_metrics: true + +# Balanced-control datasets + +identifying_variables.csv.balanced-ctrl: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_ctrl_vars.jsonl + n_samples: 500 + renderer: csv +identifying_variables.csv.balanced-ctrl-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_ctrl_vars.jsonl + renderer: csv + group_metrics: true + +identifying_variables.language-corrset.balanced-ctrl: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_ctrl_vars.jsonl + n_samples: 500 + renderer: language-corrset +identifying_variables.language-corrset.balanced-ctrl-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_ctrl_vars.jsonl + renderer: language-corrset + group_metrics: true + +identifying_variables.corrset.balanced-ctrl: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_ctrl_vars.jsonl + n_samples: 500 + renderer: corrset +identifying_variables.corrset.balanced-ctrl-large: + class: evals.elsuite.identifying_variables.eval:IdentifyingVariables + args: + samples_jsonl: identifying_variables/balanced_ctrl_vars.jsonl + renderer: corrset + group_metrics: true diff --git a/evals/registry/evals/incontext_rl.yaml b/evals/registry/evals/incontext_rl.yaml new file mode 100644 index 0000000000..e66f358569 --- /dev/null +++ b/evals/registry/evals/incontext_rl.yaml @@ -0,0 +1,62 @@ +incontext_rl: + id: incontext_rl.gymnasium.v0 + metrics: [] + +incontext_rl.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: True + samples_jsonl: incontext_rl/samples.jsonl + +incontext_rl.raw.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: False + samples_jsonl: incontext_rl/samples.jsonl + +incontext_rl.gymnasium.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: True + samples_jsonl: incontext_rl/samples_gymnasium_only.jsonl + +incontext_rl.gymnasium.raw.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: False + samples_jsonl: incontext_rl/samples_gymnasium_only.jsonl + +incontext_rl.short.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: True + max_steps: 100 + samples_jsonl: incontext_rl/samples.jsonl + +incontext_rl.raw.short.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: False + max_steps: 100 + samples_jsonl: incontext_rl/samples.jsonl + +incontext_rl.gymnasium.short.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: True + max_steps: 100 + samples_jsonl: incontext_rl/samples_gymnasium_only.jsonl + +incontext_rl.gymnasium.raw.short.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: False + max_steps: 100 + samples_jsonl: incontext_rl/samples_gymnasium_only.jsonl + +incontext_rl.dev.v0: + class: evals.elsuite.incontext_rl.eval:InContextRl + args: + use_explanations: True + max_steps: 5 + samples_jsonl: incontext_rl/samples.jsonl \ No newline at end of file diff --git a/evals/registry/evals/skill_acquisition.yaml b/evals/registry/evals/skill_acquisition.yaml new file mode 100644 index 0000000000..2b4594efff --- /dev/null +++ b/evals/registry/evals/skill_acquisition.yaml @@ -0,0 +1,107 @@ +# --------------- +# Miskito dataset +# --------------- + +skill_acquisition.miskito: + id: skill_acquisition.miskito.zero_shot.full + metrics: [delta_accuracy] + description: Evaluates whether models can learn the Miskito language through retrieval. + +# Miskito manipulation + translation dataset, zero- and few-shot. +skill_acquisition.miskito.zero_shot.full: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_all.jsonl + target_language: miskito + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 +skill_acquisition.miskito.zero_shot.dev5: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_all.jsonl + target_language: miskito + n_samples: 5 + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 + +skill_acquisition.miskito.few_shot.full: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_all_fewshot.jsonl + target_language: miskito + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 +skill_acquisition.miskito.few_shot.dev5: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_all_fewshot.jsonl + target_language: miskito + n_samples: 5 + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 + +# Miskito translation-only, zero- and few-shot. +skill_acquisition.miskito.zero_shot.translation.full: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_translation.jsonl + target_language: miskito + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 +skill_acquisition.miskito.zero_shot.translation.dev5: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_translation.jsonl + target_language: miskito + n_samples: 5 + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 + +skill_acquisition.miskito.few_shot.translation.full: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_translation_fewshot.jsonl + target_language: miskito + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 +skill_acquisition.miskito.few_shot.translation.dev5: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_translation_fewshot.jsonl + target_language: miskito + n_samples: 5 + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 + +# Miskito manipulation-only, zero- and few-shot. +skill_acquisition.miskito.zero_shot.manipulation.full: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_manipulation.jsonl + target_language: miskito + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 +skill_acquisition.miskito.zero_shot.manipulation.dev5: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_manipulation.jsonl + target_language: miskito + n_samples: 5 + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 + +skill_acquisition.miskito.few_shot.manipulation.full: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_manipulation_fewshot.jsonl + target_language: miskito + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 +skill_acquisition.miskito.few_shot.manipulation.dev5: + class: evals.elsuite.skill_acquisition.eval:SkillAcquisition + args: + samples_jsonl: skill_acquisition/miskito/variants/miskito_test_manipulation_fewshot.jsonl + target_language: miskito + n_samples: 5 + knowledge_base_directory: skill_acquisition/miskito/knowledge_base/ + max_replies: 30 \ No newline at end of file diff --git a/evals/registry/evals/track_the_stat.yaml b/evals/registry/evals/track_the_stat.yaml new file mode 100644 index 0000000000..c64ce0ed40 --- /dev/null +++ b/evals/registry/evals/track_the_stat.yaml @@ -0,0 +1,22 @@ +track_the_stat: + id: track_the_stat.mode + metrics: + [ + "avg_max_length", + "stddev_max_length", + "median_max_length", + "max_max_length", + "min_max_length", + "violation_rate", + ] + description: "Perform a sequential task by keeping track of state implicitly" + +track_the_stat.mode: + class: evals.elsuite.track_the_stat.eval:TrackTheStat + args: + task: mode + +track_the_stat.median: + class: evals.elsuite.track_the_stat.eval:TrackTheStat + args: + task: median diff --git a/evals/registry/evals/twenty_questions.yaml b/evals/registry/evals/twenty_questions.yaml new file mode 100644 index 0000000000..af3491ffcf --- /dev/null +++ b/evals/registry/evals/twenty_questions.yaml @@ -0,0 +1,60 @@ +twenty_questions: + id: twenty_questions.full + description: Tests models on the 20 questions game. + metrics: [score, accuracy, average_num_guesses, average_num_questions, average_num_violations, average_num_gamemaster_refusals, average_num_incorrect_guesses, average_word_difficulty] + +twenty_questions.full: + class: evals.elsuite.twenty_questions.eval:TwentyQuestions + args: + samples_jsonl: twenty_questions/dataset.jsonl + gamemaster_spec: twenty_questions/gamemaster/gpt-4-turbo-preview + max_questions: 20 + max_replies: 40 + +twenty_questions.shortlist.full: + class: evals.elsuite.twenty_questions.eval:TwentyQuestions + args: + samples_jsonl: twenty_questions/dataset.jsonl + gamemaster_spec: twenty_questions/gamemaster/gpt-4-turbo-preview + shortlist_variant: True + max_questions: 20 + max_replies: 40 + +twenty_questions.dev5: + class: evals.elsuite.twenty_questions.eval:TwentyQuestions + args: + samples_jsonl: twenty_questions/dataset.jsonl + gamemaster_spec: twenty_questions/gamemaster/gpt-4-turbo-preview + n_samples: 5 + max_questions: 20 + max_replies: 40 + +twenty_questions.shortlist.dev5: + class: evals.elsuite.twenty_questions.eval:TwentyQuestions + args: + samples_jsonl: twenty_questions/dataset.jsonl + gamemaster_spec: twenty_questions/gamemaster/gpt-4-turbo-preview + n_samples: 5 + shortlist_variant: True + num_shortlist_items: 5 + max_questions: 20 + max_replies: 40 + +twenty_questions.dev100: + class: evals.elsuite.twenty_questions.eval:TwentyQuestions + args: + samples_jsonl: twenty_questions/dataset.jsonl + gamemaster_spec: twenty_questions/gamemaster/gpt-4-turbo-preview + n_samples: 100 + max_questions: 20 + max_replies: 40 + +twenty_questions.shortlist.dev100: + class: evals.elsuite.twenty_questions.eval:TwentyQuestions + args: + samples_jsonl: twenty_questions/dataset.jsonl + gamemaster_spec: twenty_questions/gamemaster/gpt-4-turbo-preview + n_samples: 100 + shortlist_variant: True + max_questions: 20 + max_replies: 40 diff --git a/evals/registry/solvers/already_said_that.yaml b/evals/registry/solvers/already_said_that.yaml new file mode 100644 index 0000000000..71f0b65ff2 --- /dev/null +++ b/evals/registry/solvers/already_said_that.yaml @@ -0,0 +1,79 @@ +already_said_that/random_baseline: + class: evals.elsuite.already_said_that.solvers:RandomBaselineSolver + +already_said_that/human_cli: + class: evals.elsuite.already_said_that.solvers:AlreadySaidThatHuman + args: + human_cli_solver: + class: evals.solvers.human_cli_solver:HumanCliSolver + args: + registry: null + +already_said_that/cot/gpt-3.5-turbo: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + persistent_memory: False + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + +already_said_that/cot/gpt-4-turbo-preview: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + persistent_memory: False + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + +already_said_that/cot_hhh/gpt-4-base: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + persistent_memory: False + cot_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 diff --git a/evals/registry/solvers/cant_do_that_anymore.yaml b/evals/registry/solvers/cant_do_that_anymore.yaml new file mode 100644 index 0000000000..951dd066bf --- /dev/null +++ b/evals/registry/solvers/cant_do_that_anymore.yaml @@ -0,0 +1,17 @@ +chess/generation/direct/gpt-3.5-turbo-instruct: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo-instruct + extra_options: + temperature: 1 + max_tokens: 4 + +chess/generation/direct/gpt-4-base: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 4 diff --git a/evals/registry/solvers/error_recovery.yaml b/evals/registry/solvers/error_recovery.yaml new file mode 100644 index 0000000000..bef801549e --- /dev/null +++ b/evals/registry/solvers/error_recovery.yaml @@ -0,0 +1,38 @@ +# TODO: use default solvers once they are versioned +error_recovery/gpt-3.5-turbo-0613: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo-0613 + +error_recovery/gpt-4-0613: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-0613 + +error_recovery/default/gpt-4-base: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + +# solver that continues the previous message +error_recovery/continue/gpt-4-base: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + continue_last_assistant_msg: True + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 diff --git a/evals/registry/solvers/function_deduction.yaml b/evals/registry/solvers/function_deduction.yaml new file mode 100644 index 0000000000..9b8837851b --- /dev/null +++ b/evals/registry/solvers/function_deduction.yaml @@ -0,0 +1,192 @@ +# OS CHAIN OF THOUGHT +function_deduction/cot/llama-2-13b-chat: + class: evals.elsuite.function_deduction.solvers:CustomCoT + args: + cot_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-13b-chat-hf + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-13b-chat-hf + extra_options: + temperature: 0 + max_tokens: 32 + +function_deduction/cot/llama-2-70b-chat: + class: evals.elsuite.function_deduction.solvers:CustomCoT + args: + cot_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-70b-chat-hf + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-70b-chat-hf + extra_options: + temperature: 0 + max_tokens: 32 + +function_deduction/cot/mixtral-8x7b-instruct: + class: evals.elsuite.function_deduction.solvers:CustomCoT + args: + cot_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: mistralai/Mixtral-8x7B-Instruct-v0.1 + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: mistralai/Mixtral-8x7B-Instruct-v0.1 + extra_options: + temperature: 0 + max_tokens: 32 + + +# CUSTOM CHAIN OF THOUGHT +function_deduction/cot/gpt-4-1106-preview: + class: evals.elsuite.function_deduction.solvers:CustomCoT + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-1106-preview + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-1106-preview + extra_options: + temperature: 0 + max_tokens: 32 + +function_deduction/cot/gpt-4-32k: + class: evals.elsuite.function_deduction.solvers:CustomCoT + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-32k + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-32k + extra_options: + temperature: 0 + max_tokens: 32 + +function_deduction/cot/gpt-3.5-turbo-16k: + class: evals.elsuite.function_deduction.solvers:CustomCoT + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo-16k + extra_options: + temperature: 1 + max_tokens: 512 + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo-16k + extra_options: + temperature: 0 + max_tokens: 32 + +function_deduction/cot/gemini-pro: + class: evals.elsuite.function_deduction.solvers:CustomCoT + args: + cot_solver: + class: evals.solvers.providers.google.gemini_solver:GeminiSolver + args: + model_name: gemini-pro + extract_solver: + class: evals.solvers.providers.google.gemini_solver:GeminiSolver + args: + model_name: gemini-pro + + +# BASE MODELS +function_deduction/gpt-4-base: + class: evals.elsuite.function_deduction.solvers:BaseModelSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 32 + +function_deduction/cot/gpt-4-base: + class: evals.elsuite.function_deduction.solvers:BaseModelCoTSolver + args: + cot_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + fixed_start: "Let's think step by step. " + extract_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 0 + max_tokens: 32 + + +# BASELINES +function_deduction/average_baseline: + class: evals.elsuite.function_deduction.baselines:AverageBaseline + +function_deduction/full_knowledge_random: + class: evals.elsuite.function_deduction.baselines:FullKnowledge + args: + mode: random + samples_jsonl: function_deduction/data.jsonl + +function_deduction/full_knowledge_best: + class: evals.elsuite.function_deduction.baselines:FullKnowledge + args: + mode: best + samples_jsonl: function_deduction/data.jsonl diff --git a/evals/registry/solvers/identifying_variables.yaml b/evals/registry/solvers/identifying_variables.yaml new file mode 100644 index 0000000000..aa6108febc --- /dev/null +++ b/evals/registry/solvers/identifying_variables.yaml @@ -0,0 +1,5 @@ +identifying_variables/random: + class: evals.elsuite.identifying_variables.solvers:RandomSolver + +identifying_variables/noctrl: + class: evals.elsuite.identifying_variables.solvers:NoCtrl diff --git a/evals/registry/solvers/incontext_rl.yaml b/evals/registry/solvers/incontext_rl.yaml new file mode 100644 index 0000000000..e374f2e75d --- /dev/null +++ b/evals/registry/solvers/incontext_rl.yaml @@ -0,0 +1,27 @@ +incontext_rl/random: + class: evals.elsuite.incontext_rl.baselines:RandomSolver + +incontext_rl/q-learning: + class: evals.elsuite.incontext_rl.baselines:QlearningSolver + +incontext_rl/anti-cot/gpt-3.5-turbo: + class: evals.elsuite.incontext_rl.anti-cot_solver:AntiCoTSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + +incontext_rl/anti-cot/gpt-4-turbo-preview: + class: evals.elsuite.incontext_rl.anti-cot_solver:AntiCoTSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 \ No newline at end of file diff --git a/evals/registry/solvers/skill_acquisition.yaml b/evals/registry/solvers/skill_acquisition.yaml new file mode 100644 index 0000000000..e187837297 --- /dev/null +++ b/evals/registry/solvers/skill_acquisition.yaml @@ -0,0 +1,287 @@ +# CoT solvers with a custom extraction prompt. +skill_acquisition/cot/gpt-3.5-turbo: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: &extract_template Given the above reasoning, what is the next action you wish to take? Please respond in the format required by the instructions. + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + +skill_acquisition/cot/gpt-4-turbo-preview: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + +skill_acquisition/cot/gemini-pro: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.providers.google.gemini_solver:GeminiSolver + args: + model_name: gemini-pro + extract_template: *extract_template + extract_solver: + class: evals.solvers.providers.google.gemini_solver:GeminiSolver + args: + model_name: gemini-pro + +skill_acquisition/cot/gpt-4: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4 + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4 + extra_options: + temperature: 1 + max_tokens: 512 + +skill_acquisition/cot_hhh/gpt-4-base: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + +skill_acquisition/assistants/gpt-4-turbo-preview: + class: evals.elsuite.skill_acquisition.solvers:SkillAcquisitionAssistantsSolver + args: + tools: + - type: code_interpreter + - type: retrieval + model: gpt-4-turbo-preview + +skill_acquisition/cot_assistant/gpt-4-turbo-preview: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.elsuite.skill_acquisition.solvers:SkillAcquisitionAssistantsSolver + args: + tools: + - type: code_interpreter + - type: retrieval + model: gpt-4-turbo-preview + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + +### Few-shot solvers. +# TODO: refactor few-shot solver so that train_jsonl is not parameterised here to reduce verbosity. +# Miskito full. +miskito_all/fewshot_direct/gpt-3.5-turbo: + class: evals.solvers.nested.fewshot_solver:FewShotSolver + args: + train_jsonl: evals/registry/data/skill_acquisition/miskito/variants/miskito_train_all.jsonl + n_shots: 3 + base_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + +miskito_all/fewshot_direct/gpt-4-turbo-preview: + class: evals.solvers.nested.fewshot_solver:FewShotSolver + args: + train_jsonl: evals/registry/data/skill_acquisition/miskito/variants/miskito_train_all.jsonl + n_shots: 3 + base_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + +miskito_all/fewshot_direct/gpt-4-32k: + class: evals.solvers.nested.fewshot_solver:FewShotSolver + args: + train_jsonl: evals/registry/data/skill_acquisition/miskito/variants/miskito_train_all.jsonl + n_shots: 3 + base_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-32k + extra_options: + temperature: 1 + max_tokens: 512 + +miskito_all/fewshot_direct/gpt-4-base: + class: evals.solvers.nested.fewshot_solver:FewShotSolver + args: + train_jsonl: evals/registry/data/skill_acquisition/miskito/variants/miskito_train_all.jsonl + n_shots: 3 + base_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + +miskito_manipulation/fewshot_direct/gpt-4-32k: + class: evals.solvers.nested.fewshot_solver:FewShotSolver + args: + train_jsonl: evals/registry/data/skill_acquisition/miskito/variants/miskito_train_manipulation.jsonl + n_shots: 3 + base_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-32k + extra_options: + temperature: 1 + max_tokens: 512 + +miskito_manipulation/fewshot_direct/gpt-4-base: + class: evals.solvers.nested.fewshot_solver:FewShotSolver + args: + train_jsonl: evals/registry/data/skill_acquisition/miskito/variants/miskito_train_manipulation.jsonl + n_shots: 3 + base_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + +# OS models +skill_acquisition/cot/llama-2-13b-chat: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-13b-chat-hf + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-13b-chat-hf + extra_options: + temperature: 1 + max_tokens: 512 + +skill_acquisition/cot/llama-2-70b-chat: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-70b-chat-hf + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-70b-chat-hf + extra_options: + temperature: 1 + max_tokens: 512 + +skill_acquisition/cot/mixtral-8x7b-instruct: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: mistralai/Mixtral-8x7B-Instruct-v0.1 + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: mistralai/Mixtral-8x7B-Instruct-v0.1 + extra_options: + temperature: 1 + max_tokens: 512 diff --git a/evals/registry/solvers/track_the_stat.yaml b/evals/registry/solvers/track_the_stat.yaml new file mode 100644 index 0000000000..fc061f9583 --- /dev/null +++ b/evals/registry/solvers/track_the_stat.yaml @@ -0,0 +1,82 @@ +track_the_stat/explicit_state/gemini-pro: + class: evals.elsuite.track_the_stat.solvers:ExplicitStateSolver + args: + underlying_solver: + class: evals.solvers.providers.google.gemini_solver:GeminiSolver + args: + model_name: gemini-pro + state_role: "user" + +track_the_stat/explicit_state/llama-2-70b-chat: + class: evals.elsuite.track_the_stat.solvers:ExplicitStateSolver + args: + underlying_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: meta-llama/Llama-2-70b-chat-hf + extra_options: + temperature: 1 + max_tokens: 512 + +track_the_stat/explicit_state/mixtral-8x7b-instruct: + class: evals.elsuite.track_the_stat.solvers:ExplicitStateSolver + args: + underlying_solver: + class: evals.solvers.together_solver:TogetherSolver + args: + completion_fn_options: + model: mistralai/Mixtral-8x7B-Instruct-v0.1 + extra_options: + temperature: 1 + max_tokens: 512 + +track_the_stat/explicit_state/gpt-3.5-turbo: + class: evals.elsuite.track_the_stat.solvers:ExplicitStateSolver + args: + underlying_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + +track_the_stat/explicit_state/gpt-4-turbo-preview: + class: evals.elsuite.track_the_stat.solvers:ExplicitStateSolver + args: + underlying_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + +track_the_stat/explicit_state/hhh/gpt-4-base: + class: evals.elsuite.track_the_stat.solvers:ExplicitStateSolver + args: + underlying_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + +track_the_stat/human_cli: + class: evals.elsuite.track_the_stat.solvers:TrackTheStatHuman + args: + human_cli_solver: + class: evals.solvers.human_cli_solver:HumanCliSolver + args: + registry: null + +track_the_stat/random_baseline: + class: evals.elsuite.track_the_stat.solvers:RandomBaselineSolver diff --git a/evals/registry/solvers/twenty_questions.yaml b/evals/registry/solvers/twenty_questions.yaml new file mode 100644 index 0000000000..81cc65468c --- /dev/null +++ b/evals/registry/solvers/twenty_questions.yaml @@ -0,0 +1,80 @@ +# CoT solvers with a custom extract template. +twenty_questions/cot/gpt-3.5-turbo: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: &extract_template Given the above reasoning, ask a question or make a guess following the task instructions. + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-3.5-turbo + extra_options: + temperature: 1 + max_tokens: 512 + +twenty_questions/cot/gpt-4-turbo-preview: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 1 + max_tokens: 512 + +twenty_questions/cot_hhh/gpt-4-base: + class: evals.solvers.nested.cot_solver:CoTSolver + args: + cot_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + extract_template: *extract_template + extract_solver: + class: evals.solvers.nested.hhh_solver:HHHSolver + args: + solver: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-base + extra_options: + temperature: 1 + max_tokens: 512 + +# Game-master uses a fixed solver, currently set to the latest-generation model. +twenty_questions/gamemaster/gpt-4-turbo-preview: + class: evals.solvers.openai_solver:OpenAISolver + args: + completion_fn_options: + model: gpt-4-turbo-preview + extra_options: + temperature: 0 + max_tokens: 1 + valid_answers: ["yes", "no", "skip"] \ No newline at end of file diff --git a/evals/utils/log_utils.py b/evals/utils/log_utils.py index d54a846f41..6ef2b5e8ff 100644 --- a/evals/utils/log_utils.py +++ b/evals/utils/log_utils.py @@ -14,6 +14,17 @@ def get_final_results_from_dir(log_dir: Union[str, Path]) -> dict[Path, dict]: return final_results_dict +def get_specs_from_dir(log_dir: Union[str, Path]) -> dict[Path, dict]: + """ + Given a directory of log files, return a dictionary mapping log file paths to specs. + """ + specs_dict = {} + for path in Path(log_dir).glob("**/*.log"): + spec = extract_spec(path) + specs_dict[path] = spec + return specs_dict + + def extract_final_results(path: Path) -> dict: """ Given a path to a log file, find and return the "final_report" dictionary. @@ -31,7 +42,7 @@ def extract_final_results(path: Path) -> dict: raise ValueError(f"Could not find final_report in {path}") -def extract_individual_results(path: Path) -> list[dict]: +def extract_individual_results(path: Path, type_string: str = "metrics") -> list[dict]: """ Given a path to a log file, grab all the individual sample results. """ @@ -42,7 +53,7 @@ def extract_individual_results(path: Path) -> list[dict]: try: loaded_line = json.loads(line) if "type" in loaded_line: - if loaded_line["type"] == "metrics": + if loaded_line["type"] == type_string: all_data.append(loaded_line["data"]) except json.decoder.JSONDecodeError: print(f"Skipping line: {line}") diff --git a/pyproject.toml b/pyproject.toml index e21afbf479..557d23a946 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,6 +40,9 @@ dependencies = [ "aiolimiter", "beartype==0.12.0", "flask", + "gymnasium", + "networkx", + "chess", ] [project.urls]