01_create_cefr_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate synthetic CEFR data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a first step we use LLMs to generate synthetic text samples in the [6 CEFR language levels from A1 to C2](https://www.coe.int/en/web/common-european-framework-reference-languages/level-descriptions). To generate a wide variety of samples we use Claude Haiku and Sonnet and GPT-4o and GPT-4o-mini models as well as Phi-3 and Gemma-2.\n",
    "\n",
    "From [our text simplification project](https://github.com/machinelearningZH/simply-simplify-language) we know, that LLMs to some degree can be steered towards the CEFR levels as well as Einfache or Leichte Sprache. We therefore assume that such text samples were in the training data. We can leverage this to our advantage and create a reference dataset to map our understandability scores to CEFR levels. We acknowledge that this is an educated guess and obviously not to be considered as ground truth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Imports**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import glob\n",
    "import re\n",
    "from tqdm.notebook import tqdm\n",
    "from dotenv import load_dotenv\n",
    "from openai import OpenAI\n",
    "from anthropic import Anthropic\n",
    "\n",
    "pd.options.mode.chained_assignment = None\n",
    "pd.options.display.max_rows = 500\n",
    "pd.options.display.max_seq_items = 500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Constants and functions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "load_dotenv()\n",
    "\n",
    "ANTHROPIC_API_KEY = os.getenv(\"ANTHROPIC_API_KEY\")\n",
    "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils_prompts import (\n",
    "    BASE_PROMPT_SITUATIONS,\n",
    "    BASE_PROMPT_SWISS,\n",
    "    BASE_PROMPT_TOPICS,\n",
    "    MODIFIERS_SITUATIONS,\n",
    "    MODIFIERS_SWISS,\n",
    "    MODIFIERS_TOPICS,\n",
    ")\n",
    "\n",
    "SYSTEM_MESSAGE = \"\"\"\"Du bist ein Experte in der deutschen Sprache. Du kennst dich exzellent mit den CEFR-Sprachniveaus von A1 bis C2 aus. Du kannst Texte sehr gut in diesen verschiedenen Niveaus schreiben. Du schreibst immer auf Deutsch.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate client to use with [LM Studio Server](https://lmstudio.ai/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "lmstudio_client = OpenAI(base_url=\"http://localhost:1234/v1\", api_key=\"lm-studio\")\n",
    "\n",
    "\n",
    "def call_lmstudio(prompt, temperature=0.5):\n",
    "    try:\n",
    "        completion = lmstudio_client.chat.completions.create(\n",
    "            model=\"bartowski/Phi-3-medium-4k-instruct-GGUF\",\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": SYSTEM_MESSAGE},\n",
    "                {\"role\": \"user\", \"content\": prompt},\n",
    "            ],\n",
    "            temperature=temperature,\n",
    "        )\n",
    "\n",
    "        return completion.choices[0].message.content\n",
    "    except Exception as e:\n",
    "        print(f\"Error: {e}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate clients for Anthropic and OpenAI APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "anthropic_client = Anthropic(api_key=ANTHROPIC_API_KEY)\n",
    "\n",
    "HAIKU = \"claude-3-haiku-20240307\"\n",
    "SONNET = \"claude-3-5-sonnet-20240620\"\n",
    "\n",
    "\n",
    "def call_anthropic(\n",
    "    prompt, model_id=\"claude-3-haiku-20240307\", temperature=0.5, max_tokens=4096\n",
    "):\n",
    "    try:\n",
    "        message = anthropic_client.messages.create(\n",
    "            model=model_id,\n",
    "            max_tokens=max_tokens,\n",
    "            temperature=temperature,\n",
    "            system=SYSTEM_MESSAGE,\n",
    "            messages=[\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": prompt,\n",
    "                }\n",
    "            ],\n",
    "        )\n",
    "        return message.content[0].text\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Error: {e}\")\n",
    "        return None\n",
    "\n",
    "\n",
    "openai_client = OpenAI()\n",
    "\n",
    "GPT4O_MINI = \"gpt-4o-mini\"\n",
    "GPT4O = \"gpt-4o\"\n",
    "\n",
    "\n",
    "def call_openai(prompt, model_id=GPT4O_MINI, temperature=0.5, max_tokens=4096):\n",
    "    try:\n",
    "        completion = openai_client.chat.completions.create(\n",
    "            model=model_id,\n",
    "            temperature=temperature,\n",
    "            max_tokens=max_tokens,\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": SYSTEM_MESSAGE},\n",
    "                {\"role\": \"user\", \"content\": prompt},\n",
    "            ],\n",
    "        )\n",
    "        return completion.choices[0].message.content\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Error: {e}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_response(response):\n",
    "    \"\"\"Extract the A1, A2, B1, B2, C1, C2 text spans from the response.\"\"\"\n",
    "    a1 = re.findall(r\"<A1>(.*?)<\", response, re.DOTALL)\n",
    "    a2 = re.findall(r\"<A2>(.*?)<\", response, re.DOTALL)\n",
    "    b1 = re.findall(r\"<B1>(.*?)<\", response, re.DOTALL)\n",
    "    b2 = re.findall(r\"<B2>(.*?)<\", response, re.DOTALL)\n",
    "    c1 = re.findall(r\"<C1>(.*?)<\", response, re.DOTALL)\n",
    "    c2 = re.findall(r\"<C2>(.*?)<\", response, re.DOTALL)\n",
    "    data = [a1, a2, b1, b2, c1, c2]\n",
    "    try:\n",
    "        data = [x[0].strip() for x in data]\n",
    "    except:\n",
    "        data = [None for x in data]\n",
    "    return pd.DataFrame(data).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create synthetic data with LM Studio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_results = []\n",
    "for modifier in tqdm(MODIFIERS_SITUATIONS):\n",
    "    print(modifier)\n",
    "    prompt = BASE_PROMPT_SITUATIONS.format(prompt=modifier)\n",
    "    result = call_lmstudio(prompt)\n",
    "    result = parse_response(result)\n",
    "    result[\"modifier\"] = modifier\n",
    "    tmp_results.append(result)\n",
    "\n",
    "data = pd.concat(tmp_results)\n",
    "data.columns = [\"A1\", \"A2\", \"B1\", \"B2\", \"C1\", \"C2\", \"modifier\"]\n",
    "data.reset_index(drop=True, inplace=True)\n",
    "\n",
    "data = data.replace(\"\", None).dropna()\n",
    "data.drop_duplicates(inplace=True)\n",
    "data.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.to_parquet(\"_input/phi34kit_situations.parq\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create synthetic data with Claude models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = HAIKU\n",
    "model_name = \"haiku\"\n",
    "\n",
    "tmp_results = []\n",
    "for modifier in tqdm(MODIFIERS_SITUATIONS):\n",
    "    print(modifier)\n",
    "    prompt = BASE_PROMPT_SITUATIONS.format(prompt=modifier)\n",
    "    result = call_anthropic(prompt, model_id=model)\n",
    "    result = parse_response(result)\n",
    "    result[\"modifier\"] = modifier\n",
    "    tmp_results.append(result)\n",
    "data = pd.concat(tmp_results)\n",
    "data.columns = [\"A1\", \"A2\", \"B1\", \"B2\", \"C1\", \"C2\", \"modifier\"]\n",
    "data.reset_index(drop=True, inplace=True)\n",
    "data[\"model\"] = model_name\n",
    "data[\"model_id\"] = model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.to_parquet(\"_input/haiku_situations.parq\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create synthetic data with GPT-4 models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_results = []\n",
    "for modifier in tqdm(MODIFIERS_SITUATIONS):\n",
    "    print(modifier)\n",
    "    prompt = BASE_PROMPT_SITUATIONS.format(prompt=modifier)\n",
    "    result = call_openai(prompt, model_id=GPT4O_MINI)\n",
    "    result = parse_response(result)\n",
    "    result[\"modifier\"] = modifier\n",
    "    tmp_results.append(result)\n",
    "\n",
    "data = pd.concat(tmp_results)\n",
    "data.columns = [\"A1\", \"A2\", \"B1\", \"B2\", \"C1\", \"C2\", \"modifier\"]\n",
    "data.reset_index(drop=True, inplace=True)\n",
    "data[\"model\"] = \"GPT-4o-mini\"\n",
    "data[\"model_id\"] = GPT4O_MINI\n",
    "\n",
    "# GPT-4o sometimes gets the tags wrong. We drop these erroneous rows.\n",
    "data = data.replace(\"\", None).dropna()\n",
    "data.drop_duplicates(inplace=True)\n",
    "data.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.to_parquet(\"_input/gpt4o-mini_situations.parq\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Combine synthetic data to final dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['_input/_synthetic-data/gemma2it9b_situations.parq',\n",
       " '_input/_synthetic-data/gemma2it9b_swiss.parq',\n",
       " '_input/_synthetic-data/gemma2it9b_topics.parq',\n",
       " '_input/_synthetic-data/gpt4o-mini_situations.parq',\n",
       " '_input/_synthetic-data/gpt4o-mini_swiss.parq',\n",
       " '_input/_synthetic-data/gpt4o-mini_topics.parq',\n",
       " '_input/_synthetic-data/gpt4o_situations.parq',\n",
       " '_input/_synthetic-data/gpt4o_swiss.parq',\n",
       " '_input/_synthetic-data/gpt4o_topics.parq',\n",
       " '_input/_synthetic-data/haiku_situations.parq',\n",
       " '_input/_synthetic-data/haiku_swiss.parq',\n",
       " '_input/_synthetic-data/haiku_topics.parq',\n",
       " '_input/_synthetic-data/phi34kit_situations.parq',\n",
       " '_input/_synthetic-data/phi34kit_swiss.parq',\n",
       " '_input/_synthetic-data/phi34kit_topics.parq',\n",
       " '_input/_synthetic-data/sonnet_situations.parq',\n",
       " '_input/_synthetic-data/sonnet_swiss.parq',\n",
       " '_input/_synthetic-data/sonnet_topics.parq']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_paths = glob.glob(\"_input/_synthetic-data/*.parq\")\n",
    "file_paths = sorted(file_paths)\n",
    "file_paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gemma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 120 entries, 0 to 119\n",
      "Data columns (total 9 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   A1        120 non-null    object\n",
      " 1   A2        120 non-null    object\n",
      " 2   B1        120 non-null    object\n",
      " 3   B2        120 non-null    object\n",
      " 4   C1        120 non-null    object\n",
      " 5   C2        120 non-null    object\n",
      " 6   model     120 non-null    object\n",
      " 7   topic     120 non-null    object\n",
      " 8   modifier  50 non-null     object\n",
      "dtypes: object(9)\n",
      "memory usage: 8.6+ KB\n"
     ]
    }
   ],
   "source": [
    "gemmas = [x for x in file_paths if \"gemma\" in x]\n",
    "frames = []\n",
    "for frame in gemmas:\n",
    "    model = frame.split(\"/\")[-1].split(\"_\")[0]\n",
    "    topic = frame.split(\"/\")[-1].split(\"_\")[1].split(\".\")[0]\n",
    "    data = pd.read_parquet(frame)\n",
    "    data[\"model\"] = model\n",
    "    data[\"topic\"] = topic\n",
    "    frames.append(data)\n",
    "\n",
    "df_gemma = pd.concat(frames).reset_index(drop=True)\n",
    "\n",
    "# Drop additional topics for now.\n",
    "df_gemma = df_gemma[df_gemma.topic.isin([\"situations\", \"swiss\", \"topics\"])]\n",
    "df_gemma.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Phi-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 120 entries, 0 to 119\n",
      "Data columns (total 9 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   A1        120 non-null    object\n",
      " 1   A2        120 non-null    object\n",
      " 2   B1        120 non-null    object\n",
      " 3   B2        120 non-null    object\n",
      " 4   C1        120 non-null    object\n",
      " 5   C2        120 non-null    object\n",
      " 6   modifier  120 non-null    object\n",
      " 7   model     120 non-null    object\n",
      " 8   topic     120 non-null    object\n",
      "dtypes: object(9)\n",
      "memory usage: 8.6+ KB\n"
     ]
    }
   ],
   "source": [
    "phis = [x for x in file_paths if \"phi\" in x]\n",
    "frames = []\n",
    "for frame in phis:\n",
    "    model = frame.split(\"/\")[-1].split(\"_\")[0]\n",
    "    topic = frame.split(\"/\")[-1].split(\"_\")[1].split(\".\")[0]\n",
    "    data = pd.read_parquet(frame)\n",
    "    data[\"model\"] = model\n",
    "    data[\"topic\"] = topic\n",
    "    frames.append(data)\n",
    "\n",
    "df_phi = pd.concat(frames).reset_index(drop=True)\n",
    "df_phi.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GPT-4o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 120 entries, 0 to 119\n",
      "Data columns (total 10 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   A1        120 non-null    object\n",
      " 1   A2        120 non-null    object\n",
      " 2   B1        120 non-null    object\n",
      " 3   B2        120 non-null    object\n",
      " 4   C1        120 non-null    object\n",
      " 5   C2        120 non-null    object\n",
      " 6   modifier  120 non-null    object\n",
      " 7   model     120 non-null    object\n",
      " 8   model_id  120 non-null    object\n",
      " 9   topic     120 non-null    object\n",
      "dtypes: object(10)\n",
      "memory usage: 9.5+ KB\n"
     ]
    }
   ],
   "source": [
    "gpt4os = [x for x in file_paths if \"gpt4o_\" in x]\n",
    "frames = []\n",
    "for frame in gpt4os:\n",
    "    topic = frame.split(\"/\")[-1].split(\"_\")[1].split(\".\")[0]\n",
    "    data = pd.read_parquet(frame)\n",
    "    data[\"topic\"] = topic\n",
    "    frames.append(data)\n",
    "\n",
    "df_gpt4o = pd.concat(frames).reset_index(drop=True)\n",
    "df_gpt4o.drop_duplicates(inplace=True)\n",
    "df_gpt4o.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GPT-4o-mini"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 120 entries, 0 to 119\n",
      "Data columns (total 10 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   A1        120 non-null    object\n",
      " 1   A2        120 non-null    object\n",
      " 2   B1        120 non-null    object\n",
      " 3   B2        120 non-null    object\n",
      " 4   C1        120 non-null    object\n",
      " 5   C2        120 non-null    object\n",
      " 6   modifier  120 non-null    object\n",
      " 7   model     120 non-null    object\n",
      " 8   model_id  120 non-null    object\n",
      " 9   topic     120 non-null    object\n",
      "dtypes: object(10)\n",
      "memory usage: 9.5+ KB\n"
     ]
    }
   ],
   "source": [
    "gpt4minis = [x for x in file_paths if \"gpt4o-mini\" in x]\n",
    "frames = []\n",
    "for frame in gpt4minis:\n",
    "    topic = frame.split(\"/\")[-1].split(\"_\")[1].split(\".\")[0]\n",
    "    data = pd.read_parquet(frame)\n",
    "    data[\"topic\"] = topic\n",
    "    frames.append(data)\n",
    "\n",
    "df_gpt4mini = pd.concat(frames).reset_index(drop=True)\n",
    "df_gpt4mini.drop_duplicates(inplace=True)\n",
    "df_gpt4mini.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Haiku"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 120 entries, 0 to 119\n",
      "Data columns (total 10 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   A1        120 non-null    object\n",
      " 1   A2        120 non-null    object\n",
      " 2   B1        120 non-null    object\n",
      " 3   B2        120 non-null    object\n",
      " 4   C1        120 non-null    object\n",
      " 5   C2        120 non-null    object\n",
      " 6   modifier  120 non-null    object\n",
      " 7   model     120 non-null    object\n",
      " 8   model_id  120 non-null    object\n",
      " 9   topic     120 non-null    object\n",
      "dtypes: object(10)\n",
      "memory usage: 9.5+ KB\n"
     ]
    }
   ],
   "source": [
    "haikus = [x for x in file_paths if \"haiku\" in x]\n",
    "frames = []\n",
    "for frame in haikus:\n",
    "    topic = frame.split(\"/\")[-1].split(\"_\")[1].split(\".\")[0]\n",
    "    data = pd.read_parquet(frame)\n",
    "    data[\"topic\"] = topic\n",
    "    frames.append(data)\n",
    "\n",
    "df_haiku = pd.concat(frames).reset_index(drop=True)\n",
    "df_haiku.drop_duplicates(inplace=True)\n",
    "df_haiku.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sonnet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 120 entries, 0 to 119\n",
      "Data columns (total 10 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   A1        120 non-null    object\n",
      " 1   A2        120 non-null    object\n",
      " 2   B1        120 non-null    object\n",
      " 3   B2        120 non-null    object\n",
      " 4   C1        120 non-null    object\n",
      " 5   C2        120 non-null    object\n",
      " 6   modifier  120 non-null    object\n",
      " 7   model     120 non-null    object\n",
      " 8   model_id  120 non-null    object\n",
      " 9   topic     120 non-null    object\n",
      "dtypes: object(10)\n",
      "memory usage: 9.5+ KB\n"
     ]
    }
   ],
   "source": [
    "sonnets = [x for x in file_paths if \"sonnet\" in x]\n",
    "frames = []\n",
    "for frame in sonnets:\n",
    "    topic = frame.split(\"/\")[-1].split(\"_\")[1].split(\".\")[0]\n",
    "    data = pd.read_parquet(frame)\n",
    "    data[\"topic\"] = topic\n",
    "    frames.append(data)\n",
    "\n",
    "df_sonnet = pd.concat(frames).reset_index(drop=True)\n",
    "df_sonnet.drop_duplicates(inplace=True)\n",
    "df_sonnet.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 720 entries, 0 to 719\n",
      "Data columns (total 8 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   A1      720 non-null    object\n",
      " 1   A2      720 non-null    object\n",
      " 2   B1      720 non-null    object\n",
      " 3   B2      720 non-null    object\n",
      " 4   C1      720 non-null    object\n",
      " 5   C2      720 non-null    object\n",
      " 6   model   720 non-null    object\n",
      " 7   topic   720 non-null    object\n",
      "dtypes: object(8)\n",
      "memory usage: 45.1+ KB\n"
     ]
    }
   ],
   "source": [
    "df = pd.concat([df_gemma, df_gpt4o, df_gpt4mini, df_haiku, df_sonnet, df_phi])\n",
    "df.reset_index(drop=True, inplace=True)\n",
    "df.drop(columns=[\"modifier\", \"model_id\"], inplace=True)\n",
    "df.to_parquet(\"_input/cefr_synthetic.parq\")\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "model\n",
       "gemma2it9b     120\n",
       "GPT-4o         120\n",
       "GPT-4o-mini    120\n",
       "Haiku          120\n",
       "Sonnet         120\n",
       "phi34kit       120\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.model.value_counts()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "std",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}