notebooks/generate_survey_questions.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3fe322b8",
   "metadata": {},
   "source": [
    "Survey Question Generator\n",
    "---\n",
    "Author: Peter Zhang\n",
    "\n",
    "This notebook generates a `.json` file of questions that is added to the Qualtrics survey. Make sure to run `create_dataset.py` first to create a dataset of MMLU questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e2f8834a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from os.path import join\n",
    "import random\n",
    "\n",
    "import pandas as pd\n",
    "import yaml\n",
    "\n",
    "import config"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "18ece34b",
   "metadata": {},
   "source": [
    "# Data\n",
    "\n",
    "Load the results of the 02/18/2023 scratchpad evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1505f452",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_folder = \"../data/model_output\"\n",
    "results_file = \"results_scratchpad_0218.csv\"\n",
    "results_df = pd.read_csv(join(results_folder, results_file))\n",
    "results_df = results_df[results_df[\"topic\"].isin(config.topics)] # subset to only target topics"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0da17450",
   "metadata": {},
   "source": [
    "## Questions\n",
    "\n",
    "We replace every single quote (`'`) with a backtick (`) to avoid a Javascript parse error when uploading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f4af3b39",
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = [\n",
    "    \"question\",\n",
    "    \"choice_A\",\n",
    "    \"choice_B\",\n",
    "    \"choice_C\",\n",
    "    \"choice_D\",\n",
    "    \"answer\",\n",
    "    \"justification\",\n",
    "    \"correct_answer\",\n",
    "    \"topic\"\n",
    "]\n",
    "for col in columns:\n",
    "    results_df[col] = results_df[col].str.replace(\"'\", \"`\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "95295032",
   "metadata": {},
   "source": [
    "We manually map the question indices because the original version involved manual ordering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "cf69f369",
   "metadata": {},
   "outputs": [],
   "source": [
    "mapping = json.load(open(\"survey_remap/mapping.json\"))\n",
    "qualtrics_json = {}\n",
    "for i, row in results_df.iterrows():\n",
    "    qid = mapping[str(i)]\n",
    "    qualtrics_json[qid] = list(row[columns])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a1d6f29d",
   "metadata": {},
   "source": [
    "We use three shuffles of the questions in the survey because Qualtrics shows questions in order of the index. This version is the **Phase 1** file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "332f607a",
   "metadata": {},
   "outputs": [],
   "source": [
    "phase1_qualtrics_file = \"../data/survey_questions/qualtrics_0413.json\"\n",
    "json.dump(qualtrics_json, open(phase1_qualtrics_file, \"w\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1146e9fb",
   "metadata": {},
   "source": [
    "For the **Phase 2** version, we switch the first and second halves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6912473c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# switch the first and second half of results dataframe\n",
    "results_df = pd.concat([results_df.iloc[results_df.shape[0]//2:], results_df.iloc[:results_df.shape[0]//2]])\n",
    "results_df.reset_index(drop=True, inplace=True)\n",
    "qualtrics_json = {}\n",
    "for i, row in results_df.iterrows():\n",
    "    qualtrics_json[str(i)] = list(row[columns])\n",
    "phase2_qualtrics_file = \"../data/survey_questions/qualtrics_0417.json\"\n",
    "json.dump(qualtrics_json, open(phase2_qualtrics_file, \"w\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2e2db0c2",
   "metadata": {},
   "source": [
    "Finally, for the **Phase 3** version, we perform a random shuffle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f3d5f321",
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(0)\n",
    "values = list(qualtrics_json.values())\n",
    "random.shuffle(values)\n",
    "questions_json = dict(zip(qualtrics_json.keys(), values))\n",
    "phase3_qualtrics_file = \"../data/survey_questions/qualtrics_0421.json\"\n",
    "json.dump(questions_json, open(phase3_qualtrics_file, \"w\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}