notebooks/01_dataset_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c1ed508f-c0e0-4f28-8e2c-a73f897cba53",
   "metadata": {},
   "source": [
    "# Exploring Temporal GNN Embeddings for Darknet Traffic Analysis\n",
    "## Preprocessing Stages\n",
    "___\n",
    "\n",
    "## Table of Contents\n",
    "1. Filter Traces\n",
    "2. Temporal GNN Preprocessing\n",
    "3. NLP preprocessing\n",
    "\n",
    "This notebook contains the main codes to (i) process raw traces filtering unwanted data; (ii) generate bipartite graphs from filtered traces and extract node features; (iii) generate textual corpora which will be processed by NLP algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "52ec6d94-69f2-4cc3-bd25-c825afae28dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from glob import glob \n",
    "import pandas as pd\n",
    "import json\n",
    "import sys\n",
    "sys.path.append('../')\n",
    "\n",
    "from src.preprocessing import *\n",
    "\n",
    "# Total snapshots of the collection\n",
    "TOT_DAYS = 31 \n",
    "# Drop source hosts sending less than FILTER packets per snapshot\n",
    "FILTER = 5 \n",
    "# Generate the corpora keeping the top TOP_PORTS daily ports +1 as languages\n",
    "TOP_PORTS = 2500 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77a16e68-9c24-4ca7-a6ad-ec9d11fdd811",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1. Filter Traces"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2179b3b-88fc-4061-b791-4ff47e953b6b",
   "metadata": {},
   "source": [
    "This code block represents a processing loop that iterates over different days' data. Each iteration follows these steps:\n",
    "\n",
    "1. **Loading Raw Traces:** Data for a specific day is loaded into a dataframe. The day's information is used to indicate progress.\n",
    "\n",
    "2. **Concatenate Dataframes:** The individual dataframes obtained for the day are combined into a single dataframe called `df`.\n",
    "\n",
    "3. **Packets Filtering:** The `apply_packets_filter` function is used to filter packets in the `df` dataframe based on a specified filter called `FILTER`. This step narrows down the data based on specific packet properties.\n",
    "\n",
    "4. **Ports Filtering:** The `apply_port_filter` function is applied to further refine the `df` dataframe based on the specified top ports `TOP_PORTS`. This step is especially useful when focusing on specific network port activity.\n",
    "\n",
    "5. **Appending Processed Data:** The processed dataframe `df` is appended to the `processed_df` list, creating a collection of dataframes for each day.\n",
    "\n",
    "6. **Saving Raw Dataset:** The `df` dataframe, containing processed data for the day, is saved to a CSV file named after the day. This step ensures that the processed data is stored for further analysis.\n",
    "\n",
    "7. **Finalizing the Iteration:** The iteration for the current day is complete, and the loop advances to the next day, if available.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "763d719c-4c13-42a7-9a39-0bc93137ad11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3160c618d02c414389ca28a7028bcd94",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Setting up:   0%|          | 0/775 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm_notebook as tqdm\n",
    "\n",
    "# Load Ground Truth and get the raw trace files list\n",
    "flist = glob(f'../data/traces/*.log.gz')\n",
    "\n",
    "# Initialize progress bar\n",
    "pbar = tqdm(total=len(flist)+TOT_DAYS, desc='Setting up')\n",
    "\n",
    "processed_df = []\n",
    "for i in range(TOT_DAYS):\n",
    "    dfs = []\n",
    "    for file in sorted(flist)[24*i:24*(i+1)]:\n",
    "        # Retrieve current day traces\n",
    "        day = file.split('trace_')[-1][:8]\n",
    "        df = load_single_file(file, day)\n",
    "        dfs.append(df)\n",
    "        \n",
    "        # Update progress bar\n",
    "        pbar.set_description(f'{day} Loading raw traces')\n",
    "        pbar.update(1)\n",
    "        \n",
    "    # Get daily dataframe\n",
    "    df = pd.concat(dfs, axis=0, ignore_index=True)\n",
    "    \n",
    "    # Packets filter\n",
    "    pbar.set_description(f'{day} Running packets filter')\n",
    "    df = apply_packets_filter(df, FILTER)\n",
    "    \n",
    "    # Ports filter\n",
    "    pbar.set_description(f'{day} Running port filter')\n",
    "    df = apply_port_filter(df, TOP_PORTS)\n",
    "    \n",
    "    processed_df.append(df)\n",
    "    \n",
    "    pbar.set_description(f'{day} Saving \"raw\" dataset')\n",
    "    df.to_csv(f'../data/raw/raw_{day}.csv', index=False)\n",
    "    \n",
    "    # Update progress bar\n",
    "    pbar.update(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44515fa5-00b1-47d8-96f7-c246eeeb8b44",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2. Temporal GNN preprocessing\n",
    "\n",
    "The following codes produce temporal bipartite graphs from darknet traces. Each graph is made of the source host node layer and destination TCP port node layer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceec308e-242c-4d62-848d-7f6c2f671090",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.1 Loading Filtered Traces and Ground Truth"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c5fc667-8e9d-48c7-a010-4efd1b5159be",
   "metadata": {
    "tags": []
   },
   "source": [
    "This code snippet focuses on loading and concatenating processed trace data. The following steps outline the process:\n",
    "\n",
    "1. **Loading Ground Truth Data:** The ground truth data is loaded from a CSV file located at `../data/ground_truth/ground_truth.csv`. This data is essential for comparison and evaluation purposes.\n",
    "\n",
    "2. **Loading Preprocessed Traces:** An empty list `raw_df` is initialized to store individual dataframes for each trace file. A loop iterates over a list of trace files found in the `../data/raw/` directory.\n",
    "\n",
    "3. **Loading and Appending:** Within the loop, each trace file is loaded as a dataframe using the `pd.read_csv` function. The loaded dataframe is appended to the `raw_df` list.\n",
    "\n",
    "4. **Combining Dataframes:** After loading all trace files, the individual dataframes stored in the `raw_df` list are combined into a single dataframe using the `pd.concat` function. The `ignore_index=True` parameter ensures that the index is reset in the final combined dataframe.\n",
    "\n",
    "5. **Finalizing Processed Data:** The combined dataframe, named `raw_df`, now contains the processed trace data from all the loaded files. It is ready for further analysis and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "324a9c30-4b61-46f5-a40c-41482f6bd4a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30de99b345d04920b52f4be10a6fa452",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading filtered traces:   0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm_notebook as tqdm\n",
    "from src.preprocessing.gnn import *\n",
    "\n",
    "# Load ground truth\n",
    "gt = pd.read_csv(f'../data/ground_truth/ground_truth.csv')\n",
    "\n",
    "# Load preprocessed traces\n",
    "raw_df = []\n",
    "for file in tqdm(sorted(glob(f'../data/raw/*')), desc='Loading filtered traces'):\n",
    "    raw_df.append(pd.read_csv(file))\n",
    "\n",
    "# Concatenate the loaded dataframes\n",
    "raw_df = pd.concat(raw_df, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbf41195-ab77-4026-bd96-8e038b5bc7f4",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.2 Generate graph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bae9a103-8cc5-426e-b630-8579c0dbb7ad",
   "metadata": {
    "tags": []
   },
   "source": [
    "This code segment involves the aggregation of edges and the generation of snapshots. It follows these steps:\n",
    "\n",
    "1. **Aggregating Edges and Merging Ground Truth:** Edges are aggregated, and the ground truth data is merged with the aggregated edges. The `dst_port` column in the dataframe is converted to string type. This process prepares the data for further manipulation.\n",
    "\n",
    "2. **Getting Distinct Nodes:** Unique source IP addresses (`src_ip`) and destination port numbers (`dst_port`) are extracted from the dataframe. These distinct nodes serve as identifiers for network nodes in the analysis.\n",
    "\n",
    "3. **Building Lookup Dictionaries:** Lookup dictionaries are constructed to map unique source IP addresses and destination port numbers to numerical identifiers. This mapping is essential for node representation in the analysis.\n",
    "\n",
    "4. **Converting Nodes to IDs:** The IP addresses and port numbers in the dataframe are replaced with their corresponding numerical identifiers using the lookup dictionaries. This conversion prepares the data for generating snapshots.\n",
    "\n",
    "5. **Generating Snapshots:** The dataframe is processed in snapshots, each corresponding to a specific time interval (`day`). For each interval, a snapshot is extracted, containing the relevant network data.\n",
    "\n",
    "6. **Saving Snapshots:** Each generated snapshot is saved as a text file in the `../data/graph/` directory. These snapshot files capture the network state at different intervals.\n",
    "\n",
    "7. **Saving Lookup Dictionaries:** The lookup dictionaries for IP addresses and port numbers are saved as JSON files in the `../data/graph/` directory. These dictionaries provide the mapping between numerical identifiers and their corresponding network nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "24b8fd61-59bb-443d-83ea-2e4feca66f3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c704bd16bbc4e5fb4fb4bb4d8ecca4c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Agregating edges:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbf0967766a8419e9714d1d6a0dfd7a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating snapshots:   0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm_notebook as tqdm\n",
    "\n",
    "# Initialize progress bar\n",
    "pbar = tqdm(total=4, desc='Agregating edges')\n",
    "\n",
    "# Aggregate edges and merge gt\n",
    "df = aggregate_edges(raw_df, gt)\n",
    "df['dst_port'] = df['dst_port'].astype(str)\n",
    "pbar.update(1)\n",
    "\n",
    "# Get distinct nodes\n",
    "pbar.set_description(f'Getting distinct nodes')\n",
    "unique_ips = df.src_ip.unique()\n",
    "unique_ports = df.dst_port.unique()\n",
    "pbar.update(1)\n",
    "\n",
    "# Build lookup dictionaries\n",
    "pbar.set_description(f'Building lookup dictionaries')\n",
    "ip_lookup = {ip:i for i,ip in enumerate(unique_ips)}\n",
    "port_lookup = {p:i+len(ip_lookup) for i,p in enumerate(unique_ports)}\n",
    "pbar.update(1)\n",
    "\n",
    "# Convert nodes to node IDs\n",
    "pbar.set_description(f'Converting nodes to IDs')\n",
    "df['src_ip'] = df['src_ip'].map(lambda x: ip_lookup[x])\n",
    "df['dst_port'] = df['dst_port'].map(lambda x: port_lookup[x])\n",
    "pbar.update(1)\n",
    "pbar.close()\n",
    "\n",
    "# Process each snapshot\n",
    "intervals = sorted(df.interval.unique())\n",
    "for day in tqdm(intervals, desc='Generating snapshots'):\n",
    "    # Extract current snapshot\n",
    "    snapshot = extract_single_snapshot(df, day)\n",
    "\n",
    "    # Save current snapshot\n",
    "    fname = f'../data/graph/{day}.txt'\n",
    "    with open(fname, 'w') as file:\n",
    "        file.write(snapshot)\n",
    "        \n",
    "# Save Host node lookup dictionary\n",
    "with open(f'../data/graph/ip_lookup.json', 'w') as file:\n",
    "    json.dump(ip_lookup, file)\n",
    "# Save Port node lookup dictionary\n",
    "with open(f'../data/graph/port_lookup.json', 'w') as file:\n",
    "    json.dump(port_lookup, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f8c02d-1a87-4cf6-b44b-673989961387",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.3 Extract node features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "065dae28-876f-4e46-b95b-6063cbcddece",
   "metadata": {},
   "source": [
    "This code snippet focuses on feature extraction and data saving. The following steps outline the process:\n",
    "\n",
    "1. **Loading Lookup Dictionaries:** The lookup dictionaries for IP addresses and port numbers are loaded from JSON files. These dictionaries contain mappings between numerical identifiers and network nodes.\n",
    "\n",
    "2. **Iterating Over Time Intervals:** The code iterates over the total intervals found in the `raw_df` dataframe, each corresponding to a specific time interval (`day`).\n",
    "\n",
    "3. **Extracting Current Snapshot:** A snapshot is extracted from the `raw_df` dataframe for the current time interval (`day`).\n",
    "\n",
    "4. **Extracting Node Features (IP Layer):** A series of functions are applied to extract various node features related to the IP layer. These functions include obtaining contacted destination ports, statistics per destination port, contacted destination IP addresses, statistics per destination IP address, and packet statistics based on source IP addresses. These features provide insights into the network activity at the IP level.\n",
    "\n",
    "5. **Uniform Features with IP Lookup:** The extracted IP-based node features are uniformized using the `ip_lookup` dictionary to map IP addresses to numerical identifiers.\n",
    "\n",
    "6. **Extracting Node Features (Port Layer):** Similar to the IP layer, node features for the port layer are extracted. Functions are applied to gather information about contacted source IP addresses, statistics per source IP address, contacted destination IP addresses (dummy values), statistics per destination IP address (dummy values), and packet statistics based on destination port numbers.\n",
    "\n",
    "7. **Uniform Features with Port Lookup:** The extracted port-based node features are uniformized using the `port_lookup` dictionary to map port numbers to numerical identifiers.\n",
    "\n",
    "8. **Concatenating Node Features:** The node features extracted from both IP and port layers are concatenated into a single dataframe named `features`. This dataframe captures the network characteristics for the current time interval (`day`).\n",
    "\n",
    "9. **Sorting and Saving Features:** The `features` dataframe is sorted by index and saved as a CSV file. The saved file captures the feature data for the specific time interval and layer. The destination path for the saved file is determined based on the value of `DSET_TYPE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0b90d561-7599-41ab-a32c-c1152ec6a23b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47a4bc83e573487c96b9d8a9a038a89c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm_notebook as tqdm\n",
    "\n",
    "# Load saved lookup dictionary for Host node layer\n",
    "with open(f'../data/graph/ip_lookup.json', 'r') as file:\n",
    "    ip_lookup = json.loads(file.read())\n",
    "# Load saved lookup dictionary for Port node layer\n",
    "with open(f'../data/graph/port_lookup.json', 'r') as file:\n",
    "    port_lookup = json.loads(file.read())\n",
    "\n",
    "tot_intervals = sorted(raw_df.interval.unique())\n",
    "for day in tqdm(tot_intervals):\n",
    "    # Extract current snapshot\n",
    "    snapshot = raw_df[raw_df.interval==day]\n",
    "    \n",
    "    # Extract node features -- Host layer\n",
    "    node_ip_features = [\n",
    "        get_contacted_dst_ports(snapshot),\n",
    "        get_stats_per_dst_port(snapshot),\n",
    "        get_contacted_dst_ips(snapshot),\n",
    "        get_stats_per_dst_ip(snapshot),\n",
    "        get_packet_statistics(snapshot, by='src_ip')\n",
    "    ]\n",
    "    node_ip_features = uniform_features(\n",
    "        node_ip_features, ip_lookup, 'src_ip'\n",
    "    )\n",
    "\n",
    "    # Extract node features -- Port layer\n",
    "    node_port_features = [\n",
    "        get_contacted_src_ips(snapshot),\n",
    "        get_stats_per_src_ip(snapshot),\n",
    "        get_contacted_dst_ips(snapshot, dummy=True),\n",
    "        get_stats_per_dst_ip(snapshot, dummy=True),\n",
    "        get_packet_statistics(snapshot, by='dst_port')\n",
    "    ]\n",
    "    node_port_features = uniform_features(\n",
    "        node_port_features, port_lookup, 'dst_port'\n",
    "    )\n",
    "    \n",
    "    # Concatenate the node features and save\n",
    "    features = pd.concat([node_ip_features, node_port_features]).sort_index()\n",
    "    features.to_csv(f'../data/features/features_{day}.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c01dc1d0-2a6c-47c0-9b66-8fd3033246ab",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3. NLP preprocessing\n",
    "\n",
    "The following codes start from preprocessed filtered traces and generate textual corpora which can be processed by NLP algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8838649a-0095-471f-a3d6-0c3bf0f52690",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3.1 Loading Filtered Traces"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3149efe3-86c5-4b0d-89d8-2f77f6a74d58",
   "metadata": {},
   "source": [
    "This code segment focuses on loading preprocessed traces and initializing data. The following steps outline the process:\n",
    "\n",
    "1. **Loading Preprocessed Traces:** The code checks if the `raw_df` variable is already initialized. If not, it initializes an empty list named `raw_df`. The code then iterates over a list of trace files found in the `../data/raw/` directory.\n",
    "\n",
    "2. **Loading and Concatenating:** Within the loop, each trace file is loaded as a dataframe using the `pd.read_csv` function. The loaded dataframes are appended to the `raw_df` list.\n",
    "\n",
    "3. **Combining Dataframes:** After loading all trace files, the individual dataframes stored in the `raw_df` list are combined into a single dataframe using the `pd.concat` function. The `ignore_index=True` parameter ensures that the index is reset in the final combined dataframe.\n",
    "\n",
    "4. **Data Initialization:** If the `raw_df` variable was previously uninitialized, it is now populated with the concatenated dataframe containing preprocessed trace data. This dataframe will serve as the basis for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9bd5576a-7275-421b-9c32-16eeaa02bbd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.notebook import tqdm_notebook as tqdm\n",
    "from src.preprocessing.nlp import *\n",
    "import pickle\n",
    "\n",
    "# Load preprocessed traces\n",
    "if type(raw_df) == type(None):\n",
    "    raw_df = []\n",
    "    for file in tqdm(sorted(glob(f'../data/raw/*'))):\n",
    "        raw_df.append(pd.read_csv(file))\n",
    "    \n",
    "    # Concatenate dataframes\n",
    "    raw_df = pd.concat(raw_df, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd11584-1d69-4ceb-8cd7-78bfea5d1b35",
   "metadata": {},
   "source": [
    "### 3.2 Corpus generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf0ad01d-5641-4242-9d7f-267828fa3d0f",
   "metadata": {},
   "source": [
    "This code segment focuses on generating a corpus and saving it to disk. The following steps outline the process:\n",
    "\n",
    "1. **Iterating Over Time Intervals:** The code iterates over the total intervals found in the `raw_df` dataframe, each corresponding to a specific time interval (`day`).\n",
    "\n",
    "2. **Extracting Current Snapshot and Sorting:** A snapshot is extracted from the `raw_df` dataframe for the current time interval (`day`). The snapshot is sorted by the `ts` column in ascending order.\n",
    "\n",
    "3. **Generating Corpus by Port:** A corpus is generated based on the `src_ip` values grouped by `dst_port`. The list of `dst_port` values is ordered based on the frequency of occurrence. The resulting corpus structure contains a list of `src_ip` values for each `dst_port`.\n",
    "\n",
    "4. **Moving \"Other\" Port to Bottom:** The `src_ip` lists for the top 2500 `dst_port` values are reordered. The remaining ones are associated with the \"other\" port and moved to the bottom of the corpus.\n",
    "\n",
    "5. **Flattening the Corpus:** The corpus is flattened into a single array of `src_ip` values across all `dst_port` values.\n",
    "\n",
    "6. **Removing Duplicates and Splitting:** Duplicate `src_ip` values are removed from the corpus. The corpus is then split into equally sized chunks, each containing a maximum of 1000 `src_ip` values.\n",
    "\n",
    "7. **Converting to List Format:** The corpus chunks are converted to list format for compatibility with pickle.\n",
    "\n",
    "8. **Saving the Corpus:** The generated corpus is saved as a pickle file named after the specific time interval (`day`). The pickle file is stored in the `../data/corpus/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8b5c2a5e-ac50-4e0d-8525-e5a064049ca4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54bdcc33e5c74268ac9468778ec9ba69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm_notebook as tqdm\n",
    "\n",
    "tot_intervals = sorted(raw_df.interval.unique())\n",
    "for day in tqdm(tot_intervals):\n",
    "    # Extract current snapshot\n",
    "    snapshot = raw_df[raw_df.interval==day].sort_values('ts')\n",
    "    \n",
    "    # Get the list of src IP by port\n",
    "    port_order = snapshot.value_counts('dst_port').index\n",
    "    corpus = snapshot.groupby('dst_port').agg({'src_ip':list}).reindex(port_order)\n",
    "\n",
    "    # Move port \"other\" at bottom\n",
    "    corpus = pd.concat([corpus.iloc[1:], corpus.iloc[:1]], ignore_index=True)\n",
    "    corpus = np.hstack([x for x in corpus.src_ip])\n",
    "\n",
    "    # Remove duplicates and split the corpus in equally sized chunks\n",
    "    corpus = drop_duplicates(corpus)\n",
    "    corpus = split_array(corpus, step=1000)\n",
    "    corpus = [list(x) for x in corpus]\n",
    "\n",
    "    # Save the corpus\n",
    "    with open(f'../data/corpus/corpus_{day}.pkl', 'wb') as file:\n",
    "        pickle.dump(corpus, file)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}