Assignment2_Part3_(ii).ipynb

{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# (ii) Network Performance by Hidden Layer Size\n",
    "\n",
    "| Trial No | Number of Nodes in Hidden Layer | MSE  |      |      |      | R²   |      |      |      |\n",
    "| -------- | ------------------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n",
    "|          |                                 | Trn  | Val  | Test | All  | Trn  | Val  | Test | All  |\n",
    "| A.       | 5                               | 0.9124 | 0.2714 | 0.5482 | 0.7090 | 0.9882 | 0.9960 | 0.9882 | 0.9898 |\n",
    "| B.       | 10                              | 0.6381 | 0.2405 | 0.3766 | 0.5047 | 0.9918 | 0.9965 | 0.9919 | 0.9928 |\n",
    "| C.       | 20                              | 0.6381 | 0.2405 | 0.3766 | 0.5047 | 0.9918 | 0.9965 | 0.9919 | 0.9928 |\n",
    "\n",
    "## Detailed Analysis\n",
    "\n",
    "### 1. Configuration Space Exploration\n",
    "\n",
    "My experiments covered a comprehensive set of configurations:\n",
    "- Hidden layer sizes: 5, 10, and 20 neurons\n",
    "- Learning rates: 0.001, 0.01, and 0.1\n",
    "- Batch sizes: 16, 32, and 64\n",
    "- Random seeds: 0, 42, and 123\n",
    "\n",
    "### 2. Performance Analysis by Hidden Layer Size\n",
    "\n",
    "#### a) 5 Neurons\n",
    "- Best performance achieved:\n",
    "  - Learning rate: 0.001\n",
    "  - Batch size: 64\n",
    "  - Seed: 0\n",
    "- Shows good performance but generally less accurate than larger networks\n",
    "- Advantages: Simplest architecture, fastest training\n",
    "- Disadvantages: Slightly higher error rates\n",
    "\n",
    "#### b) 10 Neurons\n",
    "- Significant improvement over 5 neurons\n",
    "- Multiple configurations achieved excellent results\n",
    "- Best configuration:\n",
    "  - Learning rate: 0.1\n",
    "  - Batch size: 32\n",
    "  - Seed: 123\n",
    "- Metrics:\n",
    "  - Training: MSE = 0.7606, R² = 0.9902\n",
    "  - Validation: MSE = 0.2995, R² = 0.9956\n",
    "  - Test: MSE = 0.4341, R² = 0.9907\n",
    "  - Overall: MSE = 0.6012, R² = 0.9914\n",
    "\n",
    "#### c) 20 Neurons\n",
    "- Similar performance to 10 neurons\n",
    "- Best configuration:\n",
    "  - Learning rate: 0.1\n",
    "  - Batch size: 32\n",
    "  - Seed: 123\n",
    "- Metrics:\n",
    "  - Training: MSE = 0.8223, R² = 0.9894\n",
    "  - Validation: MSE = 0.3914, R² = 0.9943\n",
    "  - Test: MSE = 0.2723, R² = 0.9941\n",
    "  - Overall: MSE = 0.6238, R² = 0.9911\n",
    "\n",
    "### 3. Best Model Selection\n",
    "\n",
    "After analyzing all configurations, the best overall model is:\n",
    "\n",
    "**Configuration:**\n",
    "- Hidden layer size: 20 neurons\n",
    "- Learning rate: 0.1\n",
    "- Batch size: 32\n",
    "- Random seed: 123\n",
    "\n",
    "**Performance:**\n",
    "- Achieved the best test R² (0.9941) and test MSE (0.2723)\n",
    "- Shows excellent consistency across datasets\n",
    "- Training: MSE = 0.8223, R² = 0.9894\n",
    "- Validation: MSE = 0.3914, R² = 0.9943\n",
    "- Test: MSE = 0.2723, R² = 0.9941\n",
    "- Overall: MSE = 0.6238, R² = 0.9911\n",
    "\n",
    "### 4. Key Findings\n",
    "\n",
    "1. **Learning Rate Impact:**\n",
    "   - Higher learning rates (0.1) often led to better performance when combined with larger batch sizes\n",
    "   - Lower learning rates (0.001) were more stable but sometimes converged to suboptimal solutions\n",
    "\n",
    "2. **Batch Size Effects:**\n",
    "   - Larger batch sizes (32, 64) generally produced more consistent results\n",
    "   - Best performances were often achieved with a batch size of 32\n",
    "\n",
    "3. **Architecture Complexity:**\n",
    "   - Increasing from 5 to 10 neurons showed significant improvement\n",
    "   - Further increase to 20 neurons provided marginal benefits in some configurations\n",
    "   - The 20-neuron model ultimately achieved the best test performance"
   ],
   "id": "91624c2181c42f1c"
  },
  {
   "cell_type": "code",
   "id": "initial_id",
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "end_time": "2024-10-05T16:41:00.837441Z",
     "start_time": "2024-10-05T12:01:04.283127Z"
    }
   },
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Input\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.metrics import r2_score\n",
    "import tensorflow as tf\n",
    "\n",
    "def create_and_train_model(X_train, X_val, X_test, y_train, y_val, y_test, \n",
    "                          hidden_neurons, learning_rate, batch_size, seed):\n",
    "    # Set random seed for reproducibility\n",
    "    tf.random.set_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    \n",
    "    # Create model with Input layer\n",
    "    model = Sequential()\n",
    "    model.add(Input(shape=(X_train.shape[1],)))  # This creates an input layer accepting all 14 features\n",
    "    model.add(Dense(hidden_neurons, activation='sigmoid'))\n",
    "    model.add(Dense(1, activation='linear'))\n",
    "    \n",
    "    # Compile model\n",
    "    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n",
    "    model.compile(optimizer=optimizer, loss='mse')\n",
    "    \n",
    "    # Early stopping\n",
    "    early_stopping = EarlyStopping(\n",
    "        monitor='val_loss',\n",
    "        patience=100,\n",
    "        restore_best_weights=True\n",
    "    )\n",
    "    \n",
    "    # Train model\n",
    "    history = model.fit(\n",
    "        X_train, y_train,\n",
    "        validation_data=(X_val, y_val),\n",
    "        epochs=20000,\n",
    "        batch_size=batch_size,\n",
    "        callbacks=[early_stopping],\n",
    "        verbose=0\n",
    "    )\n",
    "    \n",
    "    # Evaluate model\n",
    "    X_all = np.vstack([X_train, X_val, X_test])\n",
    "    y_all = np.concatenate([y_train, y_val, y_test])\n",
    "    \n",
    "    # Make predictions\n",
    "    y_pred_train = model.predict(X_train, verbose=0)\n",
    "    y_pred_val = model.predict(X_val, verbose=0)\n",
    "    y_pred_test = model.predict(X_test, verbose=0)\n",
    "    y_pred_all = model.predict(X_all, verbose=0)\n",
    "    \n",
    "    # Calculate MSE\n",
    "    mse_train = np.mean((y_train - y_pred_train.flatten()) ** 2)\n",
    "    mse_val = np.mean((y_val - y_pred_val.flatten()) ** 2)\n",
    "    mse_test = np.mean((y_test - y_pred_test.flatten()) ** 2)\n",
    "    mse_all = np.mean((y_all - y_pred_all.flatten()) ** 2)\n",
    "    \n",
    "    # Calculate R²\n",
    "    r2_train = r2_score(y_train, y_pred_train)\n",
    "    r2_val = r2_score(y_val, y_pred_val)\n",
    "    r2_test = r2_score(y_test, y_pred_test)\n",
    "    r2_all = r2_score(y_all, y_pred_all)\n",
    "    \n",
    "    return {\n",
    "        'mse': {'train': mse_train, 'val': mse_val, 'test': mse_test, 'all': mse_all},\n",
    "        'r2': {'train': r2_train, 'val': r2_val, 'test': r2_test, 'all': r2_all}\n",
    "    }\n",
    "\n",
    "# Load and prepare data\n",
    "def prepare_data():\n",
    "    # Load the data\n",
    "    df = pd.read_csv('Body_Fat.csv')\n",
    "    \n",
    "    # Separate features and target\n",
    "    X = df.drop('BodyFat', axis=1)\n",
    "    y = df['BodyFat']\n",
    "    \n",
    "    # Scale the features\n",
    "    scaler = MinMaxScaler()\n",
    "    X_scaled = scaler.fit_transform(X)\n",
    "    \n",
    "    # Split the data\n",
    "    X_train_val, X_test, y_train_val, y_test = train_test_split(\n",
    "        X_scaled, y, test_size=0.2, random_state=42\n",
    "    )\n",
    "    X_train, X_val, y_train, y_val = train_test_split(\n",
    "        X_train_val, y_train_val, test_size=0.25, random_state=42\n",
    "    )\n",
    "    \n",
    "    return X_train, X_val, X_test, y_train, y_val, y_test\n",
    "\n",
    "def run_experiments():\n",
    "    X_train, X_val, X_test, y_train, y_val, y_test = prepare_data()\n",
    "    \n",
    "    # Hyperparameters to try\n",
    "    hidden_neurons_options = [5, 10, 20]\n",
    "    learning_rates = [0.001, 0.01, 0.1]\n",
    "    batch_sizes = [16, 32, 64]\n",
    "    seeds = [0, 42, 123]\n",
    "    \n",
    "    results = []\n",
    "    \n",
    "    for neurons in hidden_neurons_options:\n",
    "        for lr in learning_rates:\n",
    "            for batch_size in batch_sizes:\n",
    "                for seed in seeds:\n",
    "                    result = create_and_train_model(\n",
    "                        X_train, X_val, X_test, y_train, y_val, y_test, \n",
    "                        neurons, lr, batch_size, seed\n",
    "                    )\n",
    "                    results.append({\n",
    "                        'neurons': neurons,\n",
    "                        'learning_rate': lr,\n",
    "                        'batch_size': batch_size,\n",
    "                        'seed': seed,\n",
    "                        **result\n",
    "                    })\n",
    "                    \n",
    "                    print(f\"Neurons: {neurons}, Learning Rate: {lr}, Batch Size: {batch_size}, Seed: {seed}\")\n",
    "                    print(f\"MSE - Train: {result['mse']['train']:.4f}, Val: {result['mse']['val']:.4f}, \"\n",
    "                          f\"Test: {result['mse']['test']:.4f}, All: {result['mse']['all']:.4f}\")\n",
    "                    print(f\"R² - Train: {result['r2']['train']:.4f}, Val: {result['r2']['val']:.4f}, \"\n",
    "                          f\"Test: {result['r2']['test']:.4f}, All: {result['r2']['all']:.4f}\")\n",
    "                    print(\"-\" * 80)\n",
    "    \n",
    "    # Find and print the best model based on test R²\n",
    "    best_result = max(results, key=lambda x: x['r2']['test'])\n",
    "    print(\"\\nBest Model Configuration:\")\n",
    "    print(f\"Neurons: {best_result['neurons']}\")\n",
    "    print(f\"Learning Rate: {best_result['learning_rate']}\")\n",
    "    print(f\"Batch Size: {best_result['batch_size']}\")\n",
    "    print(f\"Seed: {best_result['seed']}\")\n",
    "    print(f\"Test R²: {best_result['r2']['test']:.4f}\")\n",
    "    print(f\"Test MSE: {best_result['mse']['test']:.4f}\")\n",
    "    \n",
    "    return results, best_result\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    results, best_result = run_experiments()"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 1.1987, Val: 0.3188, Test: 0.6658, All: 0.9128\n",
      "R² - Train: 0.9845, Val: 0.9953, Test: 0.9857, All: 0.9869\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 1.0750, Val: 0.3454, Test: 0.4620, All: 0.8033\n",
      "R² - Train: 0.9861, Val: 0.9950, Test: 0.9901, All: 0.9885\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 1.1429, Val: 0.3626, Test: 0.5130, All: 0.8575\n",
      "R² - Train: 0.9853, Val: 0.9947, Test: 0.9890, All: 0.9877\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 1.2560, Val: 0.4044, Test: 0.4944, All: 0.9295\n",
      "R² - Train: 0.9838, Val: 0.9941, Test: 0.9894, All: 0.9867\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 1.3100, Val: 0.5140, Test: 0.9288, All: 1.0717\n",
      "R² - Train: 0.9831, Val: 0.9925, Test: 0.9800, All: 0.9846\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 1.1650, Val: 0.3487, Test: 0.4988, All: 0.8650\n",
      "R² - Train: 0.9850, Val: 0.9949, Test: 0.9893, All: 0.9876\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 0.9124, Val: 0.2714, Test: 0.5482, All: 0.7090\n",
      "R² - Train: 0.9882, Val: 0.9960, Test: 0.9882, All: 0.9898\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 1.2840, Val: 0.9607, Test: 0.8506, All: 1.1308\n",
      "R² - Train: 0.9834, Val: 0.9860, Test: 0.9817, All: 0.9838\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.001, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.2695, Val: 0.3220, Test: 0.4872, All: 0.9194\n",
      "R² - Train: 0.9836, Val: 0.9953, Test: 0.9895, All: 0.9868\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 0.9893, Val: 0.1997, Test: 0.5306, All: 0.7367\n",
      "R² - Train: 0.9872, Val: 0.9971, Test: 0.9886, All: 0.9894\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 1.2197, Val: 0.4658, Test: 0.4911, All: 0.9196\n",
      "R² - Train: 0.9843, Val: 0.9932, Test: 0.9894, All: 0.9868\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 1.0863, Val: 0.2741, Test: 0.6692, All: 0.8375\n",
      "R² - Train: 0.9860, Val: 0.9960, Test: 0.9856, All: 0.9880\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 1.0935, Val: 0.5545, Test: 0.9312, All: 0.9516\n",
      "R² - Train: 0.9859, Val: 0.9919, Test: 0.9800, All: 0.9864\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 0.7963, Val: 0.4571, Test: 1.2104, All: 0.8115\n",
      "R² - Train: 0.9897, Val: 0.9933, Test: 0.9740, All: 0.9884\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 1.2482, Val: 0.2791, Test: 0.9031, All: 0.9822\n",
      "R² - Train: 0.9839, Val: 0.9959, Test: 0.9806, All: 0.9859\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 1.1550, Val: 0.3417, Test: 0.4767, All: 0.8531\n",
      "R² - Train: 0.9851, Val: 0.9950, Test: 0.9898, All: 0.9878\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 1.6858, Val: 1.2274, Test: 1.2554, All: 1.5059\n",
      "R² - Train: 0.9783, Val: 0.9821, Test: 0.9730, All: 0.9784\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.01, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.1566, Val: 0.3901, Test: 0.6033, All: 0.8895\n",
      "R² - Train: 0.9851, Val: 0.9943, Test: 0.9870, All: 0.9872\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 1.8250, Val: 0.9235, Test: 0.9392, All: 1.4633\n",
      "R² - Train: 0.9765, Val: 0.9865, Test: 0.9798, All: 0.9790\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 1.3300, Val: 0.7596, Test: 2.4407, All: 1.4393\n",
      "R² - Train: 0.9829, Val: 0.9889, Test: 0.9475, All: 0.9794\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 2.0834, Val: 0.6568, Test: 0.7197, All: 1.5187\n",
      "R² - Train: 0.9731, Val: 0.9904, Test: 0.9845, All: 0.9782\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 1.8210, Val: 0.7916, Test: 1.0234, All: 1.4512\n",
      "R² - Train: 0.9765, Val: 0.9884, Test: 0.9780, All: 0.9792\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 2.9473, Val: 1.7145, Test: 1.7374, All: 2.4530\n",
      "R² - Train: 0.9620, Val: 0.9750, Test: 0.9627, All: 0.9648\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 0.8252, Val: 0.5557, Test: 1.5717, All: 0.9217\n",
      "R² - Train: 0.9894, Val: 0.9919, Test: 0.9662, All: 0.9868\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 2.0670, Val: 0.9568, Test: 1.1153, All: 1.6497\n",
      "R² - Train: 0.9734, Val: 0.9860, Test: 0.9760, All: 0.9764\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 1.9980, Val: 1.0775, Test: 1.2562, All: 1.6616\n",
      "R² - Train: 0.9742, Val: 0.9843, Test: 0.9730, All: 0.9762\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 5, Learning Rate: 0.1, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.6000, Val: 0.6567, Test: 0.5662, All: 1.1999\n",
      "R² - Train: 0.9794, Val: 0.9904, Test: 0.9878, All: 0.9828\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 1.0102, Val: 0.3204, Test: 0.4458, All: 0.7564\n",
      "R² - Train: 0.9870, Val: 0.9953, Test: 0.9904, All: 0.9892\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 0.9455, Val: 0.2657, Test: 0.4394, All: 0.7055\n",
      "R² - Train: 0.9878, Val: 0.9961, Test: 0.9906, All: 0.9899\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 1.1431, Val: 0.3735, Test: 0.6099, All: 0.8794\n",
      "R² - Train: 0.9853, Val: 0.9945, Test: 0.9869, All: 0.9874\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 0.7446, Val: 0.3071, Test: 0.6961, All: 0.6462\n",
      "R² - Train: 0.9904, Val: 0.9955, Test: 0.9850, All: 0.9907\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 0.7842, Val: 0.3012, Test: 0.4833, All: 0.6255\n",
      "R² - Train: 0.9899, Val: 0.9956, Test: 0.9896, All: 0.9910\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 1.2150, Val: 0.3851, Test: 0.5195, All: 0.9063\n",
      "R² - Train: 0.9843, Val: 0.9944, Test: 0.9888, All: 0.9870\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 0.8078, Val: 0.2501, Test: 0.6823, All: 0.6695\n",
      "R² - Train: 0.9896, Val: 0.9963, Test: 0.9853, All: 0.9904\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 0.8531, Val: 0.6835, Test: 1.5022, All: 0.9502\n",
      "R² - Train: 0.9890, Val: 0.9900, Test: 0.9677, All: 0.9864\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.001, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.2528, Val: 0.4063, Test: 0.5298, All: 0.9352\n",
      "R² - Train: 0.9838, Val: 0.9941, Test: 0.9886, All: 0.9866\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 1.0427, Val: 0.2390, Test: 0.5467, All: 0.7797\n",
      "R² - Train: 0.9866, Val: 0.9965, Test: 0.9882, All: 0.9888\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 0.9443, Val: 0.2287, Test: 0.7422, All: 0.7586\n",
      "R² - Train: 0.9878, Val: 0.9967, Test: 0.9840, All: 0.9891\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 1.1389, Val: 0.2645, Test: 0.5540, All: 0.8436\n",
      "R² - Train: 0.9853, Val: 0.9961, Test: 0.9881, All: 0.9879\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 1.0581, Val: 0.2841, Test: 0.4353, All: 0.7754\n",
      "R² - Train: 0.9864, Val: 0.9959, Test: 0.9906, All: 0.9889\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 0.6961, Val: 0.3243, Test: 0.8400, All: 0.6500\n",
      "R² - Train: 0.9910, Val: 0.9953, Test: 0.9819, All: 0.9907\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 0.8582, Val: 0.1913, Test: 0.6566, All: 0.6825\n",
      "R² - Train: 0.9889, Val: 0.9972, Test: 0.9859, All: 0.9902\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 1.0499, Val: 0.3073, Test: 0.4467, All: 0.7776\n",
      "R² - Train: 0.9865, Val: 0.9955, Test: 0.9904, All: 0.9889\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 0.8086, Val: 0.4833, Test: 1.0488, All: 0.7914\n",
      "R² - Train: 0.9896, Val: 0.9929, Test: 0.9775, All: 0.9887\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.01, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.0028, Val: 0.2069, Test: 0.3181, All: 0.7031\n",
      "R² - Train: 0.9871, Val: 0.9970, Test: 0.9932, All: 0.9899\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 1.1128, Val: 0.4649, Test: 0.6837, All: 0.8948\n",
      "R² - Train: 0.9857, Val: 0.9932, Test: 0.9853, All: 0.9872\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 1.2873, Val: 0.9644, Test: 0.7686, All: 1.1170\n",
      "R² - Train: 0.9834, Val: 0.9859, Test: 0.9835, All: 0.9840\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 0.7507, Val: 0.4346, Test: 0.7921, All: 0.6951\n",
      "R² - Train: 0.9903, Val: 0.9937, Test: 0.9830, All: 0.9900\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 1.3810, Val: 0.4558, Test: 0.5633, All: 1.0283\n",
      "R² - Train: 0.9822, Val: 0.9933, Test: 0.9879, All: 0.9853\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 0.9031, Val: 0.6617, Test: 2.4413, All: 1.1656\n",
      "R² - Train: 0.9884, Val: 0.9903, Test: 0.9475, All: 0.9833\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 0.7606, Val: 0.2995, Test: 0.4341, All: 0.6012\n",
      "R² - Train: 0.9902, Val: 0.9956, Test: 0.9907, All: 0.9914\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 0.8390, Val: 0.3638, Test: 0.3408, All: 0.6420\n",
      "R² - Train: 0.9892, Val: 0.9947, Test: 0.9927, All: 0.9908\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 0.7650, Val: 0.5730, Test: 0.5666, All: 0.6860\n",
      "R² - Train: 0.9901, Val: 0.9916, Test: 0.9878, All: 0.9902\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 10, Learning Rate: 0.1, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.0438, Val: 0.3098, Test: 0.7521, All: 0.8362\n",
      "R² - Train: 0.9865, Val: 0.9955, Test: 0.9838, All: 0.9880\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 1.1003, Val: 0.3388, Test: 0.5264, All: 0.8300\n",
      "R² - Train: 0.9858, Val: 0.9951, Test: 0.9887, All: 0.9881\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 1.1503, Val: 0.4655, Test: 0.5757, All: 0.8954\n",
      "R² - Train: 0.9852, Val: 0.9932, Test: 0.9876, All: 0.9872\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 1.1414, Val: 0.4347, Test: 0.5354, All: 0.8757\n",
      "R² - Train: 0.9853, Val: 0.9936, Test: 0.9885, All: 0.9874\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 0.7806, Val: 0.3486, Test: 0.3862, All: 0.6134\n",
      "R² - Train: 0.9899, Val: 0.9949, Test: 0.9917, All: 0.9912\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 0.6978, Val: 0.3204, Test: 0.4389, All: 0.5690\n",
      "R² - Train: 0.9910, Val: 0.9953, Test: 0.9906, All: 0.9918\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 0.7106, Val: 0.2784, Test: 0.5357, All: 0.5877\n",
      "R² - Train: 0.9908, Val: 0.9959, Test: 0.9885, All: 0.9916\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 0.6381, Val: 0.2405, Test: 0.3766, All: 0.5047\n",
      "R² - Train: 0.9918, Val: 0.9965, Test: 0.9919, All: 0.9928\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 0.8422, Val: 0.4980, Test: 0.5642, All: 0.7163\n",
      "R² - Train: 0.9891, Val: 0.9927, Test: 0.9879, All: 0.9897\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.001, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.0699, Val: 0.2981, Test: 0.7223, All: 0.8433\n",
      "R² - Train: 0.9862, Val: 0.9956, Test: 0.9845, All: 0.9879\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 0.8548, Val: 0.2492, Test: 0.5924, All: 0.6791\n",
      "R² - Train: 0.9890, Val: 0.9964, Test: 0.9873, All: 0.9903\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 0.9221, Val: 0.2674, Test: 0.8479, All: 0.7746\n",
      "R² - Train: 0.9881, Val: 0.9961, Test: 0.9818, All: 0.9889\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 0.6602, Val: 0.2791, Test: 0.6232, All: 0.5756\n",
      "R² - Train: 0.9915, Val: 0.9959, Test: 0.9866, All: 0.9917\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 0.8444, Val: 0.1955, Test: 0.6149, All: 0.6666\n",
      "R² - Train: 0.9891, Val: 0.9971, Test: 0.9868, All: 0.9904\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 0.7796, Val: 0.2390, Test: 0.4254, All: 0.5985\n",
      "R² - Train: 0.9899, Val: 0.9965, Test: 0.9909, All: 0.9914\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 0.8022, Val: 0.2342, Test: 0.7399, All: 0.6746\n",
      "R² - Train: 0.9897, Val: 0.9966, Test: 0.9841, All: 0.9903\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 0.8499, Val: 0.2604, Test: 0.4870, All: 0.6572\n",
      "R² - Train: 0.9890, Val: 0.9962, Test: 0.9895, All: 0.9906\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 0.7177, Val: 0.4916, Test: 0.4901, All: 0.6259\n",
      "R² - Train: 0.9907, Val: 0.9928, Test: 0.9895, All: 0.9910\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.01, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 0.8867, Val: 0.1940, Test: 0.3797, All: 0.6439\n",
      "R² - Train: 0.9886, Val: 0.9972, Test: 0.9918, All: 0.9908\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 16, Seed: 0\n",
      "MSE - Train: 0.7498, Val: 0.4291, Test: 0.8638, All: 0.7079\n",
      "R² - Train: 0.9903, Val: 0.9937, Test: 0.9814, All: 0.9899\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 16, Seed: 42\n",
      "MSE - Train: 1.3475, Val: 0.4563, Test: 0.4972, All: 0.9951\n",
      "R² - Train: 0.9826, Val: 0.9933, Test: 0.9893, All: 0.9857\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 16, Seed: 123\n",
      "MSE - Train: 0.4999, Val: 0.3922, Test: 0.5596, All: 0.4902\n",
      "R² - Train: 0.9936, Val: 0.9943, Test: 0.9880, All: 0.9930\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 32, Seed: 0\n",
      "MSE - Train: 1.0214, Val: 0.3456, Test: 0.5629, All: 0.7919\n",
      "R² - Train: 0.9868, Val: 0.9950, Test: 0.9879, All: 0.9886\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 32, Seed: 42\n",
      "MSE - Train: 1.4775, Val: 0.4697, Test: 1.5788, All: 1.2941\n",
      "R² - Train: 0.9810, Val: 0.9931, Test: 0.9661, All: 0.9814\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 32, Seed: 123\n",
      "MSE - Train: 0.8223, Val: 0.3914, Test: 0.2723, All: 0.6238\n",
      "R² - Train: 0.9894, Val: 0.9943, Test: 0.9941, All: 0.9911\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 64, Seed: 0\n",
      "MSE - Train: 0.9063, Val: 0.2415, Test: 0.6823, All: 0.7264\n",
      "R² - Train: 0.9883, Val: 0.9965, Test: 0.9853, All: 0.9896\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 64, Seed: 42\n",
      "MSE - Train: 0.6423, Val: 0.5924, Test: 0.4701, All: 0.5973\n",
      "R² - Train: 0.9917, Val: 0.9913, Test: 0.9899, All: 0.9914\n",
      "--------------------------------------------------------------------------------\n",
      "Neurons: 20, Learning Rate: 0.1, Batch Size: 64, Seed: 123\n",
      "MSE - Train: 1.2757, Val: 0.3521, Test: 0.7944, All: 0.9914\n",
      "R² - Train: 0.9836, Val: 0.9949, Test: 0.9829, All: 0.9858\n",
      "--------------------------------------------------------------------------------\n",
      "\n",
      "Best Model Configuration:\n",
      "Neurons: 20\n",
      "Learning Rate: 0.1\n",
      "Batch Size: 32\n",
      "Seed: 123\n",
      "Test R²: 0.9941\n",
      "Test MSE: 0.2723\n"
     ]
    }
   ],
   "execution_count": 1
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}