v1 (5).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3a94c6ec-8b68-4fa3-94b0-cb4e93d99d24",
   "metadata": {},
   "source": [
    "# Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2814d8bc-e97b-4c40-b751-198fee8cab0a",
   "metadata": {},
   "source": [
    "1) Convert 'Date' to datetime format.\n",
    "2) Set 'Date' as the index.\n",
    "3) Check for missing values."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2f1afba-5992-4584-bcb5-d5747b846b08",
   "metadata": {},
   "source": [
    "###\n",
    "Next steps in preprocessing might include:\n",
    "1) Normalizing or Scaling the Data: Especially relevant for machine learning models like LSTM and XGBoost to ensure all features contribute equally.\n",
    "2) Creating Lag Features: Useful for time series forecasting, particularly for machine learning models.\n",
    "3) Partitioning the Data: Splitting the data into training and testing sets based on the specified dates."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ff295a6-4635-470f-8abb-b898769dbb4b",
   "metadata": {},
   "source": [
    "###\n",
    "The steps you've outlined are a good start for preprocessing the data, but there are \n",
    "some additional steps we could take to further prepare the data for your analysis, \n",
    "especially given the use of sophisticated models like LSTM and XGBoost. Here’s an enhanced \n",
    "preprocessing script that includes normalization and additional feature engineering to potentially improve model performance:\n",
    "\n",
    " 1) Data Normalization - Standardize the stock prices and volumes.\n",
    " 2) Feature Engineering - Add technical indicators and time features.\n",
    " 3) Train-Test Split - Clearly split the data according to your specified dates.\n",
    "\n",
    " Here's how you can implement these steps:\n",
    "\n",
    " Adding technical indicators like SMA, EMA, RSI, and MACD to the data depends on your specific analysis and modeling goals. Here's why these indicators might be important:\r\n",
    " \n",
    "\r\n",
    "SMA (Simple Moving Average):\r\n",
    "Purpose: To smooth out price data and identify trends over a specified period.\r\n",
    "Use Case: It's useful for identifying long-term trends and making investment decisi\n",
    "\n",
    "ons.\r\n",
    "EMA (Exponential Moving Average):\r\n",
    "Purpose: Similar to SMA, but gives more weight to recent data points.\r\n",
    "Use Case: Helps in tracking more current price trends and short-term rev\n",
    "\n",
    "\n",
    "rsals.\r\n",
    "RSI (Relative Strength Index):\r\n",
    "Purpose: Measures the speed and change of price movements.\r\n",
    "Use Case: Indicates overbought or oversold conditions, which can signal potential \n",
    "\n",
    "reversals.\r\n",
    "MACD (Moving Average Convergence Divergence):\r\n",
    "Purpose: Shows the relationship between two moving averages and identifies changes in the trend.\r\n",
    "Use Case: Useful for spotting trend reversals and momentum shifts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91123ec0-b2b5-4d14-9c90-01db2e594e90",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install plotly pandas numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31c7da4d-eeab-4019-988a-fb531629dc06",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import talib\n",
    "import logging\n",
    "import plotly.io as pio\n",
    "import plotly.offline as py\n",
    "import plotly.graph_objs as go\n",
    "from IPython.display import display, HTML\n",
    "from plotly.subplots import make_subplots\n",
    "\n",
    "\n",
    "# Setting up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname=s - %(message=s')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Downloads stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        data['Date'] = pd.to_datetime(data['Date'])\n",
    "        data.set_index('Date', inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def preprocess_data(data: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Preprocesses the stock data by adding technical indicators.\"\"\"\n",
    "    logging.info(\"Starting preprocessing of data\")\n",
    "    \n",
    "    # Adding technical indicators without normalizing\n",
    "    data['SMA_10'] = talib.SMA(data['Close'], timeperiod=10)\n",
    "    data['EMA_10'] = talib.EMA(data['Close'], timeperiod=10)\n",
    "    data['RSI'] = talib.RSI(data['Close'], timeperiod=14)\n",
    "    data['MACD'], data['MACD_signal'], _ = talib.MACD(data['Close'])\n",
    "    data.fillna(method='bfill', inplace=True)\n",
    "    logging.info(\"Data preprocessing complete\")\n",
    "    return data\n",
    "\n",
    "def main() -> pd.DataFrame:\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    return preprocessed_data\n",
    "\n",
    "# Run main and get the preprocessed data\n",
    "preprocessed_data = main()\n",
    "\n",
    "# Displaying the first 5 rows with a title\n",
    "display(HTML('<h2>First 5 Rows of the DataFrame</h2>'))\n",
    "display(HTML(preprocessed_data.head(5).to_html()))\n",
    "# Displaying the last 5 rows with a title\n",
    "display(HTML('<h2>Last 5 Rows of the DataFrame</h2>'))\n",
    "display(HTML(preprocessed_data.tail(5).to_html()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a501e9c-a7a4-41ba-a686-cf46b58d52e1",
   "metadata": {},
   "source": [
    "# Data aggregation\n",
    "\n",
    "convert daily data into monthly data with some aggregation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65397f9b-96f0-483b-b585-35616a2e0dcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def aggregate_to_monthly_average(data: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Aggregates daily data into monthly averages.\"\"\"\n",
    "    try:\n",
    "        logging.info(\"Starting the aggregation of data to monthly averages.\")\n",
    "        monthly_data_avg = data.resample('M').mean()\n",
    "        logging.info(\"Data successfully aggregated to monthly averages.\")\n",
    "        return monthly_data_avg\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to aggregate data: {e}\")\n",
    "        raise\n",
    "\n",
    "def plot_trend_comparison(daily_data: pd.DataFrame, monthly_data: pd.DataFrame, title=\"Daily vs. Monthly Trend Comparison\") -> None:\n",
    "    \"\"\"Plots the daily trend and monthly average trend side-by-side using Plotly.\"\"\"\n",
    "    try:\n",
    "        logging.info(\"Starting to plot the trend comparison.\")\n",
    "        fig = make_subplots(rows=1, cols=2, subplot_titles=('Daily Trend', 'Monthly Average Trend'))\n",
    "        \n",
    "        # Daily plot\n",
    "        daily_trace = go.Scatter(x=daily_data.index, y=daily_data['Close'], mode='lines', name='Daily Close', line=dict(color='blue'))\n",
    "        fig.add_trace(daily_trace, row=1, col=1)\n",
    "        \n",
    "        # Monthly plot\n",
    "        monthly_trace = go.Scatter(x=monthly_data.index, y=monthly_data['Close'], mode='lines', name='Monthly Close', line=dict(color='red'))\n",
    "        fig.add_trace(monthly_trace, row=1, col=2)\n",
    "        \n",
    "        # Layout\n",
    "        fig.update_layout(title_text=title, hovermode='x', showlegend=False)\n",
    "        \n",
    "        pio.show(fig)  # Display the plot inline\n",
    "        logging.info(\"Successfully plotted the trend comparison.\")\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to plot the trend comparison: {e}\")\n",
    "        raise\n",
    "\n",
    "# Aggregate and plot the data\n",
    "monthly_data = aggregate_to_monthly_average(preprocessed_data)\n",
    "plot_trend_comparison(preprocessed_data, monthly_data)\n",
    "\n",
    "\n",
    "#print(monthly_data.head(5))\n",
    "# Displaying the last 5 rows with a title\n",
    "display(HTML('<h2>Last 5 Rows of the DataFrame</h2>'))\n",
    "display(HTML(monthly_data.tail(5).to_html()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ca29b1d-d350-4dcf-b3bb-19ff493990e1",
   "metadata": {},
   "source": [
    "# convert to stationary to implement model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57d0ad49-1dfc-43b7-8704-778f8ed6977a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.tsa.stattools import adfuller\n",
    "\n",
    "def test_stationarity(timeseries: pd.Series) -> None:\n",
    "    \"\"\"Tests and logs the stationarity of the provided timeseries.\"\"\"\n",
    "    logging.info(\"Testing the stationarity of the timeseries\")\n",
    "    dftest = adfuller(timeseries, autolag='AIC')\n",
    "    dfoutput = pd.Series(dftest[:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])\n",
    "    for key,value in dftest[4].items():\n",
    "        dfoutput['Critical Value (%s)' % key] = value\n",
    "    logging.info(\"Results of Dickey-Fuller Test:\")\n",
    "    logging.info(dfoutput)\n",
    "\n",
    "def make_stationary(data: pd.DataFrame, target_column: str) -> pd.DataFrame:\n",
    "    \"\"\"Transforms the target column to make the data stationary.\"\"\"\n",
    "    logging.info(\"Making data stationary\")\n",
    "    \n",
    "    # Apply log transformation\n",
    "    data[f'{target_column}_log'] = np.log(data[target_column])\n",
    "    \n",
    "    # Apply differencing\n",
    "    data[f'{target_column}_stationary'] = data[f'{target_column}_log'].diff().dropna()\n",
    "\n",
    "    # Test stationarity\n",
    "    test_stationarity(data[f'{target_column}_stationary'].dropna())\n",
    "\n",
    "    return data\n",
    "\n",
    "# Example usage\n",
    "def main():\n",
    "    # Assuming preprocessed_data is already loaded from previous steps\n",
    "    target_column = 'Close'  # Define which column to make stationary\n",
    "    preprocessed_data.dropna(inplace=True)  # Drop NaN values for stationarity testing\n",
    "    stationary_data = make_stationary(preprocessed_data, target_column)\n",
    "    \n",
    "    logging.info(\"Stationarity transformation complete\")\n",
    "    return stationary_data\n",
    "\n",
    "# Run main to perform stationarity transformation\n",
    "stationary_data = main()\n",
    "\n",
    "\n",
    "# Print the stationary data\n",
    "#print(stationary_data)\n",
    "\n",
    "# Displaying the last 5 rows with a title\n",
    "display(HTML('<h2>Last 5 Rows of the DataFrame</h2>'))\n",
    "display(HTML(stationary_data.head(5).to_html()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "574d4b21-e047-48d9-9b05-754a0c26ac25",
   "metadata": {},
   "source": [
    "###\n",
    "Explanation:\n",
    "test_stationarity Function: Uses the Augmented Dickey-Fuller (ADF) test to check for stationarity and logs the result.\n",
    "make_stationary Function: Applies log transformation and differencing to make the data stationary.\n",
    "Advanced Code: Incorporates logging, clean structure, and leverages standard libraries for ADF testing and transformations.\n",
    "\n",
    "\n",
    "After converting the data into stationary the total rows is 1233 before converting total rows is 1235?\r\n",
    "Whatares the reason\n",
    "\n",
    "The difference in the total number of rows before and after making the data stationary might be due to the removal of NaN values during the stationary transformation process.?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffa14686-a190-4a1d-a66b-19c134d47213",
   "metadata": {},
   "source": [
    "# 1)  ARIMA model Implementation - Univariate model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8293647-d517-4ded-8645-e9d2c62eda35",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Second test with forecast the 12 months data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ac90291-14f5-4519-854f-fa2388a3aea2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import logging\n",
    "from statsmodels.tsa.statespace.sarimax import SARIMAX\n",
    "from plotly.subplots import make_subplots\n",
    "import plotly.graph_objs as go\n",
    "from statsmodels.tsa.stattools import acf, pacf\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# Setting up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname=s - %(message=s')\n",
    "\n",
    "# Function to split data into training and testing datasets\n",
    "def split_data(data: pd.DataFrame, start_train: str, end_train: str, start_test: str, end_test: str) -> (pd.DataFrame, pd.DataFrame):\n",
    "    \"\"\"Splits the data into training and testing datasets.\"\"\"\n",
    "    train_data = data[start_train:end_train]\n",
    "    test_data = data[start_test:end_test]\n",
    "    return train_data, test_data\n",
    "\n",
    "# Function to implement SARIMAX model and return fitted model\n",
    "def train_sarimax(train_data: pd.Series, p: int, d: int, q: int, seasonal_order=(0, 0, 0, 0)):\n",
    "    \"\"\"Trains the SARIMAX model and returns the fitted model.\"\"\"\n",
    "    model = SARIMAX(train_data, order=(p, d, q), seasonal_order=seasonal_order)\n",
    "    model_fit = model.fit()\n",
    "    return model_fit\n",
    "\n",
    "# Function to visualize the train/test/forecast results with Plotly\n",
    "def plot_train_test_forecast(train_data: pd.Series, test_data: pd.Series, forecast: pd.Series) -> None:\n",
    "    \"\"\"Visualizes the train, test, and forecast data using Plotly.\"\"\"\n",
    "    fig = make_subplots(rows=1, cols=1)\n",
    "    \n",
    "    # Add traces for training, testing, and forecasted data\n",
    "    fig.add_trace(go.Scatter(x=train_data.index, y=train_data, mode='lines', name='Train Data'))\n",
    "    fig.add_trace(go.Scatter(x=test_data.index, y=test_data, mode='lines', name='Test Data'))\n",
    "    fig.add_trace(go.Scatter(x=forecast.index, y=forecast, mode='lines', name='Forecast'))\n",
    "    \n",
    "    # Set plot layout\n",
    "    fig.update_layout(\n",
    "        title='Train, Test and Forecast Data',\n",
    "        xaxis_title='Date',\n",
    "        yaxis_title='Stock Price',\n",
    "        hovermode='x',\n",
    "        showlegend=True\n",
    "    )\n",
    "    \n",
    "    fig.show()\n",
    "\n",
    "# Function to plot ACF and PACF with Plotly\n",
    "def plot_acf_pacf(data: pd.Series) -> None:\n",
    "    \"\"\"Plots the ACF and PACF using Plotly.\"\"\"\n",
    "    acf_values = acf(data, nlags=40)\n",
    "    pacf_values = pacf(data, nlags=40)\n",
    "    \n",
    "    fig = make_subplots(rows=1, cols=2, subplot_titles=('ACF', 'PACF'))\n",
    "\n",
    "    fig.add_trace(go.Bar(x=np.arange(len(acf_values)), y=acf_values, name='ACF'), row=1, col=1)\n",
    "    fig.add_trace(go.Bar(x=np.arange(len(pacf_values)), y=pacf_values, name='PACF'), row=1, col=2)\n",
    "\n",
    "    fig.update_layout(title_text='ACF and PACF Plots', showlegend=False)\n",
    "    fig.show()\n",
    "\n",
    "# Main function for SARIMAX implementation and forecasting\n",
    "def main():\n",
    "    # Assuming `stationary_data` is already loaded from previous steps\n",
    "    target_column = 'Close_stationary'\n",
    "    train_start, train_end = '01-04-2019', '31-03-2023'\n",
    "    test_start, test_end = '01-04-2023', '31-03-2024'\n",
    "    forecast_start, forecast_end = '01-04-2024', '31-03-2025'\n",
    "\n",
    "    train_data, test_data = split_data(stationary_data[target_column].dropna(), train_start, train_end, test_start, test_end)\n",
    "    \n",
    "    # Plot ACF and PACF plots for SARIMAX parameter selection\n",
    "    plot_acf_pacf(train_data)\n",
    "    \n",
    "    # Train SARIMAX model with assumed p, d, q values and seasonal_order (these should be fine-tuned)\n",
    "    p, d, q, seasonal_order = 1, 1, 1, (1, 1, 1, 12)\n",
    "    model_fit = train_sarimax(train_data, p, d, q, seasonal_order)\n",
    "    \n",
    "    # Forecast for the next 12 months\n",
    "    forecast_steps = pd.date_range(start=forecast_start, end=forecast_end, freq='M').size\n",
    "    forecast = model_fit.forecast(steps=forecast_steps)\n",
    "    \n",
    "    # Set the forecast index to the expected date range\n",
    "    forecast.index = pd.date_range(start=forecast_start, end=forecast_end, freq='M')\n",
    "\n",
    "    # Visualization\n",
    "    plot_train_test_forecast(train_data, test_data, forecast)\n",
    "    \n",
    "    # Print SARIMAX results\n",
    "    print(model_fit.summary())\n",
    "\n",
    "# Run the main function\n",
    "main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a1b357b-7da9-4d75-aebd-d466abb59144",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb993c22-fc33-480a-8ff3-87e7cc028456",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "449c4dc1-fa14-427a-a63d-8ba628e4a26d",
   "metadata": {},
   "source": [
    "# Analyze Model Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecf3c5c0-778a-4c93-9529-b326ed8460c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import logging\n",
    "from statsmodels.tsa.statespace.sarimax import SARIMAX\n",
    "from plotly.subplots import make_subplots\n",
    "import plotly.graph_objs as go\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
    "from statsmodels.tsa.stattools import acf, pacf\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from scipy import stats\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname=s - %(message=s')\n",
    "\n",
    "# Function to split data into training and testing datasets\n",
    "def split_data(data: pd.DataFrame, start_train: str, end_train: str, start_test: str, end_test: str) -> (pd.DataFrame, pd.DataFrame):\n",
    "    \"\"\"Splits the data into training and testing datasets.\"\"\"\n",
    "    train_data = data[start_train:end_train]\n",
    "    test_data = data[start_test:end_test]\n",
    "    return train_data, test_data\n",
    "\n",
    "# Function to implement SARIMAX model and return fitted model\n",
    "def train_sarimax(train_data: pd.Series, p: int, d: int, q: int, seasonal_order=(0, 0, 0, 0)):\n",
    "    \"\"\"Trains the SARIMAX model and returns the fitted model.\"\"\"\n",
    "    model = SARIMAX(train_data, order=(p, d, q), seasonal_order=seasonal_order)\n",
    "    model_fit = model.fit()\n",
    "    return model_fit\n",
    "\n",
    "# Function to analyze residuals\n",
    "def plot_residuals(residuals):\n",
    "    \"\"\"Analyzes residuals for autocorrelation, normality, and heteroscedasticity.\"\"\"\n",
    "    fig, ax = plt.subplots(1, 3, figsize=(18, 5))\n",
    "    \n",
    "    # Residuals vs Time\n",
    "    sns.lineplot(x=residuals.index, y=residuals, ax=ax[0])\n",
    "    ax[0].set_title('Residuals vs Time')\n",
    "    \n",
    "    # Histogram\n",
    "    sns.histplot(residuals, kde=True, ax=ax[1])\n",
    "    ax[1].set_title('Histogram of Residuals')\n",
    "    \n",
    "    # Q-Q Plot\n",
    "    stats.probplot(residuals, dist=\"norm\", plot=ax[2])\n",
    "    ax[2].set_title('Q-Q Plot')\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "# Function to calculate and print model accuracy metrics\n",
    "def calculate_accuracy(test_data, forecast):\n",
    "    \"\"\"Calculates and prints MSE, MAE, and RMSE.\"\"\"\n",
    "    mse = mean_squared_error(test_data, forecast)\n",
    "    mae = mean_absolute_error(test_data, forecast)\n",
    "    rmse = np.sqrt(mse)\n",
    "    \n",
    "    logging.info(f'MSE: {mse:.3f}')\n",
    "    logging.info(f'MAE: {mae:.3f}')\n",
    "    logging.info(f'RMSE: {rmse:.3f}')\n",
    "    \n",
    "    print(f'MSE: {mse:.3f}')\n",
    "    print(f'MAE: {mae:.3f}')\n",
    "    print(f'RMSE: {rmse:.3f}')\n",
    "\n",
    "# Main function to train and analyze SARIMAX model\n",
    "def main():\n",
    "    # Assuming `stationary_data` is already loaded from previous steps\n",
    "    target_column = 'Close_stationary'\n",
    "    train_start, train_end = '01-04-2019', '31-03-2023'\n",
    "    test_start, test_end = '01-04-2023', '31-03-2024'\n",
    "    \n",
    "    train_data, test_data = split_data(stationary_data[target_column].dropna(), train_start, train_end, test_start, test_end)\n",
    "    \n",
    "    # Train SARIMAX model with assumed p, d, q values and seasonal_order (these should be fine-tuned)\n",
    "    p, d, q, seasonal_order = 1, 1, 1, (1, 1, 1, 12)\n",
    "    model_fit = train_sarimax(train_data, p, d, q, seasonal_order)\n",
    "    \n",
    "    # Forecast for the test period\n",
    "    forecast = model_fit.forecast(steps=len(test_data))\n",
    "    \n",
    "    # Plot residuals\n",
    "    plot_residuals(model_fit.resid)\n",
    "    \n",
    "    # Evaluate accuracy metrics\n",
    "    calculate_accuracy(test_data, forecast)\n",
    "    \n",
    "    # Print SARIMAX summary\n",
    "    print(model_fit.summary())\n",
    "\n",
    "# Run the main function\n",
    "main()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f436f7-9401-4b79-bcc1-6578849ea2e8",
   "metadata": {},
   "source": [
    "####\n",
    "\n",
    "Explanation of the Code:\n",
    "Stationarity Transformation: The make_stationary function transforms the series to a stationary form and checks stationarity with the ADF test.\n",
    "Parameter Tuning: The time_series_cv function applies cross-validation with grid search to select the best model parameters.\n",
    "Residual Analysis: The plot_residuals function analyzes residuals for time-based patterns, normality, and other issues.\n",
    "Accuracy Calculation: The calculate_accuracy function computes model evaluation metrics to assess accuracy.\n",
    "Main Function: The main function orchestrates the data preprocessing, parameter tuning, model training, and final evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83d395e4-e817-49a9-aa48-74b34365132e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecc6288c-5825-43b4-b279-8f8c2a48aac5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0c76bd37-2c36-46fa-8404-a653aed0a75a",
   "metadata": {},
   "source": [
    "# implement the Prophet model for forecasting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4050eb3e-8391-4454-b641-b17e97c71ec7",
   "metadata": {},
   "source": [
    "### Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bc8c17f-7530-4f0f-ad92-9b8bfd6e85fe",
   "metadata": {},
   "source": [
    "# Preprocessing & Tuning Prophet model\n",
    "\n",
    "### Final Code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53f65fe-7fc3-40a0-ac68-687965c7d31b",
   "metadata": {},
   "source": [
    "# To further improve:\n",
    "\n",
    "Additional Preprocessing: You might consider more advanced techniques for handling missing values or outliers specific to your dataset.\n",
    "Parameter Tuning: Consider tuning the Prophet model parameters (like seasonality, holidays, etc.) to improve forecast accuracy.\n",
    "Cross-Validation: Use cross-validation methods available in Prophet to validate and adjust the forecast model.\n",
    "\n",
    "\n",
    "\n",
    "focuses on additional preprocessing, parameter tuning, and cross-validation for the Prophet model. The improvements include handling seasonality, holidays, and automatic parameter tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6c25a6f-06cb-45f4-a204-9fbd199f81bc",
   "metadata": {},
   "source": [
    "# Prophet Model Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "599ad08f-c723-44b8-99b3-d5572b03a500",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import yfinance as yf\n",
    "import logging\n",
    "from prophet import Prophet\n",
    "from prophet.diagnostics import cross_validation, performance_metrics\n",
    "import plotly.graph_objs as go\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Download stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def clean_data(data: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Clean stock data by filling missing values and removing duplicates.\"\"\"\n",
    "    logging.info(\"Cleaning data\")\n",
    "    data.fillna(method='ffill', inplace=True)\n",
    "    data.fillna(method='bfill', inplace=True)\n",
    "    data.drop_duplicates(inplace=True)\n",
    "    logging.info(\"Data cleaning complete\")\n",
    "    return data\n",
    "\n",
    "def preprocess_data_for_prophet(data: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Prepare data for Prophet by renaming columns and formatting.\"\"\"\n",
    "    logging.info(\"Preparing data for Prophet model\")\n",
    "    prophet_data = data[['Date', 'Close']].rename(columns={'Date': 'ds', 'Close': 'y'})\n",
    "    logging.info(\"Data preparation complete\")\n",
    "    return prophet_data\n",
    "\n",
    "def tune_prophet_parameters() -> dict:\n",
    "    \"\"\"Tune Prophet model parameters for better accuracy.\"\"\"\n",
    "    return {\n",
    "        'seasonality_mode': 'multiplicative',\n",
    "        'yearly_seasonality': True,\n",
    "        'weekly_seasonality': True,\n",
    "        'daily_seasonality': False\n",
    "    }\n",
    "\n",
    "def prophet_cross_validation(data: pd.DataFrame, params: dict):\n",
    "    \"\"\"Perform cross-validation on the Prophet model.\"\"\"\n",
    "    logging.info(\"Performing cross-validation with Prophet\")\n",
    "    model = Prophet(**params)\n",
    "    model.fit(data)\n",
    "    df_cv = cross_validation(model, initial='730 days', period='180 days', horizon='365 days')\n",
    "    df_p = performance_metrics(df_cv)\n",
    "    logging.info(f\"Cross-validation metrics:\\n{df_p}\")\n",
    "    return df_cv, df_p\n",
    "\n",
    "def plot_cross_validation(df_cv: pd.DataFrame):\n",
    "    \"\"\"Plot the cross-validation forecast against actuals.\"\"\"\n",
    "    logging.info(\"Plotting cross-validation forecast\")\n",
    "    trace_actual = go.Scatter(x=df_cv['ds'], y=df_cv['y'], mode='lines', name='Actual')\n",
    "    trace_forecast = go.Scatter(x=df_cv['ds'], y=df_cv['yhat'], mode='lines', name='Forecast')\n",
    "    trace_upper = go.Scatter(x=df_cv['ds'], y=df_cv['yhat_upper'], mode='lines', name='Upper Confidence', line=dict(dash='dash'))\n",
    "    trace_lower = go.Scatter(x=df_cv['ds'], y=df_cv['yhat_lower'], mode='lines', name='Lower Confidence', line=dict(dash='dash'))\n",
    "    layout = go.Layout(title='Prophet Cross-Validation Forecast', xaxis={'title': 'Date'}, yaxis={'title': 'Stock Price'})\n",
    "    fig = go.Figure(data=[trace_actual, trace_forecast, trace_upper, trace_lower], layout=layout)\n",
    "    fig.show()\n",
    "\n",
    "def main():\n",
    "    # Define parameters\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "\n",
    "    # Download and preprocess data\n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    stock_data = clean_data(stock_data)\n",
    "    prophet_data = preprocess_data_for_prophet(stock_data)\n",
    "\n",
    "    # Perform cross-validation with Prophet\n",
    "    params = tune_prophet_parameters()\n",
    "    df_cv, _ = prophet_cross_validation(prophet_data, params)\n",
    "\n",
    "    # Plot cross-validation results\n",
    "    plot_cross_validation(df_cv)\n",
    "\n",
    "    # Display data summary in table format\n",
    "    logging.info(\"Displaying summary of the stock data:\")\n",
    "    summary = stock_data.describe()\n",
    "    logging.info(\"\\n\" + summary.to_string())  # Log summary as text\n",
    "    display(summary)  # Display summary in table format using pandas\n",
    "\n",
    "# Run the main function\n",
    "if __name__ == '__main__':\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "742ee133-ce44-4cb0-baaa-f0d001476bb0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1e95f623-8fa3-48a1-bbf3-6458780fc8d3",
   "metadata": {},
   "source": [
    "# Prophet Model Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a019d511-cc18-4782-8117-0b69c3303099",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import logging\n",
    "from prophet import Prophet\n",
    "import plotly.graph_objs as go\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "def detect_and_remove_outliers(data: pd.DataFrame, column: str) -> pd.DataFrame:\n",
    "    \"\"\"Detect and remove outliers in the specified column using the IQR method.\"\"\"\n",
    "    logging.info(\"Detecting and removing outliers\")\n",
    "    Q1 = data[column].quantile(0.25)\n",
    "    Q3 = data[column].quantile(0.75)\n",
    "    IQR = Q3 - Q1\n",
    "    mask = (data[column] >= Q1 - 1.5 * IQR) & (data[column] <= Q3 + 1.5 * IQR)\n",
    "    filtered_data = data[mask]\n",
    "    logging.info(\"Outlier removal complete\")\n",
    "    return filtered_data\n",
    "\n",
    "def preprocess_data_for_prophet(data: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Prepare data for Prophet by renaming columns and formatting.\"\"\"\n",
    "    logging.info(\"Preparing data for Prophet model\")\n",
    "    prophet_data = data[['Close']].reset_index()\n",
    "    prophet_data.rename(columns={'Date': 'ds', 'Close': 'y'}, inplace=True)\n",
    "    logging.info(\"Data preparation complete\")\n",
    "    return prophet_data\n",
    "\n",
    "def forecast_prophet(data: pd.DataFrame, periods: int) -> pd.DataFrame:\n",
    "    \"\"\"Forecast future stock prices using the Prophet model.\"\"\"\n",
    "    logging.info(\"Forecasting with Prophet\")\n",
    "    model = Prophet()\n",
    "    model.fit(data)\n",
    "    future = model.make_future_dataframe(periods=periods, freq='M')\n",
    "    forecast = model.predict(future)\n",
    "    return forecast\n",
    "\n",
    "def plot_forecast(actual_data: pd.DataFrame, forecast_data: pd.DataFrame) -> None:\n",
    "    \"\"\"Plot actual and forecasted stock prices.\"\"\"\n",
    "    trace1 = go.Scatter(x=actual_data['ds'], y=actual_data['y'], mode='lines', name='Actual')\n",
    "    trace2 = go.Scatter(x=forecast_data['ds'], y=forecast_data['yhat'], mode='lines', name='Forecast')\n",
    "    layout = go.Layout(title='Prophet Model Forecast', xaxis={'title': 'Date'}, yaxis={'title': 'Stock Price'})\n",
    "    fig = go.Figure(data=[trace1, trace2], layout=layout)\n",
    "    fig.show()\n",
    "\n",
    "def main():\n",
    "    # Define parameters\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    forecast_periods = 12\n",
    "\n",
    "    # Download and preprocess data\n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    stock_data = clean_data(stock_data)\n",
    "    stock_data = detect_and_remove_outliers(stock_data, 'Close')\n",
    "    prophet_data = preprocess_data_for_prophet(stock_data)\n",
    "\n",
    "    # Forecast with Prophet\n",
    "    forecast = forecast_prophet(prophet_data, forecast_periods)\n",
    "\n",
    "    # Display the forecast\n",
    "    display(HTML('<h2>First 5 Rows of the Forecast</h2>'))\n",
    "    display(HTML(forecast.head(5).to_html()))\n",
    "\n",
    "    # Plot the forecast\n",
    "    plot_forecast(prophet_data, forecast)\n",
    "\n",
    "    # Provide summary of the forecast\n",
    "    forecast_summary = forecast[['ds', 'yhat']].tail(forecast_periods).describe()\n",
    "    logging.info(\"Summary of the 12-month forecasted stock prices:\")\n",
    "    display(HTML('<h2>12-Month Forecast Summary</h2>'))\n",
    "    display(HTML(forecast_summary.to_html()))\n",
    "    print(forecast_summary)\n",
    "\n",
    "# Run the main function\n",
    "if __name__ == '__main__':\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32ab716a-b4dc-408a-84ec-25253c9239c8",
   "metadata": {},
   "source": [
    "# Prophet Model Implementation ---- Completed "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fb8587f-744b-4334-b633-f4e6de6b36a4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "58cfd27f-cc7d-49ec-8f41-76ece9398c1e",
   "metadata": {},
   "source": [
    "# XGBoost Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fdf3911-5f3c-450e-8016-e020312b3425",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "447455ed-700e-4f93-acc1-82c1447d6591",
   "metadata": {},
   "source": [
    "### Pre Processing "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59186642-6f73-485b-9f16-bb891eacb19a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import plotly.graph_objs as go\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message=s)')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Downloads stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        data['Date'] = pd.to_datetime(data['Date'])\n",
    "        data.set_index('Date', inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def preprocess_data(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Preprocesses the stock data for exploratory data analysis and model preparation.\"\"\"\n",
    "    # Handling missing values\n",
    "    missing_summary = df.isnull().sum()\n",
    "    logging.info(f\"Missing Values per Column:\\n{missing_summary}\")\n",
    "    df.dropna(inplace=True)\n",
    "\n",
    "    # Removing duplicates\n",
    "    duplicates_count = df.duplicated().sum()\n",
    "    logging.info(f\"Number of Duplicate Rows: {duplicates_count}\")\n",
    "    df.drop_duplicates(inplace=True)\n",
    "\n",
    "    # Statistical Summary\n",
    "    summary_stats = df.describe()\n",
    "    logging.info(f\"Statistical Summary:\\n{summary_stats}\")\n",
    "\n",
    "    # Convert the summary statistics to an HTML table\n",
    "    summary_html = summary_stats.to_html()\n",
    "    with open('statistical_summary.html', 'w') as f:\n",
    "        f.write(summary_html)\n",
    "\n",
    "    return df\n",
    "\n",
    "def stock_data_eda(df: pd.DataFrame):\n",
    "    \"\"\"Performs exploratory data analysis on stock data.\"\"\"\n",
    "    # Line chart for Closing, Opening, and Adjusted Closing Price\n",
    "    fig = go.Figure()\n",
    "    fig.add_trace(go.Scatter(x=df.index, y=df['Close'], mode='lines', name='Close Price'))\n",
    "    fig.add_trace(go.Scatter(x=df.index, y=df['Open'], mode='lines', name='Open Price'))\n",
    "    fig.add_trace(go.Scatter(x=df.index, y=df['Adj Close'], mode='lines', name='Adjusted Close Price'))\n",
    "\n",
    "    # Calculate and add Average Stock Price in a Day\n",
    "    df['Average'] = (df['High'] + df['Low']) / 2\n",
    "    fig.add_trace(go.Scatter(x=df.index, y=df['Average'], mode='lines', name='Average Price'))\n",
    "\n",
    "    fig.update_layout(title='Stock Price Over Time',\n",
    "                      xaxis_title='Date', yaxis_title='Price')\n",
    "    fig.show()\n",
    "\n",
    "    # Heatmap for correlation matrix using Seaborn\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.heatmap(df.corr(), annot=True, cmap='coolwarm')\n",
    "    plt.title('Feature Correlation Matrix')\n",
    "    plt.show()\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    stock_data_eda(preprocessed_data)\n",
    "    logging.info(\"Data preprocessing and EDA completed.\")\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "320c1a7d-0299-40e7-b921-c292d6ea706a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5475faec-900c-4c28-a8fc-e0db0ea90a81",
   "metadata": {},
   "source": [
    "# Feature Engineering and Train & Test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f61abf50-4d4f-4fd7-b3d5-e184a7d317d0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f28215b0-34e1-4443-a242-2217cb58d032",
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "def feature_engineering(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Adds engineered features to the stock data for analysis.\n",
    "    \n",
    "    Args:\n",
    "        df (pd.DataFrame): Preprocessed stock data.\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: Stock data with engineered features.\n",
    "    \"\"\"\n",
    "    # Lag features\n",
    "    df['Lag1'] = df['Close'].shift(1)\n",
    "    df['Lag5'] = df['Close'].shift(5)\n",
    "    df['Lag10'] = df['Close'].shift(10)\n",
    "\n",
    "    # Moving averages\n",
    "    df['MA5'] = df['Close'].rolling(window=5).mean()\n",
    "    df['MA10'] = df['Close'].rolling(window=10).mean()\n",
    "    df['MA20'] = df['Close'].rolling(window=20).mean()\n",
    "\n",
    "    # Volatility\n",
    "    df['Volatility'] = df['Close'].rolling(window=5).std()\n",
    "\n",
    "    # Remove NaN values resulting from rolling windows\n",
    "    df.dropna(inplace=True)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def generate_eda_report(df: pd.DataFrame):\n",
    "    \"\"\"Generates an EDA report with a heatmap for the feature-engineered data.\n",
    "    \n",
    "    Args:\n",
    "        df (pd.DataFrame): Feature-engineered stock data.\n",
    "    \"\"\"\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.heatmap(df.corr(), annot=True, cmap='coolwarm')\n",
    "    plt.title('Feature Correlation Matrix')\n",
    "    plt.show()\n",
    "\n",
    "def split_data(df: pd.DataFrame, test_size: float = 0.2) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n",
    "    \"\"\"Splits the data into training and testing sets.\n",
    "    \n",
    "    Args:\n",
    "        df (pd.DataFrame): The feature-engineered stock data.\n",
    "        test_size (float): Fraction of the data to be used for testing.\n",
    "\n",
    "    Returns:\n",
    "        tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]: X_train, X_test, y_train, y_test.\n",
    "    \"\"\"\n",
    "    X = df.drop(['Close'], axis=1)\n",
    "    y = df['Close']\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, shuffle=False)\n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    # Feature Engineering Step\n",
    "    engineered_data = feature_engineering(preprocessed_data)\n",
    "\n",
    "    # Generate EDA Report\n",
    "    generate_eda_report(engineered_data)\n",
    "\n",
    "    # Train-Test Split\n",
    "    X_train, X_test, y_train, y_test = split_data(engineered_data, test_size=0.2)\n",
    "    logging.info(f\"Training set size: {X_train.shape[0]}, Testing set size: {X_test.shape[0]}\")\n",
    "    logging.info(\"Data preprocessing, feature engineering, EDA report, and train-test splitting completed.\")\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "859d3b65-62b5-43a6-8787-62fcb1da51fc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "72db4c0b-124a-47e0-9008-95d8aed260ee",
   "metadata": {},
   "source": [
    "# Model implemation to forecast the stock price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d47aad8-4d2e-4c4b-a2b2-12868ceb82cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "import xgboost as xgb\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Downloads stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        data['Date'] = pd.to_datetime(data['Date'])\n",
    "        data.set_index('Date', inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def preprocess_data(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Preprocesses the stock data for exploratory data analysis and model preparation.\"\"\"\n",
    "    # Handling missing values\n",
    "    missing_summary = df.isnull().sum()\n",
    "    logging.info(f\"Missing Values per Column:\\n{missing_summary}\")\n",
    "    df.dropna(inplace=True)\n",
    "\n",
    "    # Removing duplicates\n",
    "    duplicates_count = df.duplicated().sum()\n",
    "    logging.info(f\"Number of Duplicate Rows: {duplicates_count}\")\n",
    "    df.drop_duplicates(inplace=True)\n",
    "\n",
    "    return df\n",
    "\n",
    "def feature_engineering(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Adds engineered features to the stock data for analysis.\"\"\"\n",
    "    # Lag features\n",
    "    df['Lag1'] = df['Close'].shift(1)\n",
    "    df['Lag5'] = df['Close'].shift(5)\n",
    "    df['Lag10'] = df['Close'].shift(10)\n",
    "\n",
    "    # Moving averages\n",
    "    df['MA5'] = df['Close'].rolling(window=5).mean()\n",
    "    df['MA10'] = df['Close'].rolling(window=10).mean()\n",
    "    df['MA20'] = df['Close'].rolling(window=20).mean()\n",
    "\n",
    "    # Volatility\n",
    "    df['Volatility'] = df['Close'].rolling(window=5).std()\n",
    "\n",
    "    # Remove NaN values resulting from rolling windows\n",
    "    df.dropna(inplace=True)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def generate_eda_report(df: pd.DataFrame):\n",
    "    \"\"\"Generates an EDA report with a heatmap for the feature-engineered data.\"\"\"\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.heatmap(df.corr(), annot=True, cmap='coolwarm')\n",
    "    plt.title('Feature Correlation Matrix')\n",
    "    plt.show()\n",
    "\n",
    "def split_data(df: pd.DataFrame, test_size: float = 0.2) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n",
    "    \"\"\"Splits the data into training and testing sets.\n",
    "    \n",
    "    Args:\n",
    "        df (pd.DataFrame): The feature-engineered stock data.\n",
    "        test_size (float): Fraction of the data to be used for testing.\n",
    "\n",
    "    Returns:\n",
    "        tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]: X_train, X_test, y_train, y_test.\n",
    "    \"\"\"\n",
    "    X = df.drop(['Close'], axis=1)\n",
    "    y = df['Close']\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, shuffle=False)\n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "def train_xgboost_model(X_train, y_train):\n",
    "    \"\"\"Trains an XGBoost model using grid search for hyperparameter tuning.\"\"\"\n",
    "    xg_reg = xgb.XGBRegressor(objective='reg:squarederror', seed=42)\n",
    "    \n",
    "    param_grid = {\n",
    "        'n_estimators': [100, 200],\n",
    "        'max_depth': [3, 6],\n",
    "        'learning_rate': [0.01, 0.05, 0.1],\n",
    "        'subsample': [0.8, 1.0]\n",
    "    }\n",
    "    \n",
    "    grid_search = GridSearchCV(estimator=xg_reg, param_grid=param_grid, cv=5, scoring='neg_mean_squared_error', verbose=1)\n",
    "    grid_search.fit(X_train, y_train)\n",
    "    \n",
    "    logging.info(f\"Best Hyperparameters: {grid_search.best_params_}\")\n",
    "    return grid_search.best_estimator_\n",
    "\n",
    "def forecast_stock_prices(model, X_test, steps=12):\n",
    "    \"\"\"Forecasts stock prices for the next 12 months using the trained model.\"\"\"\n",
    "    future_predictions = []\n",
    "    current_input = X_test[-1, :].reshape(1, -1)\n",
    "    \n",
    "    for _ in range(steps):\n",
    "        pred = model.predict(current_input)[0]\n",
    "        future_predictions.append(pred)\n",
    "        \n",
    "        current_input = np.roll(current_input, -1)\n",
    "        current_input[0, -1] = pred\n",
    "\n",
    "    return future_predictions\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    # Feature Engineering Step\n",
    "    engineered_data = feature_engineering(preprocessed_data)\n",
    "\n",
    "    # Generate EDA Report\n",
    "    generate_eda_report(engineered_data)\n",
    "\n",
    "    # Train-Test Split\n",
    "    X_train, X_test, y_train, y_test = split_data(engineered_data, test_size=0.2)\n",
    "    logging.info(f\"Training set size: {X_train.shape[0]}, Testing set size: {X_test.shape[0]}\")\n",
    "\n",
    "    # Train XGBoost Model\n",
    "    model = train_xgboost_model(X_train, y_train)\n",
    "\n",
    "    # Forecast Stock Prices for the Next 12 Months\n",
    "    predictions = forecast_stock_prices(model, X_test, steps=12)\n",
    "    logging.info(f\"Predictions for the next 12 months: {predictions}\")\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57d35f8b-2804-4c37-93b4-d9bf292efca6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "import xgboost as xgb\n",
    "\n",
    "\n",
    "def forecast_stock_prices(model, X_test, steps=12):\n",
    "    \"\"\"Forecasts stock prices for the next 12 months using the trained model.\"\"\"\n",
    "    future_predictions = []\n",
    "    current_input = X_test[-1, :].reshape(1, -1)\n",
    "    \n",
    "    for _ in range(steps):\n",
    "        pred = model.predict(current_input)[0]\n",
    "        future_predictions.append(pred)\n",
    "        \n",
    "        current_input = np.roll(current_input, -1)\n",
    "        current_input[0, -1] = pred\n",
    "\n",
    "    return future_predictions\n",
    "\n",
    "def plot_forecast(y_test, predictions):\n",
    "    \"\"\"Plots the actual and forecasted stock prices.\"\"\"\n",
    "    forecast_dates = pd.date_range(start=y_test.index[-1] + pd.DateOffset(1), periods=len(predictions), freq='M')\n",
    "    forecast_series = pd.Series(predictions, index=forecast_dates)\n",
    "    \n",
    "    plt.figure(figsize=(12, 6))\n",
    "    plt.plot(y_test.index, y_test, label='Actual')\n",
    "    plt.plot(forecast_series.index, forecast_series, label='Forecast', linestyle='--')\n",
    "    plt.xlabel('Date')\n",
    "    plt.ylabel('Stock Price')\n",
    "    plt.title('Actual vs Forecasted Stock Price')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    # Feature Engineering Step\n",
    "    engineered_data = feature_engineering(preprocessed_data)\n",
    "\n",
    "    # Generate EDA Report\n",
    "    generate_eda_report(engineered_data)\n",
    "\n",
    "    # Train-Test Split\n",
    "    X_train, X_test, y_train, y_test = split_data(engineered_data, test_size=0.2)\n",
    "    logging.info(f\"Training set size: {X_train.shape[0]}, Testing set size: {X_test.shape[0]}\")\n",
    "\n",
    "    # Train XGBoost Model\n",
    "    model = train_xgboost_model(X_train, y_train)\n",
    "\n",
    "    # Forecast Stock Prices for the Next 12 Months\n",
    "    predictions = forecast_stock_prices(model, X_test, steps=12)\n",
    "    \n",
    "    # Show Forecast Summary and Graph\n",
    "    forecast_df = pd.DataFrame({'Forecast': predictions})\n",
    "    logging.info(f\"Forecast Summary:\\n{forecast_df}\")\n",
    "    plot_forecast(y_test, predictions)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9c53e75-a768-4504-843a-122676aeae71",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "import xgboost as xgb\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Downloads stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        data['Date'] = pd.to_datetime(data['Date'])\n",
    "        data.set_index('Date', inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def preprocess_data(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Preprocesses the stock data for exploratory data analysis and model preparation.\"\"\"\n",
    "    # Handling missing values\n",
    "    missing_summary = df.isnull().sum()\n",
    "    logging.info(f\"Missing Values per Column:\\n{missing_summary}\")\n",
    "    df.dropna(inplace=True)\n",
    "\n",
    "    # Removing duplicates\n",
    "    duplicates_count = df.duplicated().sum()\n",
    "    logging.info(f\"Number of Duplicate Rows: {duplicates_count}\")\n",
    "    df.drop_duplicates(inplace=True)\n",
    "\n",
    "    return df\n",
    "\n",
    "def feature_engineering(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Adds engineered features to the stock data for analysis.\"\"\"\n",
    "    # Lag features\n",
    "    df['Lag1'] = df['Close'].shift(1)\n",
    "    df['Lag5'] = df['Close'].shift(5)\n",
    "    df['Lag10'] = df['Close'].shift(10)\n",
    "\n",
    "    # Moving averages\n",
    "    df['MA5'] = df['Close'].rolling(window=5).mean()\n",
    "    df['MA10'] = df['Close'].rolling(window=10).mean()\n",
    "    df['MA20'] = df['Close'].rolling(window=20).mean()\n",
    "\n",
    "    # Volatility\n",
    "    df['Volatility'] = df['Close'].rolling(window=5).std()\n",
    "\n",
    "    # Remove NaN values resulting from rolling windows\n",
    "    df.dropna(inplace=True)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def generate_eda_report(df: pd.DataFrame):\n",
    "    \"\"\"Generates an EDA report with a heatmap for the feature-engineered data.\"\"\"\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.heatmap(df.corr(), annot=True, cmap='coolwarm')\n",
    "    plt.title('Feature Correlation Matrix')\n",
    "    plt.show()\n",
    "\n",
    "def split_data(df: pd.DataFrame, test_size: float = 0.2) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n",
    "    \"\"\"Splits the data into training and testing sets.\n",
    "    \n",
    "    Args:\n",
    "        df (pd.DataFrame): The feature-engineered stock data.\n",
    "        test_size (float): Fraction of the data to be used for testing.\n",
    "\n",
    "    Returns:\n",
    "        tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]: X_train, X_test, y_train, y_test.\n",
    "    \"\"\"\n",
    "    X = df.drop(['Close'], axis=1)\n",
    "    y = df['Close']\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, shuffle=False)\n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "def train_xgboost_model(X_train, y_train):\n",
    "    \"\"\"Trains an XGBoost model using grid search for hyperparameter tuning.\"\"\"\n",
    "    xg_reg = xgb.XGBRegressor(objective='reg:squarederror', seed=42)\n",
    "    \n",
    "    param_grid = {\n",
    "        'n_estimators': [100, 200],\n",
    "        'max_depth': [3, 6],\n",
    "        'learning_rate': [0.01, 0.05, 0.1],\n",
    "        'subsample': [0.8, 1.0]\n",
    "    }\n",
    "    \n",
    "    grid_search = GridSearchCV(estimator=xg_reg, param_grid=param_grid, cv=5, scoring='neg_mean_squared_error', verbose=1)\n",
    "    grid_search.fit(X_train, y_train)\n",
    "    \n",
    "    logging.info(f\"Best Hyperparameters: {grid_search.best_params_}\")\n",
    "    return grid_search.best_estimator_\n",
    "\n",
    "def forecast_stock_prices(model, X_test, steps=12):\n",
    "    \"\"\"Forecasts stock prices for the next 12 months using the trained model.\"\"\"\n",
    "    future_predictions = []\n",
    "    last_valid_index = X_test.shape[0] - 1\n",
    "    current_input = X_test.iloc[last_valid_index, :].values.reshape(1, -1)\n",
    "    \n",
    "    for _ in range(steps):\n",
    "        pred = model.predict(current_input)[0]\n",
    "        future_predictions.append(pred)\n",
    "        \n",
    "        current_input = np.roll(current_input, -1)\n",
    "        current_input[0, -1] = pred\n",
    "\n",
    "    return future_predictions\n",
    "\n",
    "def plot_forecast(y_test, predictions):\n",
    "    \"\"\"Plots the actual and forecasted stock prices.\"\"\"\n",
    "    forecast_dates = pd.date_range(start=y_test.index[-1] + pd.DateOffset(1), periods=len(predictions), freq='M')\n",
    "    forecast_series = pd.Series(predictions, index=forecast_dates)\n",
    "    \n",
    "    plt.figure(figsize=(12, 6))\n",
    "    plt.plot(y_test.index, y_test, label='Actual')\n",
    "    plt.plot(forecast_series.index, forecast_series, label='Forecast', linestyle='--')\n",
    "    plt.xlabel('Date')\n",
    "    plt.ylabel('Stock Price')\n",
    "    plt.title('Actual vs Forecasted Stock Price')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    # Feature Engineering Step\n",
    "    engineered_data = feature_engineering(preprocessed_data)\n",
    "\n",
    "    # Generate EDA Report\n",
    "    generate_eda_report(engineered_data)\n",
    "\n",
    "    # Train-Test Split\n",
    "    X_train, X_test, y_train, y_test = split_data(engineered_data, test_size=0.2)\n",
    "    logging.info(f\"Training set size: {X_train.shape[0]}, Testing set size: {X_test.shape[0]}\")\n",
    "\n",
    "    # Train XGBoost Model\n",
    "    model = train_xgboost_model(X_train, y_train)\n",
    "\n",
    "    # Forecast Stock Prices for the Next 12 Months\n",
    "    predictions = forecast_stock_prices(model, X_test, steps=12)\n",
    "    \n",
    "    # Show Forecast Summary and Graph\n",
    "    forecast_df = pd.DataFrame({'Forecast': predictions})\n",
    "    logging.info(f\"Forecast Summary:\\n{forecast_df}\")\n",
    "    plot_forecast(y_test, predictions)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f236616b-63d6-4f8f-8649-68c5c39ca045",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f53ff05-0d06-46be-bc5e-0c2f9329e123",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6c97f58-091a-4af8-a308-69c96b2f9678",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d153e8ef-8952-4899-b1db-3a2e334e95e5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74c84d56-3d13-4a01-9f96-6043cb799918",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "import xgboost as xgb\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Downloads stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        data['Date'] = pd.to_datetime(data['Date'])\n",
    "        data.set_index('Date', inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def preprocess_data(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Preprocesses the stock data for exploratory data analysis and model preparation.\"\"\"\n",
    "    # Handling missing values\n",
    "    missing_summary = df.isnull().sum()\n",
    "    logging.info(f\"Missing Values per Column:\\n{missing_summary}\")\n",
    "    df.dropna(inplace=True)\n",
    "\n",
    "    # Removing duplicates\n",
    "    duplicates_count = df.duplicated().sum()\n",
    "    logging.info(f\"Number of Duplicate Rows: {duplicates_count}\")\n",
    "    df.drop_duplicates(inplace=True)\n",
    "\n",
    "    return df\n",
    "\n",
    "def feature_engineering(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Adds engineered features to the stock data for analysis.\"\"\"\n",
    "    # Lag features\n",
    "    df['Lag1'] = df['Close'].shift(1)\n",
    "    df['Lag5'] = df['Close'].shift(5)\n",
    "    df['Lag10'] = df['Close'].shift(10)\n",
    "\n",
    "    # Moving averages\n",
    "    df['MA5'] = df['Close'].rolling(window=5).mean()\n",
    "    df['MA10'] = df['Close'].rolling(window=10).mean()\n",
    "    df['MA20'] = df['Close'].rolling(window=20).mean()\n",
    "\n",
    "    # Volatility\n",
    "    df['Volatility'] = df['Close'].rolling(window=5).std()\n",
    "\n",
    "    # Remove NaN values resulting from rolling windows\n",
    "    df.dropna(inplace=True)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def generate_eda_report(df: pd.DataFrame):\n",
    "    \"\"\"Generates an EDA report with a heatmap for the feature-engineered data.\"\"\"\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.heatmap(df.corr(), annot=True, cmap='coolwarm')\n",
    "    plt.title('Feature Correlation Matrix')\n",
    "    plt.show()\n",
    "\n",
    "def split_data(df: pd.DataFrame, test_size: float = 0.2) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n",
    "    \"\"\"Splits the data into training and testing sets.\n",
    "    \n",
    "    Args:\n",
    "        df (pd.DataFrame): The feature-engineered stock data.\n",
    "        test_size (float): Fraction of the data to be used for testing.\n",
    "\n",
    "    Returns:\n",
    "        tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]: X_train, X_test, y_train, y_test.\n",
    "    \"\"\"\n",
    "    X = df.drop(['Close'], axis=1)\n",
    "    y = df['Close']\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, shuffle=False)\n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "def train_xgboost_model(X_train, y_train):\n",
    "    \"\"\"Trains an XGBoost model using grid search for hyperparameter tuning.\"\"\"\n",
    "    xg_reg = xgb.XGBRegressor(objective='reg:squarederror', seed=42)\n",
    "    \n",
    "    param_grid = {\n",
    "        'n_estimators': [100, 200],\n",
    "        'max_depth': [3, 6],\n",
    "        'learning_rate': [0.01, 0.05, 0.1],\n",
    "        'subsample': [0.8, 1.0]\n",
    "    }\n",
    "    \n",
    "    grid_search = GridSearchCV(estimator=xg_reg, param_grid=param_grid, cv=5, scoring='neg_mean_squared_error', verbose=1)\n",
    "    grid_search.fit(X_train, y_train)\n",
    "    \n",
    "    logging.info(f\"Best Hyperparameters: {grid_search.best_params_}\")\n",
    "    return grid_search.best_estimator_\n",
    "\n",
    "def forecast_stock_prices(model, X_test, steps=12):\n",
    "    \"\"\"Forecasts stock prices for the next 12 months using the trained model.\"\"\"\n",
    "    future_predictions = []\n",
    "    last_valid_index = X_test.shape[0] - 1\n",
    "    current_input = X_test.iloc[last_valid_index, :].values.reshape(1, -1)\n",
    "    \n",
    "    for _ in range(steps):\n",
    "        pred = model.predict(current_input)[0]\n",
    "        future_predictions.append(pred)\n",
    "        \n",
    "        current_input = np.roll(current_input, -1)\n",
    "        current_input[0, -1] = pred\n",
    "\n",
    "    return future_predictions\n",
    "\n",
    "def plot_forecast(y_test, predictions):\n",
    "    \"\"\"Plots the actual and forecasted stock prices.\"\"\"\n",
    "    forecast_dates = pd.date_range(start=y_test.index[-1] + pd.DateOffset(1), periods=len(predictions), freq='M')\n",
    "    forecast_series = pd.Series(predictions, index=forecast_dates)\n",
    "    \n",
    "    plt.figure(figsize=(12, 6))\n",
    "    plt.plot(y_test.index, y_test, label='Actual')\n",
    "    plt.plot(forecast_series.index, forecast_series, label='Forecast', linestyle='--')\n",
    "    plt.xlabel('Date')\n",
    "    plt.ylabel('Stock Price')\n",
    "    plt.title('Actual vs Forecasted Stock Price')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    # Feature Engineering Step\n",
    "    engineered_data = feature_engineering(preprocessed_data)\n",
    "\n",
    "    # Generate EDA Report\n",
    "    generate_eda_report(engineered_data)\n",
    "\n",
    "    # Train-Test Split\n",
    "    X_train, X_test, y_train, y_test = split_data(engineered_data, test_size=0.2)\n",
    "    logging.info(f\"Training set size: {X_train.shape[0]}, Testing set size: {X_test.shape[0]}\")\n",
    "\n",
    "    # Train XGBoost Model\n",
    "    model = train_xgboost_model(X_train, y_train)\n",
    "\n",
    "    # Forecast Stock Prices for the Next 12 Months\n",
    "    predictions = forecast_stock_prices(model, X_test, steps=12)\n",
    "    \n",
    "    # Show Forecast Summary and Graph\n",
    "    forecast_df = pd.DataFrame({'Forecast': predictions})\n",
    "    logging.info(f\"Forecast Summary:\\n{forecast_df}\")\n",
    "    plot_forecast(y_test, predictions)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0036a567-ede3-4c2e-a7ae-db1175b70978",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ff4034cf-4450-4798-a488-9f90bf337047",
   "metadata": {},
   "source": [
    "# Final model code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a0d0900-591a-4cfe-887b-c512d2ca696f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import plotly.graph_objs as go\n",
    "from sklearn.model_selection import TimeSeriesSplit, GridSearchCV\n",
    "import xgboost as xgb\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Downloads stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        data['Date'] = pd.to_datetime(data['Date'])\n",
    "        data.set_index('Date', inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def preprocess_data(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Preprocesses the stock data for exploratory data analysis and model preparation.\"\"\"\n",
    "    # Handling missing values\n",
    "    missing_summary = df.isnull().sum()\n",
    "    logging.info(f\"Missing Values per Column:\\n{missing_summary}\")\n",
    "    df.dropna(inplace=True)\n",
    "\n",
    "    # Removing duplicates\n",
    "    duplicates_count = df.duplicated().sum()\n",
    "    logging.info(f\"Number of Duplicate Rows: {duplicates_count}\")\n",
    "    df.drop_duplicates(inplace=True)\n",
    "\n",
    "    return df\n",
    "\n",
    "def feature_engineering(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Adds engineered features to the stock data for analysis.\"\"\"\n",
    "    df['Lag1'] = df['Close'].shift(1)\n",
    "    df['Lag5'] = df['Close'].shift(5)\n",
    "    df['Lag10'] = df['Close'].shift(10)\n",
    "    df['MA5'] = df['Close'].rolling(window=5).mean()\n",
    "    df['MA10'] = df['Close'].rolling(window=10).mean()\n",
    "    df['MA20'] = df['Close'].rolling(window=20).mean()\n",
    "    df['Volatility'] = df['Close'].rolling(window=5).std()\n",
    "    df.dropna(inplace=True)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def generate_eda_report(df: pd.DataFrame):\n",
    "    \"\"\"Generates an EDA report with a heatmap for the feature-engineered data.\"\"\"\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.heatmap(df.corr(), annot=True, cmap='coolwarm')\n",
    "    plt.title('Feature Correlation Matrix')\n",
    "    plt.show()\n",
    "\n",
    "def split_data(df: pd.DataFrame, test_size: float = 0.2) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n",
    "    \"\"\"Splits the data into training and testing sets.\"\"\"\n",
    "    X = df.drop(['Close'], axis=1)\n",
    "    y = df['Close']\n",
    "    \n",
    "    split_index = int(len(X) * (1 - test_size))\n",
    "    X_train, X_test = X[:split_index], X[split_index:]\n",
    "    y_train, y_test = y[:split_index], y[split_index:]\n",
    "    \n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "def train_xgboost_model(X_train, y_train):\n",
    "    \"\"\"Trains an XGBoost model using grid search for hyperparameter tuning.\"\"\"\n",
    "    xg_reg = xgb.XGBRegressor(objective='reg:squarederror', seed=42)\n",
    "    \n",
    "    param_grid = {\n",
    "        'n_estimators': [100, 300, 500],\n",
    "        'max_depth': [3, 5, 7],\n",
    "        'learning_rate': [0.01, 0.05, 0.1, 0.3],\n",
    "        'subsample': [0.6, 0.8, 1.0]\n",
    "    }\n",
    "    \n",
    "    time_split = TimeSeriesSplit(n_splits=5)\n",
    "    grid_search = GridSearchCV(estimator=xg_reg, param_grid=param_grid, cv=time_split, scoring='neg_mean_squared_error', verbose=1)\n",
    "    grid_search.fit(X_train, y_train)\n",
    "    \n",
    "    logging.info(f\"Best Hyperparameters: {grid_search.best_params_}\")\n",
    "    return grid_search.best_estimator_\n",
    "\n",
    "def forecast_stock_prices(model, X_test, steps=12):\n",
    "    \"\"\"Forecasts stock prices for the next 12 months using the trained model.\"\"\"\n",
    "    future_predictions = []\n",
    "    last_valid_index = X_test.shape[0] - 1\n",
    "    current_input = X_test.iloc[last_valid_index, :].values.reshape(1, -1)\n",
    "    \n",
    "    for _ in range(steps):\n",
    "        pred = model.predict(current_input)[0]\n",
    "        future_predictions.append(pred)\n",
    "        \n",
    "        current_input = np.roll(current_input, -1)\n",
    "        current_input[0, -1] = pred\n",
    "\n",
    "    return future_predictions\n",
    "\n",
    "def plot_forecast(y_test, predictions):\n",
    "    \"\"\"Plots the actual and forecasted stock prices using Plotly.\"\"\"\n",
    "    forecast_dates = pd.date_range(start=y_test.index[-1] + pd.DateOffset(1), periods=len(predictions), freq='M')\n",
    "    forecast_series = pd.Series(predictions, index=forecast_dates)\n",
    "    \n",
    "    fig = go.Figure()\n",
    "    fig.add_trace(go.Scatter(x=y_test.index, y=y_test, mode='lines', name='Actual'))\n",
    "    fig.add_trace(go.Scatter(x=forecast_series.index, y=forecast_series, mode='lines', name='Forecast'))\n",
    "    fig.update_layout(title='Actual vs Forecasted Stock Price',\n",
    "                      xaxis_title='Date', yaxis_title='Stock Price')\n",
    "    fig.show()\n",
    "\n",
    "def summarize_forecast(predictions):\n",
    "    \"\"\"Summarizes forecast results by calculating descriptive statistics.\"\"\"\n",
    "    forecast_series = pd.Series(predictions)\n",
    "    summary_stats = forecast_series.describe()\n",
    "    logging.info(f\"Forecast Summary:\\n{summary_stats}\")\n",
    "\n",
    "def evaluate_model_performance(y_test, predictions):\n",
    "    \"\"\"Evaluates the model's accuracy using common metrics and prints a summary report.\"\"\"\n",
    "    forecast_dates = pd.date_range(start=y_test.index[-1] + pd.DateOffset(1), periods=len(predictions), freq='M')\n",
    "    forecast_series = pd.Series(predictions, index=forecast_dates)\n",
    "    \n",
    "    common_length = min(len(y_test), len(forecast_series))\n",
    "    y_test_common = y_test[-common_length:]\n",
    "    forecast_series_common = forecast_series[:common_length]\n",
    "    \n",
    "    mae = mean_absolute_error(y_test_common, forecast_series_common)\n",
    "    rmse = np.sqrt(mean_squared_error(y_test_common, forecast_series_common))\n",
    "    r2 = r2_score(y_test_common, forecast_series_common)\n",
    "    \n",
    "    print(\"\\nModel Performance Summary:\")\n",
    "    print(f\"Mean Absolute Error (MAE): {mae:.2f}\")\n",
    "    print(f\"Root Mean Squared Error (RMSE): {rmse:.2f}\")\n",
    "    print(f\"R-squared (R2): {r2:.2f}\")\n",
    "    \n",
    "    logging.info(f\"Model Performance:\\n\"\n",
    "                 f\"Mean Absolute Error (MAE): {mae:.2f}\\n\"\n",
    "                 f\"Root Mean Squared Error (RMSE): {rmse:.2f}\\n\"\n",
    "                 f\"R-squared (R2): {r2:.2f}\")\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    # Feature Engineering Step\n",
    "    engineered_data = feature_engineering(preprocessed_data)\n",
    "\n",
    "    # Generate EDA Report\n",
    "    generate_eda_report(engineered_data)\n",
    "\n",
    "    # Train-Test Split\n",
    "    X_train, X_test, y_train, y_test = split_data(engineered_data, test_size=0.2)\n",
    "    logging.info(f\"Training set size: {X_train.shape[0]}, Testing set size: {X_test.shape[0]}\")\n",
    "\n",
    "    # Train XGBoost Model\n",
    "    model = train_xgboost_model(X_train, y_train)\n",
    "\n",
    "    # Forecast Stock Prices for the Next 12 Months\n",
    "    predictions = forecast_stock_prices(model, X_test, steps=12)\n",
    "    \n",
    "    # Summarize and Show Forecast Graph\n",
    "    summarize_forecast(predictions)\n",
    "    plot_forecast(y_test, predictions)\n",
    "\n",
    "    # Evaluate Model Performance\n",
    "    evaluate_model_performance(y_test, predictions)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7293a79-a730-433a-80a0-2324b25072df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# without accuracy model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7b4507d-b5ba-4111-88e4-106856baf2a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yfinance as yf\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import plotly.graph_objs as go\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "import xgboost as xgb\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime=s - %(message)s')\n",
    "\n",
    "def download_stock_data(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:\n",
    "    \"\"\"Downloads stock data from Yahoo Finance.\"\"\"\n",
    "    try:\n",
    "        logging.info(f\"Downloading stock data for ticker: {ticker}\")\n",
    "        data = yf.download(ticker, start=start_date, end=end_date)\n",
    "        data.reset_index(inplace=True)\n",
    "        data['Date'] = pd.to_datetime(data['Date'])\n",
    "        data.set_index('Date', inplace=True)\n",
    "        return data\n",
    "    except Exception as e:\n",
    "        logging.error(f\"Failed to download stock data: {e}\")\n",
    "        raise\n",
    "\n",
    "def preprocess_data(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Preprocesses the stock data for exploratory data analysis and model preparation.\"\"\"\n",
    "    # Handling missing values\n",
    "    missing_summary = df.isnull().sum()\n",
    "    logging.info(f\"Missing Values per Column:\\n{missing_summary}\")\n",
    "    df.dropna(inplace=True)\n",
    "\n",
    "    # Removing duplicates\n",
    "    duplicates_count = df.duplicated().sum()\n",
    "    logging.info(f\"Number of Duplicate Rows: {duplicates_count}\")\n",
    "    df.drop_duplicates(inplace=True)\n",
    "\n",
    "    return df\n",
    "\n",
    "def feature_engineering(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    \"\"\"Adds engineered features to the stock data for analysis.\"\"\"\n",
    "    # Lag features\n",
    "    df['Lag1'] = df['Close'].shift(1)\n",
    "    df['Lag5'] = df['Close'].shift(5)\n",
    "    df['Lag10'] = df['Close'].shift(10)\n",
    "\n",
    "    # Moving averages\n",
    "    df['MA5'] = df['Close'].rolling(window=5).mean()\n",
    "    df['MA10'] = df['Close'].rolling(window=10).mean()\n",
    "    df['MA20'] = df['Close'].rolling(window=20).mean()\n",
    "\n",
    "    # Volatility\n",
    "    df['Volatility'] = df['Close'].rolling(window=5).std()\n",
    "\n",
    "    # Remove NaN values resulting from rolling windows\n",
    "    df.dropna(inplace=True)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def generate_eda_report(df: pd.DataFrame):\n",
    "    \"\"\"Generates an EDA report with a heatmap for the feature-engineered data.\"\"\"\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.heatmap(df.corr(), annot=True, cmap='coolwarm')\n",
    "    plt.title('Feature Correlation Matrix')\n",
    "    plt.show()\n",
    "\n",
    "def split_data(df: pd.DataFrame, test_size: float = 0.2) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n",
    "    \"\"\"Splits the data into training and testing sets.\n",
    "    \n",
    "    Args:\n",
    "        df (pd.DataFrame): The feature-engineered stock data.\n",
    "        test_size (float): Fraction of the data to be used for testing.\n",
    "\n",
    "    Returns:\n",
    "        tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]: X_train, X_test, y_train, y_test.\n",
    "    \"\"\"\n",
    "    X = df.drop(['Close'], axis=1)\n",
    "    y = df['Close']\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, shuffle=False)\n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "def train_xgboost_model(X_train, y_train):\n",
    "    \"\"\"Trains an XGBoost model using grid search for hyperparameter tuning.\"\"\"\n",
    "    xg_reg = xgb.XGBRegressor(objective='reg:squarederror', seed=42)\n",
    "    \n",
    "    param_grid = {\n",
    "        'n_estimators': [100, 200],\n",
    "        'max_depth': [3, 6],\n",
    "        'learning_rate': [0.01, 0.05, 0.1],\n",
    "        'subsample': [0.8, 1.0]\n",
    "    }\n",
    "    \n",
    "    grid_search = GridSearchCV(estimator=xg_reg, param_grid=param_grid, cv=5, scoring='neg_mean_squared_error', verbose=1)\n",
    "    grid_search.fit(X_train, y_train)\n",
    "    \n",
    "    logging.info(f\"Best Hyperparameters: {grid_search.best_params_}\")\n",
    "    return grid_search.best_estimator_\n",
    "\n",
    "def forecast_stock_prices(model, X_test, steps=12):\n",
    "    \"\"\"Forecasts stock prices for the next 12 months using the trained model.\"\"\"\n",
    "    future_predictions = []\n",
    "    last_valid_index = X_test.shape[0] - 1\n",
    "    current_input = X_test.iloc[last_valid_index, :].values.reshape(1, -1)\n",
    "    \n",
    "    for _ in range(steps):\n",
    "        pred = model.predict(current_input)[0]\n",
    "        future_predictions.append(pred)\n",
    "        \n",
    "        current_input = np.roll(current_input, -1)\n",
    "        current_input[0, -1] = pred\n",
    "\n",
    "    return future_predictions\n",
    "\n",
    "def plot_forecast(y_test, predictions):\n",
    "    \"\"\"Plots the actual and forecasted stock prices using Plotly.\"\"\"\n",
    "    forecast_dates = pd.date_range(start=y_test.index[-1] + pd.DateOffset(1), periods=len(predictions), freq='M')\n",
    "    forecast_series = pd.Series(predictions, index=forecast_dates)\n",
    "    \n",
    "    fig = go.Figure()\n",
    "    fig.add_trace(go.Scatter(x=y_test.index, y=y_test, mode='lines', name='Actual'))\n",
    "    fig.add_trace(go.Scatter(x=forecast_series.index, y=forecast_series, mode='lines', name='Forecast'))\n",
    "    fig.update_layout(title='Actual vs Forecasted Stock Price',\n",
    "                      xaxis_title='Date', yaxis_title='Stock Price')\n",
    "    fig.show()\n",
    "\n",
    "def summarize_forecast(predictions):\n",
    "    \"\"\"Summarizes forecast results by calculating descriptive statistics.\"\"\"\n",
    "    forecast_series = pd.Series(predictions)\n",
    "    summary_stats = forecast_series.describe()\n",
    "    logging.info(f\"Forecast Summary:\\n{summary_stats}\")\n",
    "\n",
    "def main():\n",
    "    ticker_symbol = 'TATAELXSI.NS'\n",
    "    start_date = '2019-04-01'\n",
    "    end_date = '2024-03-31'\n",
    "    \n",
    "    stock_data = download_stock_data(ticker_symbol, start_date, end_date)\n",
    "    preprocessed_data = preprocess_data(stock_data)\n",
    "    \n",
    "    # Feature Engineering Step\n",
    "    engineered_data = feature_engineering(preprocessed_data)\n",
    "\n",
    "    # Generate EDA Report\n",
    "    generate_eda_report(engineered_data)\n",
    "\n",
    "    # Train-Test Split\n",
    "    X_train, X_test, y_train, y_test = split_data(engineered_data, test_size=0.2)\n",
    "    logging.info(f\"Training set size: {X_train.shape[0]}, Testing set size: {X_test.shape[0]}\")\n",
    "\n",
    "    # Train XGBoost Model\n",
    "    model = train_xgboost_model(X_train, y_train)\n",
    "\n",
    "    # Forecast Stock Prices for the Next 12 Months\n",
    "    predictions = forecast_stock_prices(model, X_test, steps=12)\n",
    "    \n",
    "    # Summarize and Show Forecast Graph\n",
    "    summarize_forecast(predictions)\n",
    "    plot_forecast(y_test, predictions)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a24bfdb-eb07-4877-bd10-9442bbd06859",
   "metadata": {},
   "source": [
    "# Comparing Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "308b84bf-7b54-4568-9d19-d28f2fe4290c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5aecfca2-6df0-4470-be34-e6fd547decbf",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94413395-c2cd-495b-b73f-fa09e8889338",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "697edc51-27aa-4cc0-b83d-2eba6d488b14",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f319739d-57b8-48a3-a7ce-7345d850aa14",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "anaconda-2022.05-py39",
   "language": "python",
   "name": "conda-env-anaconda-2022.05-py39-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}