DLISIO_Simple_Reading.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DLISIO in a Nutshell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This is a guide to walk you through the basic functionalities of dlisio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "import dlisio\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import numpy.lib.recfunctions as rfn\n",
    "\n",
    "import hvplot.pandas\n",
    "import holoviews as hv\n",
    "from holoviews import opts, streams\n",
    "from holoviews.plotting.links import DataLink\n",
    "hv.extension('bokeh', logo=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can work with a single file using the cell below - or by adding an additional for loop to the code below, you can work through a list of files.  Another option is to use os.walk to get all .dlis files in a parent folder.  Example:\n",
    "\n",
    "    for (root, dirs, files) in os.walk(folderpath):\n",
    "        for f in files:\n",
    "            filepath = os.path.join(root, f)\n",
    "            if filepath.endswith('.' + 'dlis'):\n",
    "                print(filepath)\n",
    "                \n",
    "### But for this example, we will work with a single .dlis file specified in the cell below.  Note that there are some .dlis file formats that are not supported by DLISIO yet - good to catch them in a try except loop if you are reading files enmasse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We will load a dlis file for well 15/9-F-4 from the open source Volve dataset available here: https://data.equinor.com/dataset/Volve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath = r\"\\WL_RAW_CAL-DEN-GR-NEU-REMP_MWD_1.DLIS\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query for specific curve\n",
    "\n",
    "### Very quickly you can use regex to find certain curves in a file (helpful if you are scanning a lot of files for certain curves)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with dlisio.dlis.load(filepath) as file:\n",
    "    for d in file:\n",
    "        depth_channels = d.find('CHANNEL','TDEP')\n",
    "        for channel in depth_channels:\n",
    "            print(channel.name)\n",
    "            print(channel.curves())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining internal files and frames\n",
    "\n",
    "### Keep in mind that dlis files can contain multiple files and multiple frames.  You can quickly get a numpy array of the curves in each frame below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with dlisio.dlis.load(filepath) as file:\n",
    "    print(file.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with dlisio.dlis.load(filepath) as file:\n",
    "    for d in file:\n",
    "        for fram in d.frames:\n",
    "            print(d.channels)\n",
    "            print(fram.curves())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metadata including Origin information per frame (well name and header)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with dlisio.dlis.load(filepath) as file:\n",
    "    for d in file:\n",
    "        print(d.describe())\n",
    "        for fram in d.frames:\n",
    "            print(fram.describe())\n",
    "            for channel in d.channels:\n",
    "                print(channel.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with dlisio.dlis.load(filepath) as file:\n",
    "    for d in file:\n",
    "        for origin in d.origins:\n",
    "            print(origin.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading a full dlis file\n",
    "\n",
    "### But most likely we want a single data frame of every curve, no matter which frame it came from.  So we write a bit more code to look through each frame, then look at each channel and get the curve name and unit information along with it.  We will also save the information about which internal file and which frame each curve resides in.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curves_L = []\n",
    "curves_name = []\n",
    "longs = []\n",
    "unit = []\n",
    "files_L = []\n",
    "files_num = []\n",
    "frames = []\n",
    "frames_num = []\n",
    "with dlisio.dlis.load(filepath) as file:\n",
    "    for d in file:\n",
    "        files_L.append(d)\n",
    "        frame_count = 0\n",
    "        for fram in d.frames:\n",
    "            if frame_count == 0:\n",
    "                frames.append(fram)\n",
    "                frame_count = frame_count + 1\n",
    "                for channel in d.channels:\n",
    "                    curves_name.append(channel.name)\n",
    "                    longs.append(channel.long_name)\n",
    "                    unit.append(channel.units)\n",
    "                    files_num.append(len(files_L))\n",
    "                    frames_num.append(len(frames))\n",
    "                    curves = channel.curves()\n",
    "                    curves_L.append(curves)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curve_index = pd.DataFrame(\n",
    "{'Curve': curves_name,\n",
    "'Long': longs,\n",
    "'Unit': unit,\n",
    "'Internal_File': files_num,\n",
    "'Frame_Number': frames_num\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curve_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Pandas dataframe for the entire .dlis file\n",
    "\n",
    "### We have to be careful creating a dataframe for the whole .dlis file as often there are some curves that represent mulitple values (numpy array of list values).  So, you can use something like:\n",
    "\n",
    "df = pd.DataFrame(data=curves_L, index=curves_name).T\n",
    "\n",
    "### to view the full dlis file with lists as some of the curve values.\n",
    "\n",
    "### Or we will use the code below to process each curve's 2D numpy array, stacking it if the curve contains multiple values per sample.  Then we convert each curve into its own dataframe (uniquifying the column names by adding a .1, .2, .3...etc).  Then, to preserve the order with the curve index above, append each data frame together in order to build the final dlis full dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def df_column_uniquify(df):\n",
    "    df_columns = df.columns\n",
    "    new_columns = []\n",
    "    for item in df_columns:\n",
    "        counter = 0\n",
    "        newitem = item\n",
    "        while newitem in new_columns:\n",
    "            counter += 1\n",
    "            newitem = \"{}_{}\".format(item, counter)\n",
    "        new_columns.append(newitem)\n",
    "    df.columns = new_columns\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curve_df = pd.DataFrame()\n",
    "name_index = 0\n",
    "for c in curves_L:\n",
    "    name = curves_name[name_index]\n",
    "    np.vstack(c)\n",
    "    try:\n",
    "        num_col = c.shape[1]\n",
    "        col_name = [name] * num_col\n",
    "        df = pd.DataFrame(data=c, columns=col_name)\n",
    "        name_index = name_index + 1\n",
    "        df = df_column_uniquify(df)\n",
    "        curve_df = pd.concat([curve_df, df], axis=1)\n",
    "    except:\n",
    "        num_col = 0\n",
    "        df = pd.DataFrame(data=c, columns=[name])\n",
    "        name_index = name_index + 1\n",
    "        curve_df = pd.concat([curve_df, df], axis=1)\n",
    "        continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curve_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## If we have a simpler dlis file with a single logical file and single frame and with single data values in each channel.\n",
    "with dlisio.dlis.load(filepath) as file:\n",
    "    logical_count = 0\n",
    "    for d in file:\n",
    "        frame_count = 0\n",
    "        for fram in d.frames:\n",
    "            if frame_count == 0 & logical_count == 0:\n",
    "                curves = fram.curves()\n",
    "                curve_df = pd.DataFrame(curves, index=curves[fram.index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curve_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Then we can set the index and start making some plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curve_df = df_column_uniquify(curve_df)\n",
    "curve_df['DEPTH_Calc_ft'] = curve_df.loc[:,'TDEP'] * 0.0083333 #0.1 inch/12 inches per foot\n",
    "curve_df['DEPTH_ft'] = curve_df['DEPTH_Calc_ft']\n",
    "curve_df = curve_df.set_index(\"DEPTH_Calc_ft\")\n",
    "curve_df.index.names = [None]\n",
    "curve_df = curve_df.replace(-999.25,np.nan)\n",
    "min_val = curve_df['DEPTH_ft'].min()\n",
    "max_val = curve_df['DEPTH_ft'].max()\n",
    "curve_list = list(curve_df.columns)\n",
    "curve_list.remove('DEPTH_ft')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "curve_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def curve_plot(log, df, depthname):\n",
    "  aplot = df.hvplot(x=depthname, y=log, invert=True, flip_yaxis=True, shared_axes=True,\n",
    "                       height=600, width=300).opts(fontsize={'labels': 16,'xticks': 14, 'yticks': 14})\n",
    "  return aplot;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plotlist = [curve_plot(x, df=curve_df, depthname='DEPTH_ft') for x in curve_list]\n",
    "well_section = hv.Layout(plotlist).cols(len(curve_list))\n",
    "well_section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hopefully that is enough code to get you started working with DLISIO.  There is much more functionality which can be accessed with help(dlisio) or at the read the docs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}