Unix_Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Please make sure that you are using the bash kernel to run this notebok ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concepts explained in this tutorial:\n",
    "- What a shell is\n",
    "- Environment variables\n",
    "- $PATH\n",
    "- Navigating the file system\n",
    "- Relative vs. absolute paths\n",
    "- Chaining commands using pipes\n",
    "\n",
    "Commands covered:\n",
    "- echo\n",
    "- which\n",
    "- pwd\n",
    "- ls\n",
    "- ls -lt\n",
    "- mkdir\n",
    "- cd\n",
    "- touch\n",
    "- cat\n",
    "- cp\n",
    "- rm\n",
    "- rm -r\n",
    "- mv\n",
    "- head\n",
    "- tail\n",
    "- gzip\n",
    "- zcat\n",
    "\n",
    "Operators/aliases covered:\n",
    "- ..\n",
    "- \\>\\>\n",
    "- |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.1 Unix Basics#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start by going through some basic unix commands. \"Unix\" is a term for a family of operating systems (just like \"Windows\"). The Mac operating systems (OSX) are part of the Unix family. You will also hear the term \"Linux\" a lot - linux refers to series of operating systems that are also part of the Unix family. Unix operating systems are very popular for running servers. However, while you may be used to interacting with your laptop using a graphical interface, these servers do not support graphical interfaces (as graphical interfaces are a LOT of work to build and are less flexible). Instead, you need to interact with them through the command line. Don't worry, it's easy once you get the hang of it, and it looks really cool to people who don't know what you are doing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How commands are understood##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's clarify a little about how unix commands are understood. The program that understands your unix commands is something called a \"shell\". If you hear the term \"bash\" get thrown around, just know that this is the name of a shell. There are many different kinds of shells, and different commands are slightly different depending on the shell that is being run. For now, we will focus on the bash shell.\n",
    "\n",
    "Let us double check that the bash shell is being run. To do this, we will use the command \"echo $SHELL\" illustrated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash\n"
     ]
    }
   ],
   "source": [
    "#lines that begin with a hastag are comments; they are ignored\n",
    "#by the shell.\n",
    "echo $SHELL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's understand in detail how the command above was understood by the shell.\n",
    "\n",
    "Commands tend to have the format:<br />\n",
    "[name of the program] [one or more arguments to the program...]<br />\n",
    "(\"arguments\" just refers to all the terms that control the behaviour of the program).\n",
    "\n",
    "In the example above, \"echo\" is the name of the program. The echo program prints the value of its arguments to the screen.\n",
    "\n",
    "There is also a concept of an \"environment variable\". A variable is something that stores information, and an \"environment variable\" is something that stores information that can be accessed by the shell (i.e. they pertain to the \"environment\" that commands are run in). Environment variables can be accessed by using \"\\$\" (so \\$SHELL produces the value of the SHELL variable). In the example above, \\$SHELL gives the location where the current shell program is stored. On my Mac, this location happens to be /bin/bash. It may be slightly different when you run this notebook, but it should still end in \"bash\".\n",
    "\n",
    "How do we read a path like \"/bin/bash\"? Files in a Unix system are organized into folders (also called \"directories\"). \"/\" refers to the topmost level. \"/bin\" is the \"bin\" folder (\"bin\" is an abbreviation for \"binaries\"; \"binary\" files refers to the form that runable programs often take). So \"/bin/bash\" refers to the \"bash\" program stored in the \"bin\" folder.\n",
    "\n",
    "When the shell is told to run a program (like \"echo\"), how does the shell know where to find it? This is where the PATH environment variable comes in. The PATH variable stores the names of a number of directories, each separated by a colon. The shell looks at each of these directories in turn and sees if a runnable file (also called an \"executable\") with the appropriate name exists in any of those directories. Once it finds such an executable, it stops looking and executes it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Exercise 1.1.1 </b><br />\n",
    "Display the contents of your PATH environment variable below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "##enter the command to print out the value of PATH below\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"which\" program will tell you the exact location of the file that would be used to execute a particular program. For example, we can find the location of the \"echo\" program as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/echo\n"
     ]
    }
   ],
   "source": [
    "which echo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even find the location of the \"which\" program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/bin/which\n"
     ]
    }
   ],
   "source": [
    "which which"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Exercise 1.1.2 </b><br />\n",
    "A colleague of yours has installed one version of a program. However, when try to launch the program, the shell keeps launching a different version of the program than what they installed. What might the problem be? How could you check whether this is the problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Navigating the file system, creating and editing files##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a number of handy commands used to navigate the filesystem. Let's start with pwd, which telss you the directory that you are currently running out of:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/vptl\n"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also display the contents of the directory with the ls command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ComputationalThinkingforHumanBiologySyllabus.pdf  Unix_Basics.ipynb  class_4\n",
      "Images\t\t\t\t\t\t  class_1\t     class_5\n",
      "JupyterNtbks\t\t\t\t\t  class_2\n"
     ]
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ls command with the -lt argument can be used to reveal a lot of additional information about the files, such as file permissions, creation date and file size. You can read more about that here: http://www.tutorialspoint.com/unix/unix-file-management.htm and here: http://www.tutorialspoint.com/unix/unix-file-permission.htm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 312\n",
      "drwxrwxrwx 3 root   root     4096 Apr  5 23:11 class_5\n",
      "drwxrwxrwx 3 ubuntu ubuntu   4096 Apr  5 23:10 class_4\n",
      "drwxrwxrwx 3 root   root     4096 Apr  5 23:10 class_2\n",
      "drwxrwxrwx 6 ubuntu ubuntu   4096 Apr  5 23:04 class_1\n",
      "drwxrwxrwx 2 ubuntu ubuntu   4096 Apr  5 23:02 JupyterNtbks\n",
      "-rwxrwxrwx 1 ubuntu ubuntu  24185 Apr  5 22:28 Unix_Basics.ipynb\n",
      "drwxrwxrwx 2 ubuntu ubuntu   4096 Apr  5 21:55 Images\n",
      "-rwxrwxrwx 1 ubuntu ubuntu 269580 Apr  5 21:55 ComputationalThinkingforHumanBiologySyllabus.pdf\n"
     ]
    }
   ],
   "source": [
    "ls -lt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a new directory, use the mkdir command. We will create a new directory called \"exercise\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "mkdir exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now change into the exercise directory. To do this, we will need to use the cd command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/vptl/exercise\n"
     ]
    }
   ],
   "source": [
    "cd exercise\n",
    "#pwd will tell us the present working directory\n",
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once in the cd directory, let's make a file with the name test_file.txt. To do this, we can use the touch command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test_file.txt\n"
     ]
    }
   ],
   "source": [
    "touch test_file.txt\n",
    "ls #lists the contents of the directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write to test_file.txt. To do this, we will use the >> operator which appends the output to a file instead of printing it to the screen. If the file didn't exist, the >> operator would create the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "echo \"hello world\" >> test_file.txt\n",
    "echo \"hellooooo? rude\" >> test_file.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's confirm the write using the cat command which displays the contents of the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world\n",
      "hellooooo? rude\n"
     ]
    }
   ],
   "source": [
    "cat test_file.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FYI: if we use a single > instead of >>, this will overwrite the file rather than appending to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Exercise 1.1.3 </b>\n",
    "Write a single command to make test_file.txt contain a single line saying anything of your choosing. Hint: use the > operator mentioned above. Also know that \"touch\" will not overwrite files (rather, it will update their 'last edited' date)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world\n",
      "hellooooo? rude\n"
     ]
    }
   ],
   "source": [
    "#put your command below:\n",
    "\n",
    "cat test_file.txt #confirm that the contents are what you expect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a copy of the file with the cp command. We will call the new file \"test2_file.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test2_file.txt\ttest_file.txt\n"
     ]
    }
   ],
   "source": [
    "cp test_file.txt test2_file.txt\n",
    "ls #list the contents of the directory to confirm the copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now delete test_file.txt using rm. Note that rm is PERMANENT - there is no recycle bin and no undo, so be careful when you use it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test2_file.txt\n"
     ]
    }
   ],
   "source": [
    "rm test_file.txt\n",
    "ls #list the contents of the directory to confirm the rm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now rename test2_file.txt to test3_file.txt using the mv command (\"move\"). Why would we ever want to use mv verus just doing a cp followed by rm? For one, disk space. You may not always have the room to make a new copy of the file. The mv command works by updating information that says where the file is located without actually changing the data of the file on disk, so at no point is a copy made (Note: this is only true if you are doing the move withing the same \"drive\", which is usually the case if the first folder relative to the root is the same).\n",
    "\n",
    "However, be aware that you can overwrite files with mv or cp if the file you are moving or copying to is a file that already exists, and when you do so the change is permanent. So rename or copy carefully!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test3_file.txt\n"
     ]
    }
   ],
   "source": [
    "mv test2_file.txt test3_file.txt\n",
    "ls #list the contents of the directory to confirm the rename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now clean up the exercise directory. Change back to the previous directory with cd. The \"..\" exists in every directory and points to \"one directory up\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FYI you can also use ls to list the contents of a specific directory. Let's list the contents of the exercise directory without changing into it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test3_file.txt\n"
     ]
    }
   ],
   "source": [
    "ls exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now delete the exercise directory. Note that the standard rm command doesn't work - you will get a message saying exercise is a directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: cannot remove ‘exercise’: Is a directory\n",
      "ComputationalThinkingforHumanBiologySyllabus.pdf  Unix_Basics.ipynb  class_4\n",
      "Images\t\t\t\t\t\t  class_1\t     class_5\n",
      "JupyterNtbks\t\t\t\t\t  class_2\t     exercise\n"
     ]
    }
   ],
   "source": [
    "rm exercise\n",
    "ls #show that the deletion did not happen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To delete the directory, you need to specify the -r flag, which stands for recursive. Recursion just refers to repeating the same action on a smaller scale. In this case, the rm command will delete the contents of any subdirectories (that's where \"recursion\" comes in), and will then delete the directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ComputationalThinkingforHumanBiologySyllabus.pdf  Unix_Basics.ipynb  class_4\n",
      "Images\t\t\t\t\t\t  class_1\t     class_5\n",
      "JupyterNtbks\t\t\t\t\t  class_2\n"
     ]
    }
   ],
   "source": [
    "rm -r exercise\n",
    "ls #show that the deletion happened"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A note on relative vs. absolute paths##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you execute the pwd command (which shows the present working directory), the information that is printed out begins with a \"/\". This is called an \"absolute path\" to the present directory - \"absolute\" because it specifies the full location of the directory relative to the \"root directory\" (which is the \"/\").\n",
    "\n",
    "By contrast, when we made the exercise directory, we didn't specify a location beginning with \"/\" - instead, we just said \"mkdir exercise\", and the exercise directory was created in the present directory. This is called a \"relative path\" because the location of \"exercise\" was interpreted RELATIVE to location of the present working directory. If we had said \"mkdir ../exercise\", it would have created the exercise directory one level above the present working directory (remember \"..\" points to the directory one level up).\n",
    "\n",
    "To get the absolute path, you must take the relative path and append it to the absolute path of the present working directory. You can always specify absolute paths to commands like cd and ls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Exercise 1.1.4 </b>\n",
    "What would be the result of the following commands?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls a_dir\n",
      "a_dir\n",
      "ls a_dir/a_dir\n",
      "a_dir  hi.txt\n",
      "ls a_dir/a_dir/a_dir\n",
      "a_dir\n",
      "ls a_dir/a_dir/a_dir/a_dir\n"
     ]
    }
   ],
   "source": [
    "#-p creates nested directories if they don't exist\n",
    "mkdir -p exercise/a_dir/a_dir/a_dir/a_dir\n",
    "cd exercise\n",
    "cd a_dir/a_dir\n",
    "touch a_dir/../../a_dir/a_dir/../hi.txt\n",
    "cd ../..\n",
    "echo \"ls a_dir\"\n",
    "ls a_dir\n",
    "echo \"ls a_dir/a_dir\"\n",
    "ls a_dir/a_dir\n",
    "echo \"ls a_dir/a_dir/a_dir\"\n",
    "ls a_dir/a_dir/a_dir\n",
    "echo \"ls a_dir/a_dir/a_dir/a_dir\"\n",
    "ls a_dir/a_dir/a_dir/a_dir\n",
    "#cleanup\n",
    "cd ..\n",
    "rm -r exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Exercise 1.1.5 </b>\n",
    "What is the absolute path of hi.txt in the example below? Check if you're right by issueing the command \"cat /absolute/path/to/hi.txt\", which will throw an error if your absolute path is incorrect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "blah\n",
      "cat: /replace/with/absolute/path/to/hi.txt: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "mkdir -p exercise/a_dir/a_dir/\n",
    "echo \"blah\" > exercise/a_dir/a_dir/hi.txt\n",
    "cat exercise/a_dir/a_dir/hi.txt\n",
    "cat /replace/with/absolute/path/to/hi.txt\n",
    "#cleanup\n",
    "rm -r exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chaining commands with a pipe operator##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"|\", called a \"pipe operator\" (should be present above your return key) can be used to send the output of one command as input to another command. This is illustrated below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by creating an exercise folder with the file hi.txt that contains 3 lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "line1\n",
      "line2\n",
      "line3\n"
     ]
    }
   ],
   "source": [
    "mkdir exercise\n",
    "cd exercise\n",
    "touch hi.txt\n",
    "echo \"line1\" >> hi.txt\n",
    "echo \"line2\" >> hi.txt\n",
    "echo \"line3\" >> hi.txt\n",
    "#view the contents of hi.txt to confirm\n",
    "cat hi.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To set up an example involving the pipe operator, we're going to introduce a number of commands. The head and tail commands that can be used to view the top or bottom lines of a file, as illustrated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the top 2 lines of hi.txt\n",
      "line1\n",
      "line2\n",
      "View the bottom 1 line of hi.txt\n",
      "line3\n"
     ]
    }
   ],
   "source": [
    "echo \"View the top 2 lines of hi.txt\"\n",
    "head -2 hi.txt\n",
    "echo \"View the bottom 1 line of hi.txt\"\n",
    "tail -1 hi.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also zip up files with the gzip command. This is something you should get in the habit of doing in order to save space. It would also be a smart thing to do if you're ever transferring large files, as it would reduce the sizes of the files you need to transfer. The gzip command will automatically create a new file with .gz appended to the file name. Gzipped files do not HAVE to have the .gz extension; it's just useful to give gzipped files this extension so that you can keep track of which files are gzipped and which ones are not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hi.txt.gz\n"
     ]
    }
   ],
   "source": [
    "gzip hi.txt\n",
    "ls #view the contents of the directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FYI, files can be unzipped with the gunzip command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hi.txt\n",
      "hi.txt.gz\n"
     ]
    }
   ],
   "source": [
    "gunzip hi.txt.gz #decompress the file. The gz extension is automatically removed\n",
    "ls\n",
    "gzip hi.txt #compress the file again for our example\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say we want to view the contents of the file without unzipping the file on disk (which we may want to do because we don't have the disk space to unzip it). Because the file is compressed, the cat command would give us nonsensical results. Instead, we use the zcat command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "line1\n",
      "line2\n",
      "line3\n",
      "hi.txt.gz\n"
     ]
    }
   ],
   "source": [
    "zcat hi.txt.gz #view the contents\n",
    "ls #this shows that the file is still zipped up on disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But now let's say we want to go further and view the top two lines of hi.txt.gz without unzipping it on disk. This is where the pipe operator is useful. It will allow us to send the output of zcat to the head command to use as input, as illustrated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "line1\n",
      "line2\n"
     ]
    }
   ],
   "source": [
    "zcat hi.txt.gz | head -2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could go further and compress the output of the head command to create a gzipped file that has only the first two lines. To do this, we first use gzip -c (which produces an output of gzipped data) and then we write this output to a file using the > operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first_two_lines.txt.gz\thi.txt.gz\n",
      "line1\n",
      "line2\n"
     ]
    }
   ],
   "source": [
    "zcat hi.txt.gz | head -2 | gzip -c > first_two_lines.txt.gz\n",
    "ls #list out the files in the directory\n",
    "zcat first_two_lines.txt.gz #view the contents of the new zipped file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now clean up the exercise directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "cd ..\n",
    "rm -r exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Exercise 1.1.6 </b>\n",
    "In the cell below, print ONLY the second line of hi.txt using a one-line command. Hint: this can be accomplished using the pipe operator and commands we have covered in the section below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "mkdir exercise\n",
    "cd exercise\n",
    "touch hi.txt\n",
    "echo \"line1\" >> hi.txt\n",
    "echo \"line2\" >> hi.txt\n",
    "echo \"line3\" >> hi.txt\n",
    "\n",
    "###Add your one-line command to print the second line of hi.txt\n",
    "\n",
    "\n",
    "#cleanup\n",
    "cd ..\n",
    "rm -r exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the tutorial that I (Avanti) used to learn Unix: http://www.ee.surrey.ac.uk/Teaching/Unix/\n",
    "\n",
    "Here's a more detailed tutorial from tutorialspoint:\n",
    "http://www.tutorialspoint.com/unix/index.htm\n",
    "\n",
    "Another resource geared towards bioinformatics: http://manuals.bioinformatics.ucr.edu/home/linux‐basics\n",
    "\n",
    "Reference for commonly useful commands: https://sites.google.com/site/anshulkundaje/inotes/programming/shell-scripts\n",
    "\n",
    "Learning shell programming: http://www.learnshell.org/\n",
    "\n",
    "Debugging shell scripts: http://www.shellcheck.net/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}