cwl_tutorial_2024.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "private_outputs": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rosafilgueira/Workflows_Seminar/blob/main/cwl_tutorial_2024.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.1 - <font color='#097969'>Opening Statement<font>"
      ],
      "metadata": {
        "id": "B-X8fQy_-fMg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**Purpose**\n",
        "\n",
        "This tutorial demonstrates how to write workflows using a workflow language. An introduction to workflow theory, implementation using the CWL language, and execution on a compute environment will be demonstrated. By the end of the session, participants will be able to write and execute basic workflows in CWL. Skills learned will be easily transferable to other workflow languages (eg Nextflow) participants may want to learn in the future.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "SY8qRsCr0__B"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Goals**\n",
        "\n",
        "The workshop is split into 2 sections: *Tools* and *Workflows*.\n",
        "\n",
        "During the ***tools*** section, we will learn the mechanics of CWL. <br>\n",
        "We will learn how to write YAML, and how tools are defined in CWL. <br>\n",
        "As we go, we will write a CWL tool wrapper for the `cutadapt` software.\n",
        "\n",
        "During the ***workflows*** section, we will extend our knowledge to workflows.  <br>\n",
        "\n",
        "By the end of this tutorial, you will be able to:\n",
        "\n",
        "- Understand the main components of a workflow\n",
        "- Understand the main features of CWL\n",
        "- Write basic workflows in CWL\n",
        "- Run CWL workflows using an execution engine\n",
        "- Have the skills to continue your own learning"
      ],
      "metadata": {
        "id": "ba95t8pm-Zoy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Intended Knowledge**\n",
        "\n",
        " This tutorial will involve writing workflows using CWL and running command line tools.\n",
        "\n",
        "The following is highly recommended:\n",
        "\n",
        "- Basic familiarity with a programming language (ie Python, R, Javascript)"
      ],
      "metadata": {
        "id": "Pd95vImm-YqV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "### 1.2 - <font color='#097969'>Google Colab<font>\n",
        "\n",
        "Workflows can be developed using any IDE or environment you are familiar with. VSC is common for workflow development because it is popular, and has plugins for various workflow languages.\n",
        "\n",
        "Today, we are using Google Colab.\n",
        "[Colab](https://research.google.com/colaboratory/faq.html) is a popular service which provides an [interactive notebook](https://jupyter.org/) and a [runtime](https://research.google.com/colaboratory/faq.html#executed-code).\n",
        "The notebook allows us to write text and code, while the runtime provides some compute resources to execute our code. It is a fantastic resource for learning so we will use this environment today.\n",
        "\n",
        "TODO mention that CLI is usually used, magic here to reduce complexity\n",
        "\n",
        "**The notebook (main window)** allows you to read instructions, and write / run CWL.  \n",
        "This is where will spend most of our time.\n",
        "\n",
        "**The sidebar** allows you to perform basic tasks. You can view the notebook outline, search for text, or manipulate files in our local directory. Click the Explorer button to see which files are in this repository. If a file is opened, it will appear as a new panel on the right of the notebook.\n",
        "\n",
        "**Runtime** colab provides you with a free VM environment which runs your code in the background. <br>It has 1-2 CPU cores, a few gb MEM, and 100 GB disk. These are more than adequate for today. The runtime will timeout after 90 minutes of inactivity.  \n",
        "\n",
        "**Files can be saved** to your local machine from the file browser (sidebar -> files). <br>\n",
        "Hover over a file, click on the three dots `⋮` then select `Download`.\n",
        "\n"
      ],
      "metadata": {
        "id": "Z2_pjuCUwju6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "**Fixing Mistakes**\n",
        "\n",
        "If you make a mistake, thats ok!\n",
        "\n",
        "To undo editing a CWL cell, `ctrl + z` works as normal. <br>\n",
        "Make sure you're actively editing the cell - double click on it if needed.\n",
        "\n",
        "If you deleted a cell from the notebook, `ctrl + m + z` should do the trick.\n",
        "\n",
        "If things aren't recoverable, refreshing the page and reconnecting to a new runtime should reset everything. <br>\n",
        "In this case, make sure you re-acquire the data and re-install the software environment (see the [setup section](#setup)).\n"
      ],
      "metadata": {
        "id": "QnvwTVhmFI0r"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "**CWL Cells**\n",
        "\n",
        "This material will ask the user to write and run CWL. <br>\n",
        "When running CWL, user inputs may be needed.\n",
        "\n"
      ],
      "metadata": {
        "id": "98v5VBCHoaM-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "**Editing CWL Cells**\n",
        "\n",
        "You can edit a cell containing CWL code by `double-clicking`.<br>\n",
        "Once you are done, `click away` or `shift + enter` to stop editing. <br>\n",
        "Make sure you keep the three backticks \\`\\`\\` at the top and bottom of the cell!"
      ],
      "metadata": {
        "id": "yUiRLzNDpJTo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "<figure>\n",
        "<img src=\"https://raw.githubusercontent.com/GraceAHall/cwl-workshop/main/media/colab3.gif\" width=\"600\">\n",
        "<figcaption>Fig 1. gif illustrating how to edit a CWL cell. </figcaption>\n",
        "</figure>\n"
      ],
      "metadata": {
        "id": "GdcGHvNP0yi1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "**Running CWL Cells**\n",
        "\n"
      ],
      "metadata": {
        "id": "eOOU-SDu0F59"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To run a CWL code cell, press the ▶ button below.\n",
        "\n",
        "<figure>\n",
        "<img src=\"https://raw.githubusercontent.com/GraceAHall/cwl-workshop/main/media/colab1.gif\" width=\"600\">\n",
        "<figcaption>Fig 2. gif illustrating how to run a CWL cell. </figcaption>\n",
        "</figure>\n"
      ],
      "metadata": {
        "id": "plA7TzbVpAbn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "**Input Value Form**\n",
        "\n"
      ],
      "metadata": {
        "id": "7ygF2k91pNY-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The values for inputs can also be changed. <br>This will be used to test our CWL code given different inputs.\n",
        "\n",
        "<figure>\n",
        "<img src=\"https://raw.githubusercontent.com/GraceAHall/cwl-workshop/main/media/colab2.gif\" width=\"600\">\n",
        "<figcaption>Fig 3. gif illustrating how to change form values. </figcaption>\n",
        "</figure>\n"
      ],
      "metadata": {
        "id": "-lB-QlbLo24-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "If **Show Code** was clicked on an CWL cell run form, it can be hidden again using *right click -> form -> hide code*\n",
        "\n",
        "<figure>\n",
        "<img src=\"https://raw.githubusercontent.com/GraceAHall/cwl-workshop/main/media/colab5.gif\" width=\"600\">\n",
        "<figcaption>Fig 4. gif illustrating how to hide form code. </figcaption>\n",
        "</figure>\n"
      ],
      "metadata": {
        "id": "97qqcGvzHR0p"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n"
      ],
      "metadata": {
        "id": "IuXB48IOfDHy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.3 - <font color='#097969'>Background<font>"
      ],
      "metadata": {
        "id": "fYEu9ybl6cHK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "Computational workflows have become a core aspect of scientific research. Most researchers will need to run a repeatable analysis involving multiple software tools at some stage in their career. For this reason, Research Software Engineer (RSE) skills involving developing, executing, or maintaining workflows are becoming increasingly important.  \n",
        "\n"
      ],
      "metadata": {
        "id": "tpd56jGCyGtw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***History***\n",
        "\n",
        "Workflows are not new!\n",
        "\n",
        "They can be written in almost any computer language. <br>\n",
        "For example, a workflow can be written using a shell script, or python. <br>\n",
        "The core ingredient of what makes a workflow is simply running multiple programs to provide a more complete analysis.\n",
        "\n",
        "Today, there are more than 150 workflow systems designed to execute workflows.\n",
        "They are continually evolving as new data processing and internet technologies arise.\n",
        "\n",
        "In the last decade, <font color=\"red\">workflow languages & systems</font> have gained increasing popularity. <br>\n",
        "The choice of language / system depends on multiple factors. Whether a UI or code is preferred, which software programs are executed in the workflow, the hardware required, and even the privacy requirements of the data.\n",
        "\n",
        "Galaxy, CWL, WDL and Nextflow are currently very popular.  While Galaxy is the easiest system to get started with, it is not suitable for all workflows and all data. In these cases, researchers may need to use a dedicated workflow language; for example, CWL.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "MrHupBOvUybx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Tools and Workflows***\n",
        "\n",
        "<font color=\"red\">Workflows</font> are computational pipelines designed to perform analysis. <br> They accept input data, do some processing, then return outputs. <br>\n",
        "For processing, workflows use ***tools*** to perform the tasks needed.\n",
        "\n",
        "<font color=\"red\">Tools</font> perform a specific task in a workflow. <br>\n",
        "An example in bioinformatics is [read alignment](https://www.ebi.ac.uk/training/online/courses/functional-genomics-ii-common-technologies-and-data-analysis-methods/rna-sequencing/performing-a-rna-seq-experiment/data-analysis/read-mapping-or-alignment/) to a reference genome. <br>\n",
        "Tools accept inputs, run the software required, then return outputs back to the workflow.\n",
        "\n",
        "\n",
        "<figure>\n",
        "<img src=\"https://raw.githubusercontent.com/GraceAHall/cwl-workshop/main/media/tools_and_workflows.png\" width=\"700px\">\n",
        "<figcaption>Fig 5. Relationship between tools, workflows, and workflow languages. </figcaption>\n",
        "</figure>\n",
        "\n"
      ],
      "metadata": {
        "id": "eNjbqbv3hS1c"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Workflow Languages & Workflow Systems***\n",
        "\n",
        "A well written workflow should be easy for someone else to pick up and run without detailed instruction. <br>\n",
        "This is harder than it sounds! Different RSEs write code differently, and workflows can be tricky to execute when you consider the filesystem, software environment, and running on different machines. The goal of workflow languages & workflow systems is to solve these issues.\n",
        "\n",
        "<font color=\"red\">Workflow languages</font> are specially designed for defining computational workflows.<br> They provide a standardised syntax to define workflow components and their behaviour, as well as the movement of data through the pipeline.\n",
        "\n",
        "Workflow languages strive to achieve two goals: simplicity & expressiveness.\n",
        "In terms of simplicity - workflows should be easy to read and write. This is so the development process is smooth, and so others can read & understand your code. Expressiveness refers to the flexibility of the language - in short, can users create the exact workflow they have in mind? If not, they may be turned away.\n",
        "\n",
        "<font color=\"red\">Workflow systems</font> simplify the execution of workflows.\n",
        "They handle many complex tasks associated with workflows automatically. For example, they can manage the filesystem and software environment while executing tools and workflows, and can even cache intermediate results so workflows can be resumed later in the case of an error or crash.\n",
        "\n",
        "Due to the above, workflow languages & their systems have become fantastic way to deliver FAIR (Findable, Accessible, Interoperable, Reusable) research.  RSEs can more easily ***share*** workflow definitions because of the simplified & structured language. The finished workflows are more ***accessible*** for others due to the workflow system handling execution, and recent adoption of containerisation and cloud computing features. Collectively, the above, results in workflow languages & systems being a great choice when developing computational pipelines.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "EuiphQCwUs0E"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***In This Training Material...***\n",
        "\n",
        "This training material aims to demonstrate how CWL tools and workflows are written.\n",
        "\n",
        "In the first section, we will learn how to write CWL tools. <br>\n",
        "\n",
        "In the second section, we will focus on CWL workflows. <br>\n"
      ],
      "metadata": {
        "id": "fqeMkBpCCywM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "x3ZomnjpuDNl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "# 2 - Setup\n",
        "\n",
        "Run each of the following cells. <br>\n",
        "These are needed to get the input data, and set up the software environment. <br>\n",
        "If your ***runtime disconnects*** at any point, you may need to re-run these.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "4bJ5NsmRCqDa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "step 1: runtime\n",
        "\n",
        "- To execute code, make sure your runtime is connected.<br>\n",
        "- In the top right hand corner of the screen, click `Connect` to acquire a runtime.  <br>\n",
        "- Once connected, you should see a RAM and DISK gauge.\n",
        "\n",
        "step 2: data + software  (below)\n",
        "\n",
        "- Once you have a runtime, we need to *get the data* and *install some software*.\n",
        "- The data is input data we will use to test our tool and workflow definitions. <br>\n",
        "It is cloned from a github repository.\n",
        "- The software is required to run CWL and execute the tools in our workflow. <br>We will need `java + cwltool` to run CWL, and will need `cutadapt`, `bwa`, `samtools` and `freebayes` to execute our pipeline.\n"
      ],
      "metadata": {
        "id": "TLHyBdNfpmd3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Get Session and Exercise Data\n",
        "%%capture\n",
        "# cwl files for learning & input data\n",
        "!sudo rm -r sample_data   # remove default google colab 'sample_data'\n",
        "!sudo rm -r session       # remove 'session' folder if exists\n",
        "!sudo rm -r exercise\n",
        "!git clone https://github.com/GraceAHall/cwl-workshop session   # clone  files\n",
        "!git clone https://github.com/screx/cwl-tutorial.git exercise  # clone files for the Hands-On-Exercise\n"
      ],
      "metadata": {
        "id": "KiodAPxc63sx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k4j9jenU9A_l"
      },
      "outputs": [],
      "source": [
        "#@title Install CWL\n",
        "\n",
        "%%capture\n",
        "### thanks to Sanjeev v (https://stackoverflow.com/questions/51287258/how-can-i-use-java-in-google-colab)\n",
        "\n",
        "# cwltool\n",
        "!pip install cwltool\n",
        "\n",
        "# cwlref-runner\n",
        "!pip install cwlref-runner\n",
        "\n",
        "# some magic so we can run cells as cwl\n",
        "from google.colab import _message\n",
        "\n",
        "def write_cell_above_to_file(search_term, filename):\n",
        "  cell = get_cell_above(search_term)\n",
        "  code_block = get_cell_code_block(cell)\n",
        "  with open(filename, 'w') as fp:\n",
        "    fp.writelines(code_block)\n",
        "\n",
        "def get_cell_above(search_term):\n",
        "  # Load the notebook JSON.\n",
        "  nb = _message.blocking_request('get_ipynb')\n",
        "\n",
        "  # Search for current markdown cell (using search term)\n",
        "  for i, cell in enumerate(nb['ipynb']['cells']):\n",
        "    if search_term in ''.join(cell['source']):\n",
        "      return nb['ipynb']['cells'][i - 1]\n",
        "\n",
        "def get_cell_code_block(cell):\n",
        "  # Get the code block in previous cell\n",
        "  cell_lines = cell['source']\n",
        "  code_block = []\n",
        "  in_block = False\n",
        "  for ln in cell_lines:\n",
        "    if '```' in ln:\n",
        "      in_block = not in_block  # boolean switch\n",
        "    else:\n",
        "      if in_block:\n",
        "        code_block.append(ln)\n",
        "  return code_block\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "# 3 - Tools\n",
        "\n",
        "In this section, we will build a simple tool using `echo`.\n",
        "\n",
        "If you're unfamiliar: `echo` prints any supplied text to the console. <br>\n",
        "The goal will be to demonstrate how tools work using ***inputs***, the ***command***, and ***outputs***.\n",
        "\n",
        ">***Note***<br>\n",
        ">Don't worry about the exact syntax for now - it's explained in detail below!\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "XohrJw8jw6Nr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.1 - <font color='#097969'>What are Tools?<font>\n",
        "\n",
        "A ***tool*** represents a task to perform.\n",
        "\n",
        "They are the smallest unit of computation in a workflow. <br>\n",
        "For example - a variant calling workflow might use `bwa mem` to align reads.<br>\n",
        "In this case, we need a ***tool*** which encapsulates `bwa mem` so it can be used in our workflow."
      ],
      "metadata": {
        "id": "EBvdiVTs-ivV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "<figure>\n",
        "<img src=\"https://raw.githubusercontent.com/GraceAHall/cwl-workshop/main/media/tool1.png\" width=\"60%\">\n",
        "<figcaption>Fig 6. Diagram of a tool. </figcaption>\n",
        "</figure>"
      ],
      "metadata": {
        "id": "KTVnmPnBygjE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "Tools consist of 3 main components.\n",
        "- <font color='red'>inputs</font> - data supplied from outside world\n",
        "- <font color='red'>command</font> - the dynamically generated CLI command to execute\n",
        "- <font color='red'>outputs</font> - data we will return to outside world\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Keep in mind - the command is ***separate*** from the inputs and outputs. <br>Tool inputs ***are not*** command inputs, and tool outputs ***are not*** command outputs.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "gM1w9SCwOgnB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This structure means Tools are encapsulated - they ***do not*** know anything about the outside world. <br>\n",
        "Because they don't depend on anything external, they are reusable.\n",
        "- A single tool can be used *multiple times* within a workflow\n",
        "- A single tool can be used in *multiple different* workflows"
      ],
      "metadata": {
        "id": "fats_yIIOhsv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "<figure>\n",
        "<img src=\"https://raw.githubusercontent.com/GraceAHall/cwl-workshop/main/media/tool2.png\" width=\"60%\">\n",
        "<figcaption>Fig 7. Dataflow before, during, and post command execution. </figcaption>\n",
        "</figure>\n",
        "\n"
      ],
      "metadata": {
        "id": "pqtYBoOH59mi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "When a tool is run, there are multiple stages.\n",
        "\n",
        "Gather inputs\n",
        "- At the start, each ***input*** is fed a value.  \n",
        "\n",
        "Execute command\n",
        "- After all necessary inputs have a value, a ***command*** is generated.\n",
        "- The command is then run, performing the desired processing.\n",
        "\n",
        "Collect outputs\n",
        "- The command may have produced some ***outputs*** during runtime.\n",
        "- If we have specified some of these for collection, they will be gathered as outputs.\n",
        "- These outputs can be returned to the user (after the workflow has finished), or can be passed to other tools as inputs.\n"
      ],
      "metadata": {
        "id": "a3tP-dAJ6wmP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "U-7haPMN-ZOK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.2 - <font color='#097969'>Our First CWL Tool<font>\n",
        "\n",
        "Here is an example of the most basic tool we can write in CWL.\n",
        "\n",
        "It has no inputs, no outputs, and the final command simply evaluates to `echo`.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "w2GH2EODniBE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: [echo, hello]\n",
        "inputs: []\n",
        "outputs: []\n",
        "```\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "rsBYGytO9-Kk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:overview1'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl'"
      ],
      "metadata": {
        "id": "Z6s84vex7C9w",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Congratulations! ✨✨✨\n",
        "\n",
        "You just ran your first CWL tool.\n",
        "\n"
      ],
      "metadata": {
        "id": "y1gGD9AecwO-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's explain a little about the code you see.\n",
        "\n",
        "- The first two lines are <font color='red'>metadata</font>. These specify this is a CWL `CommandLineTool` (tool), and the CWL version is v1.2.\n",
        "- The `baseCommand` is an array of strings (text) which will appear at the start of the <font color='red'>command</font>.\n",
        "- `inputs` is a list of the <font color='red'>inputs</font> to this tool. We use them to customise our command.\n",
        "- `outputs` is a list of the <font color='red'>outputs</font> this tool produces. We tell CWL to collect specific outputs the software produces during execution.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "hN1coezTq0yh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ">***Note***<br>\n",
        ">CWL has 3 release versions - CWL v1.0, v1.1 and v1.2. <br>\n",
        ">The current version is v1.2 and should be used. <br>\n",
        ">Older versions may have slightly different syntax."
      ],
      "metadata": {
        "id": "RigHDFnI6330"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.2.2 - <font color='#097969'> Another way to implement the Echo tool: echo.cwl <font>\n",
        "\n",
        "\n",
        "Here we are going to implement an alternative method for implementing `echo tool' in CWL, using a YAML file to specify the message that we want to print out using echo.\n",
        "\n",
        "The YAML file store the actual values that later CWL worfkow will use ( in inputs descriptions)."
      ],
      "metadata": {
        "id": "nn2VsYCi5WZJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "# helloworld.yml\n",
        "\n",
        "message: Hello, world!\n",
        "```"
      ],
      "metadata": {
        "id": "4aLCMYXd58VO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:overview1.2'\n",
        "write_cell_above_to_file(search_term, '/tmp/helloworld.yml')"
      ],
      "metadata": {
        "id": "BJzAaToe6FzO",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        " Creating the CWL workflow - echo.cwl"
      ],
      "metadata": {
        "id": "lhY2fLrj7n1s"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "#!/usr/bin/ cwltool\n",
        "\n",
        "cwlVersion: v1.0\n",
        "class: CommandLineTool #this says that the CWL file describes a command line tool\n",
        "baseCommand: echo #this says we are using the echo command\n",
        "\n",
        "inputs:\n",
        "  message: # the unique identifier to the input\n",
        "    type: string # the type of input parameter that is passed in\n",
        "    inputBinding:\n",
        "        position: 1\n",
        "    label: message to print to stdout\n",
        "\n",
        "outputs: [] # here there are no outputs as we are simply printing to stdout\n",
        "```"
      ],
      "metadata": {
        "id": "Z144hxxKutEI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:overview1.3'\n",
        "write_cell_above_to_file(search_term, '/tmp/echo.cwl')\n",
        "\n",
        "!cwltool '/tmp/echo.cwl'  '/tmp/helloworld.yml'"
      ],
      "metadata": {
        "id": "MpxishUe5us2",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "\n",
        "For the echo tool we see that it takes one input parameter message and its type is string and we can define a YAML file, helloworld.yml"
      ],
      "metadata": {
        "id": "5Sk3LW-MA8Vu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.3 - <font color='#097969'>BaseCommand<font>\n",
        "\n",
        "Let's focus on the `baseCommand`.\n",
        "\n",
        "```\n",
        "baseCommand: echo              # single item\n",
        "baseCommand:                   # multiple items (list format)\n",
        "  - program1\n",
        "  - program2        \n",
        "baseCommand: [program1, program2]        # multiple items (bracket format)\n",
        "baseCommand: [\"echo\", \"hi\"]    # quoting is fine\n",
        "baseCommand: [echo, hello] # without quoting is fine too\n",
        "```\n",
        "\n",
        "The `baseCommand` is a array of strings which appear at the start of the command. <br>\n",
        "Above, we see that either *list format* or *bracket format* is allowed. <br>\n",
        "These formats are explained in the [Tools - YAML](#tools_-_yaml) section.\n",
        "\n",
        "The best approach is to use `[]`. <br>\n",
        "This is the most clear, as it visually shows that the `baseCommand` is an *array of strings.*<br>\n",
        "\n",
        "The exception is where `baseCommand` is a single item.<br>\n",
        "In this case, we can ignore using `[]` because CWL is smart enough to convert it to a single item array. <br>\n",
        "This is what we are seeing in the `baseCommand: echo` example above.\n",
        "\n",
        "Run the CWL cell below to see `baseCommand` in action with multiple items."
      ],
      "metadata": {
        "id": "mIoxNmXdupPT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "inputs: []\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "scsVJ4r5s65s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:overview1.1'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl'"
      ],
      "metadata": {
        "id": "T0Igv-vDtI_k",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "hoE6knDZpsJq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.4 - <font color='#097969'>CWL Output Log<font>\n",
        "\n",
        "Before we continue, lets look at that CWL output. <br>\n",
        "The CWL output provides useful information, summarised in the example below.\n",
        "```\n",
        "INFO [job /tmp/tmpfile.cwl] /tmp/4g1ax89r$ echo \\    # command which was executed  \n",
        "    hello \\                                          # command which was executed\n",
        "    there!                                           # command which was executed\n",
        "hello there!                                         # stdout\n",
        "INFO [job /tmp/tmpfile.cwl] completed success        # tool status\n",
        "{}                                                   # tool outputs\n",
        "INFO Final process status is success                 # final status\n",
        "```\n",
        "The CWL output spreads the ***command*** over multiple lines using `\\`.<br>\n",
        "In the text above, we can infer the ***command*** was `echo hello there!`, <br>\n",
        "***stdout*** was `hello there!`.  \n",
        "\n",
        "For the rest of this section, keep an eye on the *command* and *stdout*."
      ],
      "metadata": {
        "id": "d8_IeHuiw0IE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "kfbEhZSQpt1g"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.5 - <font color='#097969'>Adding an Input<font>\n",
        "\n",
        "Currently, our CWL tool doesn't do anything.\n",
        "\n",
        "The command line simply evaluates to `echo` - theres nothing to actually print! <br>\n",
        "Let's add a new ***input*** to our tool definition so we can provide some text after `echo`. <br>\n",
        "Our new input's name is `message`, it accepts text (`type: string`), and appears directly after the `baseCommand` (`position: 1`)."
      ],
      "metadata": {
        "id": "qqtGu7_qw2Iv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  message:                        # new\n",
        "    type: string                  # new\n",
        "    inputBinding:                 # new\n",
        "      position: 1                 # new\n",
        "\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "k-g3DSQIfhCj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:overview2'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "import os\n",
        "message = 'Hello there!' #@param {type:\"string\"}\n",
        "os.environ['MSG'] = message\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' --message \"$MSG\""
      ],
      "metadata": {
        "id": "0pB8OxB3fnyR",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can confirm this has been successful by looking at the CWL output.\n",
        "- The ***command*** (line 3) was `echo 'Hello there!'`\n",
        "- The ***stdout*** (line 4) was `Hello there!`"
      ],
      "metadata": {
        "id": "xP9fERA60lPT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">Change the text supplied to `message` and re-run the cell above.<br>\n",
        ">Does the output change?\n"
      ],
      "metadata": {
        "id": "WMAl8WmzgP3p"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "aQFejb5wpwPn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.6 - <font color='#097969'>Adding an Output<font>\n",
        "\n",
        "The final thing to add is an output.\n",
        "\n",
        "Except in special cases, tools will always have ***inputs*** and ***outputs*** <br>\n",
        "Currently, we are not collecting any outputs from this tool (regardless of anything being produced!)<br>\n",
        "Lets collect the text being printed to `stdout` as an output.\n",
        "\n"
      ],
      "metadata": {
        "id": "kHAkwIBQlSwG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  message:            \n",
        "    type: string      \n",
        "    inputBinding:     \n",
        "      position: 1     \n",
        "\n",
        "outputs:\n",
        "  message_out:                 # new\n",
        "    type: stdout               # new\n",
        "```"
      ],
      "metadata": {
        "id": "C5lsDxZrlZjR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:overview3'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "import os\n",
        "message = 'Hello there!' #@param {type:\"string\"}\n",
        "os.environ['MSG'] = message\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' --message \"$MSG\""
      ],
      "metadata": {
        "id": "lKtAVwMfkr7a",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "Instead of a blank `{}`, we now see some information about our ***tool output***.<br>\n",
        "We can see that there is an output called `message_out` which is a `File` amongst other info. <br>\n",
        "This confirms we have collected our first output, which we got from `stdout`.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "2REoDDMfy2GM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we have witnessed a basic ***CWL tool***, and have gotten a little familiar with its ***syntax***, we will start learning how to write these tools for ourselves. <br>"
      ],
      "metadata": {
        "id": "vYIJIGExemoP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "swyPaL9R_JlG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "# 4 - Tools: YAML\n",
        "\n",
        ">***Note*** - If you are familiar with YAML, feel free to skip this section\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "KHpyCQj_wzp1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before diving in, lets explain the ***format*** of the CWL code above.\n",
        "\n",
        "CWL is written with [YAML](https://learnxinyminutes.com/docs/yaml/). It is highly similar to JSON, but easier to work with because it is more human readable. <br>\n",
        "YAML is based on `key: value` pairs, where each `key` is a string (text) and each `value` is a primitive type, an array, or an object."
      ],
      "metadata": {
        "id": "G6grtdkA_wnN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "X2iHE69y_W_t"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.1 - <font color='#097969'>Key Value Pairs<font>\n",
        "\n",
        "YAML is built on `key: value` pairs.\n",
        "\n",
        "YAML `keys` are always ***strings*** (text), while the associated `value` can take multiple forms. <br>\n",
        "Have a look at the `key: value` pairs below.<br>\n",
        "Each `key` is a meaningful name, and each `value` is the value that name takes.<br>\n",
        "It makes sense to read that this person's name is `Grace`, their age is `111` years old, and their height is `184.5cm`.\n",
        "\n",
        "```\n",
        "name:   Grace         \n",
        "age:    111       \n",
        "height: 184.5     \n",
        "```"
      ],
      "metadata": {
        "id": "Sq1unLpiptP0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "TlZ2XBTw_m_T"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.2 - <font color='#097969'>Arrays<font>\n",
        "\n",
        "Rather than a single `value`, sometimes a key should be associated with multiple `values`. <br>\n",
        "In this case we can use an ***array***. <br>\n",
        "If you're familiar with python, an array is similar to a list - the difference is that each item in an array must have the same datatype.\n",
        "\n",
        "```\n",
        "hobbies:\n",
        "  - programming\n",
        "  - fashion\n",
        "  - anime\n",
        "```\n",
        "\n",
        "The YAML above states that there are 3 hobbies. <br>\n",
        "Each hobby appears on a new line, and starts with a dash `-`.  \n",
        "\n",
        "> ***Note the indentation!*** <br>\n",
        "By indenting 2 spaces, we specify this element ***belongs*** to the element above.\n",
        "\n",
        "This structure is the most common format used in CWL, but arrays can also be specified in-line if desired:\n",
        "\n",
        "```\n",
        "hobbies: [programming, fashion, anime]\n",
        "```\n",
        "\n",
        "This format is ok here, but may result in bad readability for other situations. <br>\n",
        "It is generally suggested to use the ***indented format***, except in specific cases.\n"
      ],
      "metadata": {
        "id": "EaILVvvpptRC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "E6Hk5rtS_pRk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.3 - <font color='#097969'>Objects<font>\n",
        "\n",
        "The final type a `value` can take is an ***object***. <br>\n",
        "An object is an entity with its own `key: value` pairs. These key value pairs belong to that object.  <br>\n",
        "In the example below, `friends` and `josie` are both objects.\n",
        "- `friends` is an object because the key `josie` is contained inside.\n",
        "- `josie` is an object because the keys `name`, `age`, `height` are contained inside.\n",
        "\n",
        "```\n",
        "friends:\n",
        "  josie:\n",
        "    name:   josie\n",
        "    age:    21\n",
        "    height: 175.7\n",
        "```\n",
        "\n",
        "Remember that by dropping to a new line and indenting 2 spaces, we specify this element ***belongs*** to the element above. <br>\n",
        "Therefore, `josie` is a sub-object of `friends`, and `name`, `age`, `height` belong to `josie`!"
      ],
      "metadata": {
        "id": "DcsNPoLEprXW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "Always pay close attention to the level of indentation. Mistakes in indentation cause the YAML to have different meaning. <br>\n",
        "Try to identify the mistake in the YAML below:\n",
        "\n",
        "```\n",
        "friends:\n",
        "  josie:\n",
        "    name:   josie\n",
        "    age:    21\n",
        "  height: 175.7\n",
        "```\n",
        "\n",
        "The indentation of `height` is wrong! it should be shifted right 2 spaces.  <br>\n",
        "The code above reads that there are 2 `friends` - `josie` and `height`, which is a little odd!"
      ],
      "metadata": {
        "id": "jBcGxsq2AQQ4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "IvXevr8j_qEM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.4 - <font color='#097969'>Be careful of quotes!<font>\n",
        "\n",
        "Types in YAML are auto-interpreted. <br>\n",
        "Most individual values will be interpreted as strings, unless they look like numbers (integer or float). <br>\n",
        "For example - the `baseCommand` of a CWL tool must be a single string, or an array of strings.\n",
        "\n",
        "```\n",
        "baseCommand: echo             # ok!  (single item)\n",
        "baseCommand: \"echo hi\"          # error  (multiple items, not in an array)\n",
        "baseCommand: [echo, \"42\"]     # ok!  (echo will be interpreted as string)\n",
        "baseCommand: [\"echo\", \"42\"]   # ok!  (each item is string)\n",
        "baseCommand: [echo, 42]       # error  (42 is interpreted as int)\n",
        "```\n",
        "\n",
        "In the first example, `42` is interpreted as an integer, which would cause a CWL error. <br>\n",
        "In the second example, `\"42\"` is ok because we are specifying `42` is a string using quotes.\n",
        "\n"
      ],
      "metadata": {
        "id": "vnWBrOJO0KfV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "4Mndrm5U_tv6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.2 - <font color='#097969'>The Basics<font>\n",
        "\n",
        "Inputs provide a gateway to outside world.\n",
        "\n",
        "They allow us to provide some ***data*** to the tool, which can then be used within the command.\n",
        "\n",
        "```\n",
        "inputs:\n",
        "  message:\n",
        "    type: string            \n",
        "```\n",
        "\n",
        "Tool inputs are added as sub-objects inside the `inputs` key.\n",
        "\n",
        "Inputs must have a unique name (in this case `message`). <br>\n",
        "Inputs must also specify the `type` of data they will recieve (in this case `string`).\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "DJeNytHb5lVr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***New concept:*** `types`\n",
        "> Each tool input / output must specify a ***type***.<br>\n",
        "> This lets CWL check the data supplied to this input during runtime looks correct. <br>\n",
        "> Common types include `string`, `integer`, `float`, `boolean` and `file`."
      ],
      "metadata": {
        "id": "U-BNSIpE2cA8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets explore how to create and manipulate inputs using the `echo` command.\n",
        "\n",
        "Below is an `echo` tool which has an input called `message`.<br>\n",
        "This `message` input should accept a `string`, so we can print something to the console with echo. <br>\n",
        "The command we wish to produce is `echo <message>`."
      ],
      "metadata": {
        "id": "MyfPVB3f2Z4Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  message:                       \n",
        "    type: string                 \n",
        "\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "I-suruCy5YyP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:inputBinding0'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "import os\n",
        "message = 'hi there!' #@param {type:\"string\"}\n",
        "os.environ['MSG'] = message\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' --message \"$MSG\""
      ],
      "metadata": {
        "id": "6Iep_v8J5qNv",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "😐 Hmm... that worked, but didn't go as planned!\n",
        "\n",
        "Our message isn't appearing in the ***command*** (line 3) and isn't printed to ***stdout*** (line 4). <br>\n",
        "The reason - <font color='red'>tool inputs don't automatically appear in the command!</font> <br>\n",
        "To make them appear in the command, we need to specify an `inputBinding`.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "AaNRQGqppXuV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "TmvyhWx1xkRc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.3 - <font color='#097969'>InputBinding<font>\n",
        "\n",
        "An `inputBinding` customises how something is presented in the ***command***.\n",
        "\n",
        "```\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  message:                 \n",
        "    type: string           \n",
        "    inputBinding:          \n",
        "      position: 1\n",
        "      prefix: --my-message\n",
        "```\n",
        "\n",
        "The `baseCommand` always appears at the start, whereas each tool `input` appears according to its ***inputBinding***. <br>\n",
        "In the example above, the final command would be `echo --my-message <message>`\n",
        "\n",
        "An `inputBinding` has multiple attributes, but for now lets just focus on `position`.\n"
      ],
      "metadata": {
        "id": "vX-lcu9TAxxI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "4wTWwEOsxhI-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the example below, it becomes clear that `arguments` is an ***array***. <br>\n",
        "Each `argument` is declared using a `dash -`, and any lines including and below the dash are part of that argument. <br>\n",
        "To supply a value, `valueFrom` is used.\n",
        "\n",
        "`prefix`, `position`, `separate` can also be used, and behave in the same way as `inputs`. <br>\n",
        "The ***order*** of these attributes does not matter.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "yTPd4aMAPzt9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs: []\n",
        "\n",
        "arguments:\n",
        "  - valueFrom: \"one\"\n",
        "    position: 1\n",
        "    prefix: --argument\n",
        "\n",
        "  - prefix: --argument\n",
        "    valueFrom: two\n",
        "    position: 2\n",
        "\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "RvLdmI13QjUh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:argument2'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl'"
      ],
      "metadata": {
        "id": "hIC5R0QvRUGl",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "When supplying a value using `valueFrom`, make sure it is interpreted as a string!<br>\n",
        "If anything else, CWL will report an error. <br>\n",
        "This can be done by wrapping the value in quotes, or allowing CWL to auto-interpret the type as string where possible.  \n",
        "\n",
        "```\n",
        "arguments:\n",
        "  - valueFrom: 42                # not ok! interpreted as int\n",
        "  - valueFrom: \"42\"              # ok! quoted = string\n",
        "```"
      ],
      "metadata": {
        "id": "KjHii10x7CGq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ">***Note***<br>\n",
        ">`valueFrom` can be used in a number of places, not just `arguments`."
      ],
      "metadata": {
        "id": "lzIdOR7cWEVG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "hmmmm, `position`... `prefix`... that sounds familiar.\n",
        "\n",
        "That's because each `argument` is actually just a plain old `inputBinding`!\n",
        "<br>\n",
        "They don't have a name, nor a `type`, but still need an `inputBinding` to appear in the command."
      ],
      "metadata": {
        "id": "ZoxniBwF7TKu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "🔥 <font color='red'>**CHALLENGE**</font>\n",
        "\n",
        "> ```\n",
        "arguments:\n",
        "  - valueFrom: \"200\"\n",
        "    prefix: --max-length\n",
        "  - prefix: --min-length\n",
        "    valueFrom: \"100\"\n",
        "  - valueFrom: \"30\"\n",
        "    prefix: --quality-cutoff\n",
        "  - prefix: --output\n",
        "    valueFrom: reads_1.trimmed.fq\n",
        "    position: 1\n",
        "  - position: 2\n",
        "    prefix: --paired-output\n",
        "    valueFrom: reads_2.trimmed.fq    \n",
        "```"
      ],
      "metadata": {
        "id": "6Lb8XCML9y3O"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown How many `arguments` are in the code snippet above?\n",
        "\n",
        "correct_ans = 5\n",
        "user_ans = 0 #@param {type:\"integer\"}\n",
        "\n",
        "def check_results(correct_answers, user_answers):\n",
        "  for user, correct in zip(correct_answers, user_answers):\n",
        "    if user != correct:\n",
        "      return '😐 try again'\n",
        "  return '✅ correct!'\n",
        "\n",
        "print(check_results([correct_ans], [user_ans]))"
      ],
      "metadata": {
        "id": "EtzItznw-BI4",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "D8b4Ez15gWxc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.9 - <font color='#097969'>Default Values (extension)<font>\n",
        "\n",
        "`Inputs` can have a ***default*** value. <br>\n",
        "If no value is supplied to the `input` during execution, the default will take its place in the command.\n",
        "\n"
      ],
      "metadata": {
        "id": "P0AYjHRmH8cZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is different from an ***optional*** tool input:\n",
        "\n",
        "- If an optional input is not supplied a value, it will be missing from the command.\n",
        "- If an input with a default is not supplied a value, it will still be present in the command."
      ],
      "metadata": {
        "id": "tq-kVSQF73ZO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "A default value for an input is specified using `default: <default>`.<br>\n",
        "`default` is an attribute of the `input` (same as `type`), rather than the input's `inputBinding`.\n"
      ],
      "metadata": {
        "id": "iaTjT6u37v2z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  threads:\n",
        "    type: int\n",
        "    default: 8\n",
        "    inputBinding:\n",
        "      prefix: --threads\n",
        "\n",
        "outputs: []\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "yF_q8msVfqY4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:inputDefault1'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "null = None\n",
        "threads = 3 #@param {type:\"raw\"}\n",
        "\n",
        "import os\n",
        "if isinstance(threads, int):\n",
        "  os.environ['THREADS'] = '--threads ' + str(threads)\n",
        "else:\n",
        "  os.environ['THREADS'] = ''\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' $THREADS"
      ],
      "metadata": {
        "cellView": "form",
        "id": "H4NJSL7QfwMA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "What about ***optional*** inputs with a ***default***?\n",
        "\n",
        "This is fine! That said, it is a little redundant.  <br>\n",
        "It will behave exactly the same regardless of whether it is marked ***optional*** or not. <br>\n",
        "This is because ***default*** means the input will always have a value, even if we don't supply one.  \n",
        "\n",
        "Run the cell below to check."
      ],
      "metadata": {
        "id": "GmFEOr0N8_K6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  int_optional:\n",
        "    type: int?\n",
        "    inputBinding:\n",
        "      prefix: --int-optional\n",
        "\n",
        "  int_default:\n",
        "    type: int\n",
        "    default: 8\n",
        "    inputBinding:\n",
        "      prefix: --int-default\n",
        "\n",
        "  int_optional_default:\n",
        "    type: int?\n",
        "    default: 8\n",
        "    inputBinding:\n",
        "      prefix: --int-optional-default\n",
        "\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "Xiu_mhKJdXPm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:inputDefault2'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "null = None\n",
        "int_optional = 3 #@param {type:\"raw\"}\n",
        "int_default = 2 #@param {type:\"raw\"}\n",
        "int_optional_default = 1 #@param {type:\"raw\"}\n",
        "\n",
        "import os\n",
        "if isinstance(int_optional, int):\n",
        "  os.environ['OPTTHREADS'] = '--int_optional ' + str(int_optional)\n",
        "else:\n",
        "  os.environ['OPTTHREADS'] = ''\n",
        "\n",
        "if isinstance(int_default, int):\n",
        "  os.environ['DEFTHREADS'] = '--int_default ' + str(int_default)\n",
        "else:\n",
        "  os.environ['DEFTHREADS'] = ''\n",
        "\n",
        "if isinstance(int_optional_default, int):\n",
        "  os.environ['OPTDEFTHREADS'] = '--int_optional_default ' + str(int_optional_default)\n",
        "else:\n",
        "  os.environ['OPTDEFTHREADS'] = ''\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' $OPTTHREADS $DEFTHREADS $OPTDEFTHREADS"
      ],
      "metadata": {
        "cellView": "form",
        "id": "sBSQ2QBPdtSl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.10 - <font color='#097969'>Flags (extension)<font>\n",
        "\n",
        "Some software arguments don't have a value. <br>\n",
        "They are either 'on' or 'off' depending on if they are present in the command or not.  <br>\n",
        "For example, `--conservative`,  or `--compress-output`.\n",
        "\n",
        "This type of argument is also known as a boolean 'flag'.  <br>\n",
        "Their `type` is `boolean`, and can either be `true` or `false`. <br>\n",
        "If set to `true`, the flag appears in the command line. If `false`, the flag does not appear.\n",
        "\n",
        "We can see them in action below."
      ],
      "metadata": {
        "id": "kAZp4KDnGJfb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ">***NOTE***\n",
        ">\n",
        ">Boolean flags are `false` by default"
      ],
      "metadata": {
        "id": "Pl6tFFyQ_0-g"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  conservative:\n",
        "    type: boolean\n",
        "    default: false\n",
        "    inputBinding:\n",
        "      prefix: --conservative\n",
        "\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "VWKZKpdk15Lv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:inputFlag1'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "conservative = 'true' #@param [\"false\", \"true\"]\n",
        "\n",
        "import os\n",
        "if conservative == 'true':\n",
        "  os.environ['CONS'] = '--conservative'\n",
        "else:\n",
        "  os.environ['CONS'] = ''\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' $CONS"
      ],
      "metadata": {
        "cellView": "form",
        "id": "WSUlwWNU2fqJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">Change the value for `conservative` from `true` to `false` in the form above then re-run the cell.<br>\n",
        ">Does `--conservative` still appear?"
      ],
      "metadata": {
        "id": "1YBdWTARAB_Q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "It is often convenient to mark flags as optional. <br>\n",
        "Marking them optional result in more concise code, and means we don't need to always provide a `true` / `false` value.\n",
        "\n"
      ],
      "metadata": {
        "id": "7YGp9Wa414Sp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  conservative:\n",
        "    type: boolean?\n",
        "    inputBinding:\n",
        "      prefix: --conservative\n",
        "\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "R-ZFnF1F69Vz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:inputFlag2'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "conservative = \"null\" #@param [\"null\", \"false\", \"true\"]\n",
        "\n",
        "import os\n",
        "if conservative == \"true\":\n",
        "  os.environ['CONS'] = '--conservative'\n",
        "else:\n",
        "  os.environ['CONS'] = ''\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' $CONS"
      ],
      "metadata": {
        "cellView": "form",
        "id": "XYbmUwxU6-Rk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.11 - <font color='#097969'>Parameter References (extension)<font>\n",
        "\n",
        "<font color=\"red\">Parameter references</font> allow us to access ***properties*** of objects within our CWL tool definition. <br>\n",
        "The most simple way to explain is by looking at some CWL.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "KjgS_EhLZ2LI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "In certain locations in a CWL tool / workflow definition, we can <font color=\"red\">refer</font> to other a CWL <font color=\"red\">parameters</font> using their names.\n",
        "\n",
        "`$()` is the syntax for these <font color=\"red\">parameter references</font>.\n",
        "\n",
        "When CWL sees `$()`, it knows that the text inside refers to a variable. <br>\n",
        "In the case above, our variable is `$(inputs.infile.basename)`.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "17pTd9lKoQOV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "\n",
        "We can understand what this variable means by looking at YAML.\n",
        "\n",
        "```\n",
        "friends:\n",
        "  josie:\n",
        "    name:   josie\n",
        "    age:    21\n",
        "    height: 175.7\n",
        "```\n",
        "\n",
        "Data within YAML files can be accessed using bracket or dot notation.\n",
        "\n",
        "Taking the above as an example, we can access data about `josie` via\n",
        "- bracket notation - `friends['josie']` or\n",
        "- dot notation - `friends.josie`\n",
        "\n",
        "To further illustrate - if we wanted to know `josie`'s age, we could write `friends.josie.age`.\n"
      ],
      "metadata": {
        "id": "y4QVS1qZfLms"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "CWL allows us to access this information in a similar way.<br>\n",
        "If we want to access attributes of our `inputs`, we can write `inputs.<name>` where `<name>` is the input we are interested in.\n",
        "\n",
        "If we see `$(inputs.infile.basename)`, we know that we are accessing the `infile` input.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "aqLap1IFiB34"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***file.basename***\n",
        "\n",
        "At this point, there is still a question - what is `basename`? <br>\n",
        "We can't see `basename` in the CWL definition - so what is it?\n",
        "\n",
        "`basename` is a special attribute of `File` type data.\n",
        "\n",
        "It is created by CWL at runtime, by looking at the data which was passed to the `File`. <br>\n",
        "`File` data has a number of these hidden attributes.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "1-YZLjZBj3W1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "# 6 - Tools: Outputs\n",
        "\n",
        ">📁 ***Files***<br>\n",
        ">For this section, keep the file explorer open (sidebar -> 📁 files).<br>\n",
        ">We will look at the files produced by CWL during tool execution.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "ld-vckLi_pVY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color='red'>Outputs</font> are data we wish to broadcast to the outside world.\n",
        "\n",
        "During execution, the software executed by the ***command*** will usually produce some data.\n",
        "This is either written to stdout, or a file. <br>\n",
        "For the data we wish to collect and present to the outside world, we can capture these using `outputs`. <br>\n",
        "\n",
        "***Tool outputs*** can also become ***workflow outputs***, as is demonstrated in the workflows section.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "wHRxJHzq_wYZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ">***Note*** <br>\n",
        ">CWL will only collect the `outputs` that we specify. <br>\n",
        ">If our command produces 5 files, but we don't specify any `outputs`, the tool won't return anything! <br>\n",
        ">Those files will be forgotten.\n",
        "\n"
      ],
      "metadata": {
        "id": "m8ICcj54_7Qx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color='red'>Outputs</font> are similar to inputs in a few ways. <br>\n",
        "The must have a unique name, and an appropriate `type`.<br>\n",
        "This is almost always `stdout` or `File`."
      ],
      "metadata": {
        "id": "zAVI6wBWAFvD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 6.1 - <font color='#097969'>Stdout<font>\n",
        "\n",
        "So far, we have been using ***stdout*** to collect outputs. <br>\n",
        "We do this by setting the `type` of an output to `stdout`.\n",
        "\n",
        "```\n",
        "outputs:\n",
        "  example_out:\n",
        "    type: stdout\n",
        "```\n",
        "\n",
        "The above will capture any text written to stdout, and will present this text as the `example_out` output. <br>\n",
        "But what is actually happening to facilitate stdout capture?"
      ],
      "metadata": {
        "id": "yyzx3ebPAIhd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        "\n",
        ">Open the file explorer (sidebar -> 📁 files tab) to view the local folder structure, then run the cell below <br>\n",
        ">Note any new files which are created.<br>\n",
        ">If one appears, ***open it*** by double clicking.\n",
        ">\n",
        ">***Note***<br>\n",
        ">You may need to hit the folder 'refresh' button at the top of the file explorer to see newly created files. <br>"
      ],
      "metadata": {
        "id": "MdAV_HJxAUta"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "\n",
        "inputs:\n",
        "  message:\n",
        "    type: string\n",
        "    inputBinding:\n",
        "      position: 1\n",
        "\n",
        "outputs:\n",
        "  example_out:\n",
        "    type: stdout\n",
        "```\n"
      ],
      "metadata": {
        "id": "4IxdxRwfAfMQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:stdout1'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "message ='this is my message' #@param {type: 'string'}\n",
        "\n",
        "import os\n",
        "os.environ['MSG'] = str(message)\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' --message \"$MSG\""
      ],
      "metadata": {
        "id": "Y7nFob5CATWM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "You should have seen a new file (with a long random name) appear. It will correspond to the \"basename\" in the CWL output info for \"example_out\".\n",
        "This temporary file contains the contents we expect in `example_out`!\n",
        "\n",
        "From the task above, you may have worked out the mechanic. <br>\n",
        "Under the hood, CWL wrote stdout to a temporary file during execution, then collected that file as an output.\n"
      ],
      "metadata": {
        "id": "T69ldUDlA4eu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Rather than a long random temp name, we can specify a filename for this file using the `stdout` key. <br>\n",
        "Run the cell below, noting the name of the file produced. It should be `output.txt`.  \n",
        "\n",
        "Feel free to edit the CWL to change `stdout: output.txt` to whatever you like."
      ],
      "metadata": {
        "id": "pr5sXD0RBBlj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "stdout: output.txt       # new\n",
        "\n",
        "inputs:\n",
        "  message:\n",
        "    type: string\n",
        "    inputBinding:\n",
        "      position: 1\n",
        "\n",
        "outputs:\n",
        "  example_out:\n",
        "    type: stdout\n",
        "```"
      ],
      "metadata": {
        "id": "x1lunAS4BFrw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:stdout2'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "message ='this is my message' #@param {type: 'string'}\n",
        "\n",
        "import os\n",
        "os.environ['MSG'] = str(message)\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' --message \"$MSG\""
      ],
      "metadata": {
        "cellView": "form",
        "id": "kDBoIrzPBKOY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Refresh the files in the file explorer, and confirm `output.txt` was created.<br>\n",
        "Open it by double clicking, to confirm the contents of ***stdout*** were piped to this file."
      ],
      "metadata": {
        "id": "ojnIHLkiBZL7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 6.2 - <font color='#097969'>outputBinding (File Outputs)<font>"
      ],
      "metadata": {
        "id": "x9hWUa6gBd2d"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "From the above we can see that CWL can collect files as outputs. <br>\n",
        "Aside from stdout, we can specify any file we want for collection.\n",
        "\n",
        "This is performed using the `outputBinding` key.\n",
        "\n",
        "```\n",
        "outputs:\n",
        "  out1:\n",
        "    type: File\n",
        "    outputBinding:\n",
        "      glob: hello.txt\n",
        "```\n",
        "\n",
        "`outputBinding` instructs CWL how to collect a file (or multiple files).\n",
        "\n",
        "The most important attribute is `glob`, which allows us to specify the target file to collect.\n",
        "\n",
        "In the example below, we are extracting a tar archive. <br>\n",
        "The tar contains two files: `hello.txt` and `goodbye.txt`. The tar archive will be extracted, producing these files. <br>Imagine this is bioinformatics software which produces output files as it runs.\n",
        "\n",
        "Using `glob`, we have specified that the `hello.txt` file should be collected as the `out1` output."
      ],
      "metadata": {
        "id": "srhrIgNSBgqH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: [tar, --extract]\n",
        "\n",
        "inputs:\n",
        "  tarfile:\n",
        "    type: File\n",
        "    inputBinding:\n",
        "      prefix: --file\n",
        "\n",
        "outputs:\n",
        "  out1:\n",
        "    type: File\n",
        "    outputBinding:\n",
        "      glob: hello.txt\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "xocCdaczB8WM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:outputBinding1'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "tarfile = 'session/data/hello.tar' #@param ['session/data/hello.tar']\n",
        "\n",
        "import os\n",
        "os.environ['TARFILE'] = str(tarfile)\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' --tarfile $TARFILE"
      ],
      "metadata": {
        "cellView": "form",
        "id": "GWbZOJYCCFl4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">The tar archive `session/data/hello.tar` has another file inside. <br>\n",
        ">It's filename is `goodbye.txt`.\n",
        ">\n",
        ">Let's collect this file using a new output called `out2`. <br>\n",
        ">Edit the new `out2` output below, by setting up its `type` and `outputBinding` to collect the `goodbye.txt` file."
      ],
      "metadata": {
        "id": "8M2Uo_nuDET4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: [tar, --extract]\n",
        "\n",
        "inputs:\n",
        "  tarfile:\n",
        "    type: File\n",
        "    inputBinding:\n",
        "      prefix: --file\n",
        "\n",
        "outputs:\n",
        "  out1:\n",
        "    type: File\n",
        "    outputBinding:\n",
        "      glob: hello.txt\n",
        "\n",
        "  out2:\n",
        "    # fill in\n",
        "```"
      ],
      "metadata": {
        "id": "6mI75EW7DJfH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown <font size='4'>Run the cell above</font>\n",
        "search_term = 'id:outputBinding2'\n",
        "write_cell_above_to_file(search_term, '/tmp/tmpfile.cwl')\n",
        "\n",
        "tarfile = 'session/data/hello.tar' #@param ['session/data/hello.tar']\n",
        "\n",
        "import os\n",
        "os.environ['TARFILE'] = str(tarfile)\n",
        "\n",
        "!cwltool '/tmp/tmpfile.cwl' --tarfile $TARFILE"
      ],
      "metadata": {
        "cellView": "form",
        "id": "OiYVv-PcDQuq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "# 7 - Workflows"
      ],
      "metadata": {
        "id": "wvsLoJYTh_EH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7.1 - <font color='#097969'>Overview<font>\n",
        "\n",
        "Workflows combine multiple tools to perform a more comprehensive analysis.\n",
        "\n",
        "Rather than just running a single tool, we often want to run a group of tools in a pipeline.\n",
        "\n",
        "Take variant calling as an example - we need to perform read QC, align reads to a reference genome, then identify variants. <br>\n",
        "These three main steps can be combined end to end in a workflow. <br>\n",
        "Our workflow <font color='red'>inputs</font> are raw reads and a reference genome, and our workflow <font color='red'>outputs</font> are the identified variants."
      ],
      "metadata": {
        "id": "sIdPY8zHRXqC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Files & the File Explorer***\n",
        "\n",
        "From this point, we will start using Google Colab differently. <br>\n",
        "Previously we have been focusing on the notebook. From here, we will use colab as an IDE. <br>\n",
        "\n",
        "Instead of writing CWL inside notebook cells, we will switch to writing CWL as proper files. <br>\n",
        "We will also be looking at the files produced when we run tools / workflows.  \n",
        "\n",
        "To start, open the <font color='red'>file explorer</font> (files) in the left-hand sidebar. <br>\n",
        "You should see a folder called `session` and a few files. <br>\n",
        "\n"
      ],
      "metadata": {
        "id": "nca8U8bqYyWf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Running CWL via CLI***\n",
        "\n",
        "Up to this point, the mechanism of running CWL was hidden to avoid complexity. Now it's time to learn this for ourselves.\n",
        "\n",
        "An <font color='red'>execution engine</font> is used to run CWL tool and workflow files. From here, we will be responsible for creating / editing CWL files, and executing those files using the `cwltool` engine.\n",
        "\n",
        "In the paragraph above covering *Files & the File Explorer*, we could see that there is a `cutadapt` tool definition at *session/variants/tools/cutadapt.cwl*.\n",
        "\n",
        "Read & run the CLI commands below to see how ***tools*** and ***workflows*** can be executed using the `cwltool` engine.\n",
        "\n",
        ">***Note***<br>\n",
        ">If curious, you can open these files using the file explorer. <br>\n",
        ">If the workflow doesn't make sense, thats ok! It's all explained below."
      ],
      "metadata": {
        "id": "jiFXg38uadeW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TOOL\n",
        "#cwltool <tool.cwl> [inputs]\n",
        "!cwltool session/examples/basics/echo.cwl --message \"hello there!\""
      ],
      "metadata": {
        "id": "Uyo-HwfmXJC1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# WORKFLOW\n",
        "#cwltool <workflow.cwl> [inputs]\n",
        "!cwltool session/examples/basics/wf_outputs.cwl --greeting \"good afternoon!\""
      ],
      "metadata": {
        "id": "C1RxXomt7INz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "1drknAa4Rh9u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7.2 - <font color='#097969'>Our First CWL Workflow<font>\n",
        "\n",
        "Workflows are highly similar to tools. <br>\n",
        "Workflows have the following main components:\n",
        "- <font color='red'>metadata</font> - information about the workflow & extra details\n",
        "- <font color='red'>inputs</font> - data supplied from outside world\n",
        "- <font color='red'>steps</font> - tools to execute\n",
        "- <font color='red'>outputs</font> - data we collect after workflow execution is finished\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "5BJ1w9xSRh9u"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "class: Workflow\n",
        "cwlVersion: v1.2\n",
        "\n",
        "inputs:\n",
        "  in1: string\n",
        "\n",
        "steps:\n",
        "  echo:\n",
        "    run: ./echo.cwl\n",
        "    in:\n",
        "      message: in1\n",
        "    out: [phrase]\n",
        "    \n",
        "outputs:\n",
        "  echo_out:\n",
        "    type: File\n",
        "    outputSource: echo/phrase\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "Myk74buIOJ5H"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Inputs** are the data supplied from the outside world. Each time the workflow is run, we need to supply values for these inputs. Inputs can be used by steps in the workflow.\n",
        "\n",
        "**Steps** consist of the tools we wish to use. We need to specify the <font color='green'>*tool*</font> we wish to execute, as well as how to supply data to it's <font color='green'>*tool inputs*</font>. We also need to specify which <font color='green'>*tool outputs*</font> we are interested in.\n",
        "\n",
        "**Outputs** are the data produced by the workflow which we wish to return to the outside world. <br>\n",
        "After the workflow has finished running, these outputs are presented back to the user."
      ],
      "metadata": {
        "id": "XRU6VIR8OKho"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basic/wf_final.cwl* - double click in the file <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basic/echo.cwl*  - double click in the file <br>\n",
        "> (You have them bellow too!!). <br>\n",
        "\n",
        "> Looking at `wf_final.cwl`, you should see a single input called `greeting`, and a single output called `phrase`. <br>\n",
        ">It has a step called `echo` which runs `./echo.cwl`.<br>\n",
        ">Switch over to `echo.cwl` to see this tool definition. <br>\n",
        ">\n",
        ">After a brief look, run the cell below to execute the workflow.\n",
        "\n",
        "#### session/examples/basic/wf_final.cwl\n",
        "```\n",
        "class: Workflow\n",
        "cwlVersion: v1.2\n",
        "\n",
        "inputs:\n",
        "  greeting: string\n",
        "\n",
        "steps:\n",
        "  echo:\n",
        "    run: ./echo.cwl\n",
        "    in:\n",
        "      message: greeting\n",
        "    out: [phrase]\n",
        "\n",
        "outputs:\n",
        "  echo_out:\n",
        "    type: File\n",
        "    outputSource: echo/phrase\n",
        "```\n",
        "\n",
        "#### session/examples/basic/echo.cwl\n",
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "\n",
        "baseCommand: echo\n",
        "stdout: phrase.txt\n",
        "\n",
        "inputs:\n",
        "  message:\n",
        "    type: string\n",
        "    inputBinding:\n",
        "      position: 1\n",
        "\n",
        "outputs:\n",
        "  phrase:\n",
        "    type: stdout\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "T-UMVSgONniG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#cwltool <workflow.cwl> [inputs]\n",
        "!cwltool session/examples/basics/wf_final.cwl --greeting \"good morning!\""
      ],
      "metadata": {
        "id": "1HhFunWZNmqr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "CWL produces a useful output log while running workflows.\n",
        "\n",
        "We can use this for ***debugging***.<br>\n",
        "The start and finish of each workflow, step, and tool, as well as any errors that occur are logged. <br>\n",
        "If errors arise during execution, this information can be used to track down where the error occurred in the pipeline.\n",
        "\n"
      ],
      "metadata": {
        "id": "rgLjIw_rPBI9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "INFO [workflow ] start                   # workflow started\n",
        "INFO [step echo] start                   # step started (supplying data)\n",
        "INFO [job echo] <tmpfile> echo           # tool started\n",
        "INFO [job echo] completed success        # tool finished\n",
        "INFO [step echo] completed success       # workflow finished\n",
        "{ ... }                                  # workflow outputs\n",
        "INFO Final process status is success     # final status\n",
        "```"
      ],
      "metadata": {
        "id": "nP-qyLrDQ8ZV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Congratulations! ✨✨✨\n",
        "\n",
        "Now that we have run our first CWL workflow, we can learn how to write them ourselves. <br>\n",
        "We will learn how inputs, steps, and outputs are written for a CWL workflow. <br>\n",
        "As we learn, we will build a variant calling workflow using `cutadapt` and other tools we write along the way."
      ],
      "metadata": {
        "id": "BeevQO7lROEO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ">***Note***<br>\n",
        ">If you get stuck during the next three chapters (inputs / steps / outputs), move on to the next chapter. <br>\n",
        ">The starting point of the next chapter is the answer for the current chapter!"
      ],
      "metadata": {
        "id": "EjcMbLyzczAI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "gwzGWk5rZsWK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7.3 - <font color='#097969'>Inputs<font>"
      ],
      "metadata": {
        "id": "5m-0_GsLcymP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "Each **Workflow Input** must have a unique name and a type.\n",
        "\n",
        "They can be defined on a single line, if only `name: type` is needed.\n",
        "\n",
        "```\n",
        "inputs:\n",
        "  my_string: string\n",
        "  my_int: int\n",
        "  my_file: File\n",
        "```\n",
        "\n",
        "Inputs also allow other parameters, including `label`, `doc`, `default`, and <font color=\"red\">`secondaryFiles`</font> which will be touched on later.\n",
        "\n",
        "```\n",
        "inputs:\n",
        "  my_string:\n",
        "    type: string\n",
        "    label: \"input message\"\n",
        "    doc: \"A mandatory string message. Used by step.... etc etc\"\n",
        "    default: \"HELLO!!\"\n",
        "```\n",
        "\n",
        ">***Note***<br>\n",
        ">Workflow inputs can be optional. This is performed in the same manner as for tool inputs.\n",
        ">```\n",
        "inputs:\n",
        "  my_string: string?     # optional\n",
        "  my_int: int?           # optional\n",
        "  my_file: File\n",
        "```"
      ],
      "metadata": {
        "id": "h-1y-ibPZsWK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basic/wf_inputs.cwl*\n",
        ">\n",
        ">Add an input called `greeting`, with the type `string`. <br>\n",
        ">A value for this input is supplied in the CLI command.\n",
        ">\n",
        ">Run the cell below to check your answer. You can do that with a double click in the file, and modify it directly, in the pop-up window. To save it, just click again in the pop-up window - the '*' before the name of the file (e.g. *wf_inputs.cwl) will dissapear after saving the changes.\n",
        "\n"
      ],
      "metadata": {
        "id": "iyWCx1MkDCo1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cwltool session/examples/basics/wf_inputs.cwl --greeting \"hello!\""
      ],
      "metadata": {
        "id": "rBpyuTFGDCo2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If correct, `cwltool` will run without errors. You should see something like the following:\n",
        "```\n",
        "INFO [workflow ] start\n",
        "INFO [workflow ] completed success\n",
        "{}INFO Final process status is success\n",
        "```\n",
        "\n",
        "We started the workflow, executed zero steps and collected zero outputs."
      ],
      "metadata": {
        "id": "1Owyxl1Bb27n"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "no_cR5cPdPXB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7.4 - <font color='#097969'>Steps<font>\n",
        "\n",
        "Each ***Workflow Step*** has a unique name, and three mandatory subobjects.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "oW2oZuqZd9NG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "steps:\n",
        "  wc:\n",
        "    run: ./wc.cwl  \n",
        "    in:\n",
        "      infile: my_file      \n",
        "    out: [count]    \n",
        "```"
      ],
      "metadata": {
        "id": "z5c40r2U8Rqo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color=\"red\">`run`</font> provides a path to a CWL tool definition we wish to use. <br>\n",
        "The path can be relative to the workflow file path, or absolute.\n",
        "\n",
        "<font color=\"red\">`in`</font> allows us to feed data to the tool's inputs.<br>\n",
        "Make sure each tool input is accounted for - if any are missing, CWL will throw an error! <br>\n",
        "Both <font color=\"green\">workflow inputs</font>, and the outputs of other <font color=\"green\">workflow steps</font> can be used to feed data to tool inputs. <br>\n",
        "\n",
        "<font color=\"red\">`out`</font> specifies the tool outputs we wish to keep for this step.  <br>These can be used in other steps, or as workflow outputs.\n",
        "\n"
      ],
      "metadata": {
        "id": "ZndioDXF8glu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Feeding data from workflow inputs***\n",
        "\n",
        "workflow.cwl\n",
        "```\n",
        "class: Workflow\n",
        "cwlVersion: v1.2\n",
        "\n",
        "inputs:\n",
        "  my_file: File\n",
        "\n",
        "steps:\n",
        "  wc:\n",
        "    run: ./wc.cwl          # the tool to use is 'wc.cwl' (a CWL tool stored in the local directory)\n",
        "    in:\n",
        "      infile: my_file      # the 'infile' tool input of wc.cwl will be fed data from the 'my_file' workflow input\n",
        "    out: [count]           # the 'count' tool output of wc.cwl will be used later in the workflow\n",
        "\n",
        "outputs: []\n",
        "```\n",
        "\n",
        "./wc.cwl\n",
        "```\n",
        "cwlVersion: v1.2\n",
        "class: CommandLineTool\n",
        "baseCommand: wc\n",
        "\n",
        "inputs:\n",
        "  infile:\n",
        "    type: File\n",
        "    inputBinding:\n",
        "      position: 1\n",
        "\n",
        "outputs:\n",
        "  count:\n",
        "    type: stdout  \n",
        "```\n"
      ],
      "metadata": {
        "id": "I9NZOkfgfIK2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the example below, `workflow.cwl` has a single step called `wc`. <br>\n",
        "In the above, we see that\n",
        "- `./wc.cwl` is the tool to use\n",
        "- The <font color=\"green\">`my_file`</font> workflow input should feed the <font color=\"orange\">`infile`</font> tool input\n",
        "- We wish to keep the <font color=\"orange\">`count`</font> tool output for use somewhere else in the workflow.\n",
        "\n"
      ],
      "metadata": {
        "id": "rGC4aNnGgDB-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ">***Note***<br>\n",
        ">If a tool input is `optional`, we can choose whether to supply data to that input.<br>\n",
        ">We can omit it from the step inputs.\n"
      ],
      "metadata": {
        "id": "70XYZAKizpnJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basic/wf_steps_task1.cwl*<br>\n",
        ">\n",
        ">Execute the workflow by running the cell below. <br>\n",
        ">Note that for `[job echo]` `\"hello!\"` is being echoed.\n",
        ">\n",
        ">This workflow has two inputs - <font color=\"green\">`greeting`</font> and <font color=\"green\">`farewell`</font>.<br>\n",
        ">Swap the workflow input feeding the <font color=\"orange\">`message`</font> tool input from <font color=\"green\">`greeting`</font> to <font color=\"green\">`farewell`</font>.<br>\n",
        ">It should now echo `\"goodbye!\"`\n"
      ],
      "metadata": {
        "id": "xteX45Bnln6W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cwltool session/examples/basics/wf_steps_task1.cwl --greeting \"hello!\" --farewell \"goodbye!\""
      ],
      "metadata": {
        "id": "CEkUSrjjln6W"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Feeding data from step outputs***\n",
        "\n",
        "Tool inputs can also be fed data from the outputs of other workflow steps.<br>\n",
        "This is achieved using `step/outname` - where `step` is the name of the step, and `outname` is the name of the specific output.\n"
      ],
      "metadata": {
        "id": "fUqCGBLW93te"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "class: Workflow\n",
        "cwlVersion: v1.2\n",
        "\n",
        "inputs:\n",
        "  raw_reads: File\n",
        "  reference: File\n",
        "\n",
        "steps:\n",
        "  cutadapt:\n",
        "    run: tools/cutadapt.cwl\n",
        "    in:\n",
        "      reads: raw_reads\n",
        "    out: [trimmed_reads]\n",
        "    \n",
        "  bwa_mem:\n",
        "    run: tools/bwa_mem.cwl\n",
        "    in:\n",
        "      reads: cutadapt/trimmed_reads     # fed from a step output\n",
        "      ref: reference\n",
        "    out: [bam]\n",
        "\n",
        "outputs: []\n",
        "```"
      ],
      "metadata": {
        "id": "aIUnf4K075GK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basic/wf_steps_task2.cwl*\n",
        ">\n",
        ">Read the `wc` step. <br>\n",
        ">Is data being fed to the `infile` tool input using a <font color=\"green\">workflow input</font>, or the output of another <font color=\"green\">workflow step</font>?<br>\n",
        ">Run the cell below to see the workflow in action. <br>"
      ],
      "metadata": {
        "id": "QPbUz6wXzysm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cwltool session/examples/basics/wf_steps_task2.cwl --greeting \"hello!\" --farewell \"goodbye!\""
      ],
      "metadata": {
        "id": "-aBjj9Ov498l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basic/wf_steps_task3.cwl*\n",
        ">\n",
        ">Add a step called `echo`.\n",
        ">\n",
        ">This step should use the `./echo.cwl` tool. <br>\n",
        ">It should feed the `greeting` workflow input to the `message` tool input. <br>\n",
        ">It should collect the `phrase` tool output.\n",
        ">\n",
        ">Run the cell below to check your answer.<br>\n",
        ">The `echo` step should appear in the CWL log starting with `[job echo]`.\n",
        "\n"
      ],
      "metadata": {
        "id": "jcpO1OJUCIEx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cwltool session/examples/basics/wf_steps_task3.cwl --greeting \"hello!\""
      ],
      "metadata": {
        "id": "JWCaet-3CIqI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "E0QcnCGwReye"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7.5 - <font color='#097969'>Outputs<font>\n",
        "\n",
        "***Workflow Outputs*** collect the data we want to return after execution.<br>\n",
        "They have a unique name, a `type`, and an <font color=\"red\">`outputSource`</font>. <br>\n",
        "Their `type` is almost always `File`."
      ],
      "metadata": {
        "id": "qQf-WSF5eDT6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "class: Workflow\n",
        "cwlVersion: v1.2\n",
        "\n",
        "inputs:\n",
        "  my_string: string\n",
        "\n",
        "steps:\n",
        "  echo:\n",
        "    run: ./echo.cwl\n",
        "    in:\n",
        "      the_string: my_string\n",
        "    out: [phrase]\n",
        "\n",
        "outputs:\n",
        "  echo_out:\n",
        "    type: File\n",
        "    outputSource: echo/phrase\n",
        "```"
      ],
      "metadata": {
        "id": "lYwXwAXi_Qjs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the workflow above, we have a single workflow output called `echo_out`. <br>\n",
        "You may notice that the `outputSource` is familiar - its referencing a step output! <br>\n",
        "In practice, `outputSource` is almost always a step output.\n",
        "\n"
      ],
      "metadata": {
        "id": "_kky0MNWAJup"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basic/wf_outputs.cwl*\n",
        ">\n",
        ">Add an output to this workflow called `echo_out`. <br>\n",
        ">It's `type` should be `File`, and it should reference the `phrase` output of the `echo` step.\n",
        ">\n",
        ">Run the cell below to check your answer. <br>\n",
        ">The CWL output log should now report an output called `echo_out` being produced.\n"
      ],
      "metadata": {
        "id": "hBVU9lH6A6xJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cwltool session/examples/basics/wf_outputs.cwl --greeting \"hello!\""
      ],
      "metadata": {
        "id": "F_p83TNgAu7p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "6ZxYTJqSeCVA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 7.6 - <font color='#097969'>Job Params<font>\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "AhqkLaceReyf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We must supply inputs when running a tool or workflow with an execution engine.<br>\n",
        "Until now, we have been supplying inputs manually using the CLI :\n",
        "\n",
        "<font color='green'>`!cwltool echo.cwl --message \"hello there!\"`</font>\n",
        "\n",
        "This has been fine since we only have a single input. <br>\n",
        "Once a workflow has 5... 10... 20...+ inputs, this method can get inconvenient!<br>\n",
        "Luckily, we can specify our inputs using a YAML file instead.\n",
        "\n",
        "<font color='green'>`!cwltool echo.cwl params.yml`</font>\n",
        "\n",
        "This file is commonly named `params.yml`, `job.yml` or `inputs.yml` - whichever makes most sense to you. <br>\n",
        "Today, we have used `params.yml`.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "gNfHh3jz0tyt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚡ <font color='Orange'>**TASK**</font> <br>\n",
        ">📂 <font color='red'>**Open**</font> *session/examples/basics/params.yml*\n",
        "and inspect the contents.<br>\n",
        ">See that the `greeting` workflow input is being provided. <br>\n",
        ">\n",
        ">We also provide two extra values - `my_int` and `my_file`. <br>\n",
        ">They're just included to provide a reference for `File` and `int` type input values. <br>\n",
        ">Since their names don't correspond to any workflow input, they are dropped. <br>\n",
        ">\n",
        ">Run the cell below to demonstrate. It should run without errors.\n"
      ],
      "metadata": {
        "id": "vFPdeYnvwMvl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#cwltool <file.cwl> <params.yml>\n",
        "!cwltool session/examples/basics/wf_final.cwl session/examples/basics/params.yml"
      ],
      "metadata": {
        "id": "bo2eq9nig3Oz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>"
      ],
      "metadata": {
        "id": "3FrcWNZzXJg6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### <font color=\"white\"></font>"
      ],
      "metadata": {
        "id": "o3JyiAIL2tvP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 8 - <font color='#097969'>Continue Learning CWL<font>\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "DXjieitcuMyX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Best Practices***\n",
        "\n",
        "CWL tools and workflows can be written in many different ways.\n",
        "\n",
        "CWL provides a set of features and rules - the way these building blocks are pieced together depends on the developer!\n",
        "\n",
        "To avoid common pitfalls, the [CWL best-practices guide](https://www.commonwl.org/user_guide/topics/best-practices.html#) can be consulted. <br>\n",
        "This is a culmination of experience garnered from the CWL community, so is a highly valuable resource!\n",
        "More useful best-practices information can be found in the [CWL FAQ guide](https://www.commonwl.org/user_guide/faq.html).\n"
      ],
      "metadata": {
        "id": "1ilHc_LzeiPk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Advanced CWL***\n",
        "\n",
        "This material covered the fundamentals of CWL tools and Workflows, but there is still more you may need to learn.\n",
        "\n",
        "The main omission from this material is array inputs.<br>\n",
        "A chapter on arrays will be added at a future data as an extension.\n",
        "\n",
        "Here are some resources which will extend your CWL skills even futher:\n",
        "\n",
        "Tools\n",
        "- [CWL CommandLineTool specification](https://www.commonwl.org/v1.2/CommandLineTool.html)\n",
        "- [Expressions](https://www.commonwl.org/user_guide/topics/expressions.html)\n",
        "- [Creating files at runtime](https://www.commonwl.org/user_guide/topics/creating-files-at-runtime.html)\n",
        "\n",
        "Workflows\n",
        "- [CWL Workflow specification](https://www.commonwl.org/v1.2/Workflow.html)\n",
        "- [Conditional steps](https://www.commonwl.org/user_guide/topics/using-containers.html)\n",
        "- [Scatter parallelisation](https://www.commonwl.org/user_guide/topics/using-containers.html)\n",
        "- [Subworkflows](https://www.commonwl.org/user_guide/topics/using-containers.html)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "n3yfcKvoedw6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Communities***\n",
        "\n",
        "CWL has an active community of developers. <br>\n",
        "Becoming a part of this community is the best way to increase your skills and become an expert!\n",
        "\n",
        "Here are some of the many ways to become part of the CWL community:\n",
        "- [CWL Gitter](https://gitter.im/common-workflow-language/common-workflow-language)\n",
        "- [CWL Discourse Group](https://cwl.discourse.group/)\n",
        "- [Community Meetings](https://www.commonwl.org/community/#cwl-online-community-meetings)"
      ],
      "metadata": {
        "id": "YiEcIzOBegdh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 9 - <font color='#097969'>Exercises<font>"
      ],
      "metadata": {
        "id": "EyyROF_uW8YR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 9.1 - <font color='#097969'> Building Your CWL Tools <font>\n",
        "\n",
        "Create three CWL tools:\n",
        "\n",
        "*GREP Tool*: Follow the instructions/code from [here](https://github.com/screx/cwl-tutorial?tab=readme-ov-file#grep).\n",
        "*  Define inputs for the search term and the file to search in.\n",
        "* Specify the base command and arguments to reflect GREP's syntax\n",
        "\n",
        "* **Easier**: You can also use directy the **grep.cwl** and the **input files** (hello.txt and search.yml)from  `exercise/cl-tools/grep/`.\n",
        "\n",
        "\n",
        "\n",
        "WC Tool: Follow the instructions/code detailed [here](https://github.com/screx/cwl-tutorial?tab=readme-ov-file#wc)\n",
        "\n",
        "* Set up parameters to count occurrences.\n",
        "* Configure the output to capture the count in a designated file\n",
        "\n",
        "* **Easier**: You can also use directy the **wc.cwl** and the **input files**(count.txt, count.yml) from `exercise/cl-tools/wc/`.\n",
        "\n",
        "TAR Tool:  Follow the instructions/code detailed [here](https://github.com/screx/cwl-tutorial?tab=readme-ov-file#tar)\n",
        "\n",
        "* Detail the input as the compressed file\n",
        "* Configure the output as the uncompressed content.\n",
        "\n",
        "- **Easier**: You can also use directy the **tar.cwl** and the **input files** (christmas_carol.txt, ex.yml) from `exercise/cl-tools/tar/`.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "VBtegFmAXOG6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### Code GREP CWL tool"
      ],
      "metadata": {
        "id": "NsLdzPUKHANI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "### Run GREP CWL Tool\n",
        "#cwltool <file.cwl> <params.yml>\n",
        "\n",
        "# Option 1 - Using your own code (the one above), but the input files from exercise/cl-tools/grep/.\n",
        "\n",
        "#search_term = 'id:greptool'\n",
        "#write_cell_above_to_file(search_term, '/tmp/grep.cwl')\n",
        "#!cwltool /tmp/grep.cwl /content/exercise/cl-tools/grep/search.yml\n",
        "\n",
        "#####\n",
        "\n",
        "#Option 2 - Using just the code provided in exercise/cl-tools/grep/grep.cwl ;\n",
        "#and the input files from the input files from exercise/cl-tools/grep/.\n",
        "!cwltool /content/exercise/cl-tools/grep/grep.cwl /content/exercise/cl-tools/grep/search.yml"
      ],
      "metadata": {
        "id": "yMIPrgfaG8XC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "### Code for WC CWL Tool\n"
      ],
      "metadata": {
        "id": "ezKBShLTaJUG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "### Run WC CWL Tool\n",
        "#cwltool <file.cwl> <params.yml>\n",
        "\n",
        "# Option 1 - Using your own code (the one above), but the input files from exercise/cl-tools/wc/.\n",
        "\n",
        "#search_term = 'id:wctool'\n",
        "#write_cell_above_to_file(search_term, '/tmp/wc.cwl')\n",
        "#!cwltool /tmp/wc.cwl /content/exercise/cl-tools/wc/count.yml\n",
        "\n",
        "########\n",
        "\n",
        "#Option 2 - Using just the code provided in exercise/cl-tools/wc/wc.cwl ;\n",
        "#and the input files from the input files from exercise/cl-tools/wc/.\n",
        "!cwltool /content/exercise/cl-tools/wc/wc.cwl /content/exercise/cl-tools/wc/count.yml"
      ],
      "metadata": {
        "id": "SiLUmAVyaQbT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "### Code for TAR CWL Tool\n"
      ],
      "metadata": {
        "id": "B_uzutCZadcX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "### Run TAR CWL Tool\n",
        "#cwltool <file.cwl> <params.yml>\n",
        "\n",
        "# Option 1 - Using your own code (the one above), but the input files from exercise/cl-tools/tar/.\n",
        "\n",
        "#search_term = 'id:tartool'\n",
        "#write_cell_above_to_file(search_term, '/tmp/tar.cwl')\n",
        "#!cwltool /tmp/tar.cwl /content/exercise/cl-tools/tar/ex.yml\n",
        "\n",
        "########\n",
        "\n",
        "#Option 2 - Using just the code provided in exercise/cl-tools/tar/tar.cwl ;\n",
        "#and the input files from the input files from exercise/cl-tools/tar/.\n",
        "!cwltool /content/exercise/cl-tools/tar/tar.cwl /content/exercise/cl-tools/tar/ex.yml"
      ],
      "metadata": {
        "id": "cjb1d8asajTX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 9.2 - <font color='#097969'> Creating Your CWL Workflow <font>\n",
        "\n",
        "Design a workflow that integrates the GREP, WC, and TAR tools.\n",
        "\n",
        "The workflow should sequentially uncompressed a file; Search for a string (in this case **Scrooge**) and count the occurrences, with the result saved in count.txt.\n",
        "\n",
        "Follow the instructions/code detailed [here](https://github.com/screx/cwl-tutorial?tab=readme-ov-file#workflows).\n",
        "\n",
        "**Workflow Steps**:\n",
        "* Step 1: Start with the TAR tool to uncompress  the input file.\n",
        "\n",
        "* Step 2: Use the GREP tool to search for the desired string in the uncompressed data.\n",
        "* Step 3: Apply the WC tool on the output of GREP to count the occurrences and output to count.txt.\n",
        "\n",
        "**Workflow Execution**:\n",
        "* Ensure that each tool's output is correctly piped as the input to the subsequent tool.\n",
        "* Set up the final output file count.txt to store the count from the WC tool.\n",
        "\n",
        "\n",
        "**Easier**: You can also use directy the **basic.cwl** from `exercise/cl-tools/workflow/` and the **input file** (basic.yml) from `exercise/cl-tools/`."
      ],
      "metadata": {
        "id": "2affX4gRYwCE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### Code for CWL workflow"
      ],
      "metadata": {
        "id": "JquOT3h2amBE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "### Run CWL workflow\n",
        "#cwltool <file.cwl> <params.yml>\n",
        "\n",
        "### Run TAR CWL Tool\n",
        "#cwltool <file.cwl> <params.yml>\n",
        "\n",
        "# Option 1 - Using your own code (the one above), but the input files from exercise/cl-tools/basic.yml.\n",
        "\n",
        "#search_term = 'id:workflow'\n",
        "#write_cell_above_to_file(search_term, '/tmp/workflow.cwl')\n",
        "#!cwltool /tmp/workflow.cwl /content/exercise/cl-tools/basic.yml\n",
        "\n",
        "########\n",
        "\n",
        "#Option 2 - Using just the code provided in exercise/cl-tools/tar/tar.cwl ;\n",
        "#and the input files from the input files from exercise/cl-tools/basic.yml\n",
        "!cwltool /content/exercise/cl-tools/workflow//basic.cwl /content/exercise/cl-tools/basic.yml\n"
      ],
      "metadata": {
        "id": "CpRJLeR6aqgC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Refresh your files, to see if you have `count.txt'. If you open your `count.txt file`. You should get something similar to this:\n",
        "\n",
        "```\n",
        "359 <TMP PATH>/results.txt\n",
        "````"
      ],
      "metadata": {
        "id": "dbjSxbqrLQXR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cat count.txt"
      ],
      "metadata": {
        "id": "Y5ZsSwDKMRXp"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}