basics-worksheet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to the Python & Jupyter notebook Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, we will make ourselves familiar with the **jupyter notebook** and **Python** programming language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a jupyter notebook. It consists of multiple cells. We will write our Python code in these cells. You need to **run** the cell after you write your code. You can do this with the \"run cell\" button above, or with **`shift + enter`**. After the cell has been executed you will see a number between the **`[ ]`**, so **`[23]`** for example. If you see empty brackets, the cell has not been run and you need to run it. If you see this **`[*]`**, the cell is running. You can then also not run any other cells.<br>\n",
    "*Important*: if you adjust a cell, you need to run it again!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add new cels with the **`+`** sign in the top menu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Warming up: What are variables?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`variables` are very important in programming. A variable is a kind of box with a name in which you can store  numbers, names, expressions, and even other variables. The name is arbitrary, but it is useful if you know what it stands for later. <br>\n",
    "In Python you can easily create a variable, with ** `variable_name =` ** <br>\n",
    "*Important*: a variable name can only contain lower and uppercase letters, numbers, and underscores. No spaces or other funky characters!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`name = \"Adriana\"` <br> `age = 31` <br> <br>\n",
    "Copy and paste the above example here below and run the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now write the variable names below and see what happens when you run the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traditionally, you use the ** print () ** method to view the content of a variable: `print(variable_name)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Want to put a **note** in your code? <br>\n",
    "Use the hashtag `#` to put a comment in the code. Everythung after the hashtag won't be read by Python: `# This is a comment`<br>\n",
    "Try it out! What do you see after you have executed the cell?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable types: Strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'string' is another name for a piece of text. You need to put it in between single or double quotes. \" '.<br>\n",
    "Create a variable and put some text into it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if you don't put the text between quotes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print your string. Use `print()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also do **addition** and **multiplication** with strings. What does this do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable types: Integers & Floats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Integer is a whole number, such as 5, 6 or 2454. Create an integer and print it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Float is a number that is not whole, such as `5.67` or `823.12`. Create a float print it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do **addition, substraction, multiplication, division** (and more) with numbers. Try it out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable types: Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `list` is a collection of other variables: strings, ints, or even other lists. These are separated by a comma. It looks like this: <br>\n",
    "** `name_of_the_list = [\"text1\",\"text2\"]` ** <br>\n",
    "** `another_list = [10, 15, 60]` **<br>\n",
    "** `empty_list = []`**\n",
    "\n",
    "Let's make a list!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a list of numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also do **addition** and **multiplication** with lists. What does this do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Methods: String"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many different **methods** you can use with strings.<br>\n",
    "For example, every string has a certain lenght.<br>\n",
    "**`len()`** will count the characters in a string: `len(string)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, a string comes with a lot of junk attached to it. This happens often while scraping.<br>\n",
    "**`.strip()`** can fix this: `variable.strip()` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During scraping, we might want to separate a piece of text into two different columns. For example a date consisting of a day, month and a year can be separated by `/` (26/10/2016). Separate this text, we need to turn it into a `list`.<br>\n",
    "**`.split()`** does this for us: `my_string.split(\"/\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Methods: Integers & Floats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During scraping, we often encounter pagination on the website. This results into different URLs, such as `www.website.org/something?page=10`. To adjust the URL, we therefore need to change numbers into text.<br> **`str()`** method will do this for us: `str(number)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Methods: List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists also have a lenght! Do you remember how to find out?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build a data set, we need to save results we have scraped. The easiest way to do this is to build big lists. We will add many elements to a list using the ** `.append()` ** method: <br> `name_of_the_list.append(element)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you encounter two pieces of information in two places you'd like to put into one place. You can merge list of text variables with **`.join()`**: <br>\n",
    "`\"connecting_text\".join(name_of_the_list)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short note on syntax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Programming is a very exact craft. You need to adhere to its **syntax**. Let's see for example what happens if would like to create a list with a different kind of brackets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **syntax error** occurs when you use notation that is not appropriate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call help on what methods my variable has available using `dir()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call help on a specific method using `help()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Slightly advancing: Let's do some slicing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have cut out some information from the website and put it into a list, we need to find elements within these lists. This is called **slicing**. We use **`[]`** to slice a string or a list. In between, we put a number of the position of an element we want to filter out.<br>\n",
    "*Important* : in Python, we start counting at 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![hello-string](https://imgur.com/Ddz3Ipy.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the second character from a string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the last character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do the same with lists! In lists, every element has a number. Create a list with 5 elements and print the third one!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Getting serious: Loops and Conditions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparisons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you want to skip certain things or stop after a certain page. In Python, you can do comparisons as you would in math. The syntax can be a bit different though, because `=` is already used to assign variables for example. Use **`<` , `>` , `==` , `!=`** to compare numbers or text and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create two variables, assign one as a number and one as a string and compare them. What can you do and what can't you do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intermezzo: [write your own algorithm](https://ncase.me/joy/demo/turtle/?data=N4IgZgTghgtgpgDRALgEwFYAMAac14CaKGOIALgK4RkA2cAkgHYDOZEFAxmQJYD2LKUFC58ByANqgyATwAOcFOSq04Aeg68avCCFwatO5KABuUGhQUTUADgCM2TADpr2WwF1cM+Yv3aQAX39sKTlLEG4WNk4efmZVCDh5KDJdEA0KRhSjclDFRgoYACM4HT0ACygIlEkc72QQfKKSgH1oAHdU03NLWwB2fzcgkGEYsSERWOqQuqVqOlVKCEZUqEYAczpBWrDG4tK0iqqJaZ2CvdaoDtwuixQATkwgk58y3l5mBWuzW-qAagDgts8mcWu1Ot9LOhrANAoCvGFFipVGBtG0oBAACapViJZhbeHApr7DiHZbHIH1Xagy7g7rEADMj0GA1wCQ+ZAAaujuFBCnQ8cgwGYPp5cvURqI8YEPCA2XBOdzefyUFE4KKZhLJkNWnAwCU4IwOHABcBAkA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditionals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conditionals are based on comparisons. So **`if`** somethig applies, do something else, **`elif`** some other condition is true, do that, **`else`** do something else. Let's make a comparison.\n",
    "\n",
    "** if ** * something * ** == ** * something else * **: ** <br>\n",
    "&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ** print (** *some text* **) **<br>\n",
    "** else:** <br>\n",
    "&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ** print (** *some other text* **) **<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the spaces, or idents at the beginning of the row? These are very important in Python. This is callen **`nesting`**. See how the same code behaves if you don't use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loops are very useful when collecting data. We often have to do things with each element of a list. For example, open multiple URLs and extract data from them. We do this with a ** `for` ** loop. It looks like this: <br> <br>\n",
    "** for ** * each_element * ** in ** * name_of_the_list * **: ** <br>\n",
    "&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ** print (** *each_element* **) ** <br> <br>\n",
    "Let's make a new list and loop through it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we need to open multiple page numbers, such as  www.website.org/something?page=10, we somehow need to get to all the numbers we need. We could make a list like **`[1,2,3,4,5,6]`**. This is doable of course. But what if we need to count up to 100? 1,000? 100,000? Python has all sorts of neat functions to help us with this. To create a list of incremental numbers, we use the **`range(start,end)`** function.<br>\n",
    "**`range(0,7)`** will yield the list above. Make a range and print out all the numbers in the range below using a `for` loop!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have learned some of the basics, and you probably got many errors. That is fine and acutally pretty normal!\n",
    "Look at the debugging cheatsheet in your folder and see if you can apply the knowledge in the next excercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Excercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Put an animal name in a string `variable` **`animal`**\n",
    "2. Create a `list` **`some_numbers`** containing some arbitrary numbers\n",
    "3. Repeat the following for each number in the list with a `for` loop\n",
    "4. print the sentence `\"I have *number* *animal* s!\"` (eg.: \"I have 5 unicorns!\")\n",
    "<br>**OPTIONAL**<br>\n",
    "5. if the number is 1, print `\"I have just *number* *animal*!\"` (eg.: \"I have just 1 unicorn!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You often want to do concrete things in Python. Such as collecting data from websites and analyzing them. You can do everything purely in Python. But it is much easier with specialized so-called `libraries`. You can think of libraries as additional tools on a swiss knife: while you certainly can open a can of beans with a knife, it goeas a bit faster with a can opener.<br>\n",
    "\n",
    "Libraries that we are going to use is `BeautifulSoup, requests` and `pandas`, a library specially written to process HTML\n",
    "You get a library with `import`, for example: <br>\n",
    "`import requests`<br> \n",
    "`import pandas`<br> \n",
    "`from bs4 import BeautifulSoup as bs`<br><br>\n",
    "\n",
    "Copy and paste these in the cell beneath "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Libraries have their own methods. `requests`, library used for opening websites has a method **`requests.get()`**<br>\n",
    "Let's get the foreign news page of DN `https://www.dn.no/topic/Utenriks` and use the method **.text** to get the websites HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup`, that we will use for reading the HTML, uses ** `.find_all()` **. Let's find all headers of the page, the **`h3`** elements. BeautifulSoup also uses a method **`.text`** to extract only the text (this is only a coincidence)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We might want to put these titles in a list and make a table out of them that we can export into excel. We do this using **`pandas.DataFrame()`** and **`.to_csv()`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations!** You are already scraping now. Now we can get started for real."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}