Quotation.Rmd

# Quotation {#tidy-eval}

```{r setup, include = FALSE}
source("common.R")
library(rlang)
library(purrr)
```

## Introduction

Now that you understand the tree structure of R code, it's time to come back to one of the fundamental ideas that make `quote()` and `ast()` work: __quasiquotation__.  Quasiquotation is the combination of two ideas:

* Quoting, which allows you to capture the AST associated with a function argument 
  without evaluating it.
  
* Unquoting, which allows you to selectively evaulate parts of an expression that 
  would otherwise be quoted.

This combination of two makes it easy to make compose expressions that are mixture of direct and indirect specifications, and simplifies the solution to wide variety of challenging problems. Quasiquotation is not implement in base R, but is instead provided by the rlang package. 

Once you've mastered quasiquotation, we'll circle back around to base R. You'll learn the closest equivalents to rlang's quoting functions, learn about the variety of techniques that base R uses implement unquoting, and see some of the drawbacks.

We'll finish the chapter with a case study: using quasiquotation to construct calls "by hand". This is a useful technique for creating simple function operators with readable source code, and is a handy technique to work around functions (in base R and elsewhere) that don't support unquoting.

### Motivation

We'll start with a simple, concrete, example that helps motivate the need for unquoting, and hence quasiquotation. Imagine you're creating a lot of strings by joining together words:

```{r}
paste("Good", "morning", "Hadley")
paste("Good", "afternoon", "Alice")
```

You are sick and tired of writing all those quotes, and instead you just want to use bare words. To that end, you've managed to write the following function and it works! (We'll talk about the details of this implementation later; for now just look at the results.)

```{r}
cement <- function(...) {
  exprs(...) %>% 
    map(expr_name) %>% 
    paste(collapse = " ")
}

cement(Good, morning, Hadley)
cement(Good, afternoon, Alice)
```

This is useful because we no longer need to type quotes. The problem, however, comes when we want to use variables. It's easy to use variables with `paste()`, because we just don't surround them with double quotes:

```{r}
name <- "Hadley"
time <- "morning"

paste("Good", name, time)
```

Obviously this doesn't work with `cement()` because every input is automatically quoted:

```{r}
cement(Good, time, name)
```

Functions that behave this way in R are often called __non-standard evaluation__ (NSE for short) functions. NSE isn't quite the right name here since we're not actually evaluating anything. We'll talk more about evaluation and NSE in the next chapter.

We need some way to explicit __unquote__ the input, to tell `cement` to use the value stored inside the variable instead of the name of the variable. In this example, we need someway to tell `cement()` that `time` and `name` should be handled differently to `Good`. Quasiquotation give us a standard tool to do so: `!!`, prounounced bang-bang.

```{r}
cement(Good, !!time, !!name)
```

It's useful to compare `cement()` and `paste()` directly. `paste()` is a regular function, so we have to add the quote marks ourselves. `cement()` is a quoting function, so we need to tell it when to unquote.

```{r, eval = FALSE}
paste("Good", name, time)
cement(Good, !!time, !!name)
```

## Quasiquotation

The idea of quasiquotation was first developed by the analytic philsopher Willard van Orman Quine in the early 1940s [^1]. It's helpful in philosophy in order to clearly distinguish because use and mention, or equivalently an object and the words we use to refer to that object. For example, here are three examples from the "[Substitutional Quantifiers](https://johnmacfarlane.net/142/substitutional-quantifiers.pdf)" class taught by John MacFarlane [^2].

*  'net' is part of 'clarinet', but a net is not part of a clarinet.

*  Boston is a city. 'Boston' is the name of a city. ‘ 'Boston' ’ is not the 
   name of a city; it denotes the name of a city.

*  An hour is longer than a minute, but 'minute' is longer than 'hour'.

[^1]: You might be familiar with the name Quine from "quines", computer programs that when run return a copy of their own source code.
[^2]: In another interesting connection, John MacFarlane is the author of pandoc which is used as part of the RMarkdown toolchain to generate pdfs, ebooks, and websites like this book.

It wasn't until the mid-1970s that quasiquotation entered common use in a programming language: LISP. Useful history at <http://repository.readscheme.org/ftp/papers/pepm99/bawden.pdf>. Another way of thinking about quasiquotation is that it provides a code template. You define an AST with some "holes" that get filled in using the values of other variables.

Quasiquotation is useful in R because it allows us to have a systematic way of distinguishing when we want to refer to the name `x` vs. the contents of the variable called `x`.

### Implementation

Every function that quotes one or more arguments and provides quasiquotation must call one of  the following three functions provided by rlang:

* `expr()` captures its argument exactly as given. You'll use this most
  commonly for interactive exploration.

* `enexpr()` takes the name of an argument to a function, and returns
  the expression provided by that argument.
  
* `exprs()` captures all arguments. It has two uses: you can use it 
  interactively to generate a list of expressions, or inside a function to 
  capture all arguments provided to `...`.

(There are three variants called `quo()`, `enquo()`, `quos()` that capture both the expression and the environment. We'll come back to those in the next chapter.

Let's compare the use of `expr()` and `enexpr()` inside a function: 

```{r}
capture_1 <- function(x) expr(x)
capture_2 <- function(x) enexpr(x)

capture_1(x + y)
capture_2(x + y)
```

`expr()` always yields in When you need to construct an expression from known inputs use `expr()`. When you need to capture an expression provided by the user in an argument, use `enexpr()`.  

Depending on how you call it `exprs()` combines some of the behaviour of `expr()` and `enexpr()`. It behaves like `enexpr()` if you pass on `...`, and behaves like `expr()` for all other arguments:

```{r}
f <- function(x, ...) {
  exprs(x = x, ...)
}
f(x = y + 1, y = y + 1)
```

Generally, you'll use `exprs()` in one of two ways:

```{r}
# Interactively creating a list of expressions
exprs(x = x ^ 2, y = y ^ 3, z = z ^ 4)
# short hand for
list(x = expr(x ^ 2), y = expr(y ^ 3), z = expr(z ^ 4))

# To capture all ... inside a function
foo <- function(...) {
  dots <- exprs(...)
}
```

There's not much you can do with a list of expressions yet, but we'll see a few techniques later on in this chapter. Lists of expressions + rlang + purrr give you a surprising amount of power, which we'll get to in XXX.

The opposite of quoting is evaluating. This is a big topic, so it is covered in depth in the next chapter. For now, we'll focus on a single function: `rlang::eval_tidy()`. This takes an expression and evaluates in it.

```{r}
x <- expr(runif(5))
x

eval_tidy(x)
eval_tidy(x)
```

Notice that every time we evaluate this expression we get a different result. This makes these expression different to the lazy evaluation of functions which are only evaluated once, and then return the same results return again and again.

Quoting functions side-step evaluation, allowing you to capture the code. This allows you to inspect and transform the AST, or evaluate the code in a different way ("non-standard") to usual. Functions that use these tools are often called non-standard evaluation (NSE) functions. 

### Unquoting

There are two forms of unquoting, `!!` called unquote, and pronounced bang-bang, and `!!!` called unquote-splice, and pronounced bang-bang-bang. They both replace nodes in the AST. `!!` is a one-to-one replacement. It takes a single expression and inlines the AST at the location of the `!!`. 

```{r}
x <- expr(-1)
expr(f(!!x, y))
```

Graphically this looks like:

```{r, echo = FALSE, out.width = NULL}
knitr::include_graphics("diagrams/expression-bang-bang.png", dpi = 450)
```

`!!!` is a one-to-many replacement.  It takes a list of expressions and replaces them at the location of the `!!!`.

```{r}
x <- exprs(-1, -2)
expr(f(!!!x, y))
```


```{r, echo = FALSE, out.width = NULL}
knitr::include_graphics("diagrams/expression-bang-bang-bang.png", dpi = 450)
```

There's one final component to quasiquotation: `:=`. One challenge that sometimes crops up is that the LHS of `=` is always quoted. 

```{r}
name <- "x"
value <- 10

lobstr::ast(c(name = value))
lobstr::ast(c(name = !!value))
```

And you can't unquote on the LHS because in R's grammar is has to be a bare name, so attempting to unquote is a syntax error.

```{r, eval = FALSE}
lobstr::ast(c(!!name = !!value))
```

This doesn't work quite yet: https://github.com/tidyverse/rlang/issues/279

```{r}
lobstr::ast(c(!!name := !!value))
```

### Operator precedence and prefix forms

Sometimes using `!!` causese problems because `!!` isn't one thing - it's actually two `!` called back-to-back. We chose this notation in rlang because it's rarely used in R, and it's a clear signal that something weird is going on.

Unfortunately that sometimes causes problems because operator precednece:

```{r, error = TRUE}
x <- quote(X)
y <- quote(Y)
lobstr::ast(!!x + !!y)
```

You can solve this problem either by adding parentheses:

```{r}
lobstr::ast((!!x) + (!!y))
```

Or by using the regular function form of `!!`, `UQ()`:

```{r}
lobstr::ast(UQ(x) + UQ(y))
```

Note that using parentheses introduces additional nodes into the AST. These are usually not important.

THIS BEHAVIOUR MIGHT CHANGE.

### The downsides of non-standard evaluation {#nse-downsides}

The biggest downside of NSE is that functions that use it are no longer [referentially transparent](http://en.wikipedia.org/wiki/Referential_transparency_(computer_science)). A function is __referentially transparent__ if you can replace its arguments with their values and its behaviour doesn't change. For example, if a function, `f()`, is referentially transparent and both `x` and `y` are 10, then `f(x)`, `f(y)`, and `f(10)` will all return the same result. Referentially transparent code is easier to reason about because the names of objects don't matter, and because you can always work from the innermost parentheses outwards. \index{non-standard evaluation!drawbacks}

There are many important functions that by their very nature are not referentially transparent. Take the assignment operator. You can't take `a <- 1` and replace `a` by its value and get the same behaviour. This is one reason that people usually write assignments at the top-level of functions. It's hard to reason about code like this:

```{r}
a <- 1
b <- 2
if ((b <- a + 1) > (a <- b - 1)) {
  b <- b + 2
}
```

Using NSE prevents a function from being referentially transparent. This makes the mental model needed to correctly predict the output much more complicated. So, it's only worthwhile to use NSE if there is significant gain. For example, `library()` and `require()` can be called either with or without quotes, because internally they use `deparse(substitute(x))` plus some other tricks. This means that these two lines do exactly the same thing: \index{referential transparency}

```{r, eval = FALSE}
library(ggplot2)
library("ggplot2")
```

Things start to get complicated if the variable is associated with a value. What package will this load?

```{r, eval = FALSE}
ggplot2 <- "plyr"
library(ggplot2)
```

There are a number of other R functions that work in this way, like `ls()`, `rm()`, `data()`, `demo()`, `example()`, and `vignette()`. To me, eliminating two keystrokes is not worth the loss of referential transparency, and I don't recommend you use NSE for this purpose.

One situation where non-standard evaluation is worthwhile is `data.frame()`. If not explicitly supplied, it uses the input to automatically name the output variables:

```{r}
x <- 10
y <- "a"
df <- data.frame(x, y)
names(df)
```

I think it's worthwhile because it eliminates a lot of redundancy in the common scenario when you're creating a data frame from existing variables. More importantly, if needed, it's easy to override this behaviour by supplying names for each variable.

Non-standard evaluation allows you to write functions that are extremely powerful. However, they are harder to understand and to program with. As well as always providing an escape hatch, carefully consider both the costs and benefits of NSE before using it in a new domain.

### Exercises

1.  It's challenging to see the AST of code like `!!x + !!y` because  
    `ast()` does unquoting. We can work around this by using the base
    equivalent of `expr()` that doesn't do unquoting: `quote()`. Why does
    this work? What does it tell you about unquoting?
    
    ```{r}
    expr <- quote(!!x + !!y)
    lobstr::ast(!!expr)
    ```
    

## Base R

Now that you understand the basics of quasiquotation it's time to take a look at what base R does. Base R has no systematic equivalent of quasiquotation. 

### Quoting functions

* `quote()` is like `expr()` with quasiquotation.

* `substitute()` is similar to `enexpr()`. Its primary purpose is to
  capture unevaluated arguments, but it also does a bunch of other stuff
  including code expansion, and differs in behaviour when called from
  the top-level vs. inside a function.

* There is no built-in equivalent to `exprs()` but you can write your own:
  
    ```R
    dots <- function(...) eval(substitute(alist(...)))
    ```
    
    This uses the code expansion part of `substitute()`.

* `bquote()` is like `expr()` but provides a more limited form of 
  quasiquotation. Can only unquote, not unquote-splice, using `.`.
  Not used to provide quasiquotation for any other function in R.



### Unquoting

Since base R doesn't have quasiquotation, different functions that quote their arguments offer different approaches. Three apptend to offer one of three approaches to allow you to unquote: 

*   A pair of functions: one that quotes, and one that doesn't. For example, 
    take the very commonly used `$`. It quotes it's second argument: `df$x`. 
    There is no way to unquote with `$`, instead you use the non-quoting `[[`.
      
    ```{r}
    x <- list(var = 1, y = 2)
    var <- "y"
    
    x$var
    x[[var]]
    ```
  
*   A pair of arguments: one that quotes and one that doesn't. For example, 
    `rm()` quotes arguments in `...` but does not quote `list`.
    
    ```{r}
    x <- 1
    y <- 2
    
    rm(x)
    
    vars <- c("y", "vars")
    rm(list = vars)
    ```

*   An argument that turns quoting on or off. For example, `library()` uses 
    the `character.only` argument to an argument to switch between quoted
    and value behaviour.
    
    ```{r}
    library(MASS)
    
    pkg <- "MASS"
    library(pkg, character.only = TRUE)
    ```

Some functions will try to evaluate, and if that fails, will quote. For example, help will use the variable if available, but if not will quote. I think this behaviour is confusing and best avoided.

```{r}
help(var)

var <- "mean"
help(var)
```

Finally, many quoting functions provide no way to unquote. That means you need to construct a call and evaluate it. For example, there's no way to unquote in `~`. This makes generating model formulas challenging.



### Exercises

1.  What does the following function do? What's the escape hatch?
    Do you think that this is an appropriate use of NSE?

    ```{r}
    nl <- function(...) {
      dots <- named_dots(...)
      lapply(dots, eval, parent.frame())
    }
    ```

2.  Instead of relying on promises, you can use formulas created with `~`
    to explicitly capture an expression and its environment. What are the
    advantages and disadvantages of making quoting explicit? How does it
    impact referential transparency?

3.  Read the standard non-standard evaluation rules found at
    <http://developer.r-project.org/nonstandard-eval.pdf>.

1.  Why does `as.Date.default()` use `substitute()` and `deparse()`?
    Why does `pairwise.t.test()` use them? Read the source code.

1.  `pairwise.t.test()` assumes that `deparse()` always returns a length one 
    character vector. Can you construct an input that violates this expectation? 
    What happens?

1.  `f()`, defined above, just calls `substitute()`. Why can't we use it
    to define `g()`? In other words, what will the following code return?
    First make a prediction. Then run the code and think about the results.

    ```{r, eval = FALSE}
    f <- function(x) substitute(x)
    g <- function(x) deparse(f(x))
    g(1:10)
    g(x)
    g(x + y ^ 2 / z + exp(a * sin(b)))
    ```

1.  Base functions `match.fun()`, `page()`, and `ls()` all try to
    automatically determine whether you want standard or non-standard
    evaluation. Each uses a different approach. Figure out the essence
    of each approach then compare and contrast.

## Case study: constructing calls


In base R, you can construct a call using the `call()` function. We are going to use the similar function `rlang::lang()`. The chief difference is that `lang()` supports quasiquotation. This makes it considerably easier to generate certain types of call.

The basics of `lang()` are simple. You create a call giving the name of a function, followed by the arguments:

```{r}
lang("+", 1, 2)
lang("foo", x = 1, y = 2)
```

Here we've used a convenient shortcut: we've given it the name of the fuction as a string not a call. In most cases a string is easier to type and directly equivalent to the `quote()`d equivalent:

```{r}
lang(expr(f), 1, 2)
lang("f", 1, 2)
```

However, this will not work if the function is generated by a function call. Note the subtle difference in these two calls:

```{r}
lang(quote(f()), 1, 2)
lang("f()", 1, 2)
```

The first uses the function generated by calling `f()`, the second calls a function with the confusing name `f()`:

```{r}
`f()` <- function(x) x + 1
`f()`(1)
```
To construct more complex calls, two new quasiquotation calls come in handy: 

*   `!!!`, pronounced bang-bang-bang, the unquote-splice operator. It allows you
    to splice in a list. Simply including the list in the call doesn't yield 
    quite what you want:

    ```{r}
    args <- list(x = 1, y = 2)
    lang("f", args, z = 3)
    ```
    
    Here we the unquote-splice operator:
    
    ```{r}
    lang("f", !!!args, z = 3)
    ```

*   `:=`, pronounced colon-equals, the definition operator. It works like `=` but
    allows you to splice on the left-hand side.

    ```{r}
    var <- "x"
    val <- 10
    
    lang("foo", var = val)
    lang("foo", !!var := val)
    ```

### Working around the absense of unquoting

`~` doesn't provide any way to unquote.

```{r}
make_model <- function(resp, preds) {
  pred_sum <- purrr::reduce(preds, function(x, y) expr(UQ(x) + UQ(y)))
  eval_tidy(expr(!!resp ~ !!pred_sum))
}
make_model(expr(y), exprs(a, b, c))
```

Note the use of `reduce()` to take a list of expressions and progressively add them together. This is a pleasant side effect of 

```{r}
binary_expr_reducer <- function(op) {
  op <- enexpr(op)
  
  function(x, y) {
    expr(UQ(op)(UQ(x), UQ(y)))
  }
}

x <- exprs(a, b, c, d)
purrr::reduce(x, binary_expr_reducer(`*`))
purrr::reduce_right(x, binary_expr_reducer(`*`))

purrr::reduce(x, binary_expr_reducer(f))
purrr::reduce_right(x, binary_expr_reducer(f))
```

How to use `expr()` + `eval_tidy()` to support wrap base functions.

```{r, error = TRUE, fig.keep = "none"}
library(lattice)
xyplot(mpg ~ disp, data = mtcars)

x <- quote(mpg)
y <- quote(disp)
xyplot(x ~ y, data = mtcars)
```

### Inlining and the deparser

If you construct ASTs by hand, it's possible to construct things that you could not construct by parsing code.  For example, if you forget to quote the first argument to `lang` it will literally inline the funtion call:

```{r}
lang(sum, quote(x))
```

It's also possible to inline objects that are not constants, symbols, or calls. This is useful in a handful of places (beyond the scope of the book, but typically useful in overscoping). The main thing to be aware of is that the the printed representation does not always accurately reflect the underlying tree. Trust `ast()` over what the console will print.

R will print parentheses that don't exist in the call tree:

```{r}
x1 <- lang("+", 1, lang("+", 2, 3))
x1
lobstr::ast(!!x1)
```

It will also display integer sequences as if they were generated with `:`.

```{r}
x2 <- lang("f", c(1L, 2L, 3L, 4L, 5L))
x2
lobstr::ast(!!x2)
```

If you inline more complex objects, their attributes are not printed which might lead to confusing output:

```{r}
x3 <- lang("class", data.frame(x = 10))
x3
eval(x3)

lobstr::ast(!!x3)
```

In general, if you're ever confused, remember to check the object with `ast()`!

### Exercises

1.  The following two calls look the same, but are actually different:

    ```{r}
    (a <- call("mean", 1:10))
    (b <- call("mean", quote(1:10)))
    identical(a, b)
    ```

    What's the difference? Which one should you prefer?

1.  `standardise_call()` doesn't work so well for the following calls.
    Why?

    ```{r}
    lang_standardise(quote(mean(1:10, na.rm = TRUE)))
    lang_standardise(quote(mean(n = T, 1:10)))
    lang_standardise(quote(mean(x = 1:10, , TRUE)))
    ```

1.  Read the documentation for `rlang::lang_modify()`. How do you think
    it works? Read the source code.

1.  Use `subs()` to convert the LHS to the RHS for each of the following pairs:
    * `a + b + c` -> `a * b * c`
    * `f(g(a, b), c)` -> `(a + b) * c`
    * `f(a < b, c, d)` -> `if (a < b) c else d`

2.  For each of the following pairs of expressions, describe why you can't
    use `subs()` to convert one to the other.
    * `a + b + c` -> `a + b * c`
    * `f(a, b)` -> `f(a, b, c)`
    * `f(a, b, c)` -> `f(a, b)`

1.  Concatenating a call and an expression with `c()` creates a list. Implement
    `concat()` so that the following code works to combine a call and
    an additional argument.

    ```{r, eval = FALSE}
    concat(quote(f), a = 1, b = quote(mean(a)))
    #> f(a = 1, b = mean(a))
    ```

1.  Since `list()`s don't belong in expressions, we could create a more
    convenient call constructor that automatically combines lists into the
    arguments. Implement `make_call()` so that the following code works.

    ```{r, eval = FALSE}
    make_call(quote(mean), list(quote(x), na.rm = TRUE))
    #> mean(x, na.rm = TRUE)
    make_call(quote(mean), quote(x), na.rm = TRUE)
    #> mean(x, na.rm = TRUE)
    ```

1.  How does `mode<-` work? How does it use `call()`?