analysis_report.Rmd

---
title: "Confidence and Bias -- Analysis"
author: "Timothy J. Luke"
date: "`r Sys.Date()`"
output: github_document
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(echo = TRUE)

source("./R/wrangle.R")
source("./R/analyze.R")

```

# Does sender confidence influence bias?

## Raw judgments

To assess whether sender confidence (a manipulated variable) increases truth bias, we conducted a mixed effects logistic regression analysis. We began with a model in which veracity judgments (0 = lie, 1 = truth) were predicted by actual sender veracity (lie vs. truth) and by message content (holiday, bereavement, accident, and quarrel; dummy coded with holiday as the reference group), as well as random intercepts for each receiver and each sender. We then fit a model that added confidence condition and a third model that added the interaction term between content and confidence condition. Then we compared the models to select one using a likelihood ratio test. As can be seen below, the test suggested better fit with the addition of the interaction term.

```{r}
lrt_judge_fixed
```

The coefficient for sender confidence was significant and positive, indicating that more confident senders were viewed as more likely to be truthful when talking about holidays. Additionally, the interaction terms for sender confidence with accident messages and with quarrel messages were significant and negative. These results suggest that sender confidence had little or no effect on veracity judgments for messages about accidents, and higher sender confidence was associated with fewer truth judgments for messages about quarrels. In short, the confidence with which senders presented their stories appeared to influence receivers’ veracity judgments, but this relationship was moderated by the content of the message. Additionally, sender veracity significantly interacted with message content, such that truthful messages about bereavements and quarrels were more likely to be judged as truths, compared to messages about holidays. Table 2 displays the raw rates of truth judgments as well as mean predicted truth judgment rates derived from the retained model for each condition. 

```{r}
summary(judge_fixed_conf_int)
```

### Mean predicted truth judgment rates

For a more intuitive examination of the results of the model above, we can extract predicted truth judgment rates and calculate means for each condition, sender, and content type. For each mean predicted rate of truth judgments, I computed 95% parametric bootstrap confidence intervals using 500 simulations.

```{r}
boot_ci_fixed_inter_2
```

## Signal detection

As a robustness check, I assessed whether confidence condition predicted the signal detection index "c" -- a measure of bias. To perform this check, after calculating c for each receiver (collapsed across the content types), I fit a linear regression model predicting c from confidence condition. As can be seen below, the results of this analysis support the results of the models examining raw judgments.

```{r}
summary(sdt_model_base)
```

We think the analyses of raw judgments are more informative (and more intuitive) than the signal detection analysis. But it's nice to see this kind of consistency.

We can see the results in the descriptive statistics below as well.

```{r}
sdt_data %>% 
  group_by(confidence_condition) %>% 
  summarise(
    mean_c   = mean(c),
    sd_c     = sd(c),
    median_c = median(c),
  )
```

## Message content as a random effect

Another way of analyzing these data is to model message content as a random effect, rather than a fixed effect (as above). This approach would imply that the four content types represent a random sample from a population of possible narrative types. Whether this approach is more or less appropriate than the fixed effect approach taken above can be debated. However, the results are substantively the same as the fixed effect approach presented above.

We start by comparing three models: (1) an initial model with actual sender veracity as a predictor of veracity judgments, with random intercepts for receivers and for senders nested in content types, (2) a model which adds confidence condition as a predictor, and (3) a model that adds random slopes for confidence condition for each content type (which represent an interaction between content and confidence). We compared these models with a likelihood ratio test.

As above, the model implying an interaction between content and confidence appears to be the best fitting of the three.

```{r}
lrt_judge
```

The model with random slopes fits the data the best, even though none of the fixed effects are significant (see below). In the model with only random intercepts, we see that there is a tendency for sender confidence to increase truth judgments on average, but adding the interaction substantially increases the standard error for this coefficient. Thus, we see evidence for an interaction between content type and confidence condition, such that sender confidence has effects of different sizes and/or directions depending on the content of the message. 

These models are generally consistent with the fixed content effect approach above, as well as the signal detection model.

```{r}
summary(judge_model_conf)
```

```{r}
summary(judge_model_conf_int)
```


### Raw veracity judgment rates

Here are the overall veracity judgment rates. People were somewhat truth-biased.

```{r}
table(judge_long$judgment)/sum(table(judge_long$judgment))
```

People were more truth-biased when the sender was more confident, across the
content types.

```{r}
judge_long %>% 
  group_by(confidence_condition) %>% 
  summarise(
    truth_rate = sum(judgment == 1) / n()
  ) %>% 
  knitr::kable()
```

```{r}
judge_long %>% 
  group_by(content) %>% 
  summarise(
    truth_rate = sum(judgment == 1) / n()
  ) %>% 
  knitr::kable()
```

```{r}
judge_long %>% 
  group_by(content, confidence_condition) %>% 
  summarise(
    truth_rate = sum(judgment == 1) / n()
  ) %>% 
  knitr::kable()
```

```{r}
judge_long %>% 
  group_by(content, confidence_condition, veracity) %>% 
  summarise(
    truth_rate = sum(judgment == 1) / n()
  ) %>% 
  knitr::kable()
```

# Does sender confidence influence receiver accuracy?

## Raw judgments

To assess whether sender confidence influenced receiver accuracy, we conducted a mixed effects logistic regression analysis. As with the analysis of truth judgments, we began with a model in which judgment accuracy (0 = incorrect, 1 = correct) were predicted by actual sender veracity (lie vs. truth) and by message content (holiday, bereavement, accident, and quarrel; dummy coded with holiday as the reference group), as well as random intercepts for each receiver and each sender. We then fit a model that added confidence condition and a third model that added the interaction term between content and confidence condition. Then we compared the models to select one using a likelihood ratio test. As can be seen below, the test suggested the two subsequent models after the initial model offered no significant improvement, but the model adding the three-way interactions fit best.

```{r}
lrt_acc
```

In the retained model, the coefficient for sender veracity was significant and positive, indicating that for messages about holidays, receivers judged truthful messages more accurately. The coefficient for sender confidence was significant and negative, suggesting a decrease in accuracy for messages about holidays when senders communicated more confidently. The significant interaction between sender confidence and veracity indicates that this increase was especially pronounced for truthful holiday messages communicated confidently. We also see significant increases in accuracy for messages about accidents and about quarrels. Accuracy for deceptive messages about quarrels was further, though only moderately, increased when senders conveyed their messages more confidently. Finally, the significant three-way interactions indicate that the improvement in accuracy conferred by sender confidence observed for messages for holidays was mitigated for messages about accidents and quarrels. Raw accuracy rates and mean predicted accuracy rates derived from the retained model for each condition are displayed in Table 4. Given the pattern of response biases we observed above, all these changes in accuracy are likely largely the result of shifting response bias, rather than improved discrimination.

```{r}
summary(acc_model_conf_int_3)
```

### Accuracy rates

It can be informative to examine the raw accuracy rates across and between
conditions.

The overall accuracy rate was approximately `r round((table(accuracy_long$accuracy)/sum(table(accuracy_long$accuracy)))[2], 3) * 100`%.

```{r}
table(accuracy_long$accuracy)/sum(table(accuracy_long$accuracy))
```

The effect of the truth bias on accuracy is evident when splitting accuracy by
veracity condition.

```{r}
accuracy_long %>% 
  group_by(veracity) %>% 
  summarise(
    acc_rate = sum(accuracy == 1) / n()
  ) %>% 
  knitr::kable()
```

Overall accuracy between the content types varied slightly, but the model above 
suggests that these differences were not estimated precisely enough for us to 
conclude that they are reliable.

```{r}
accuracy_long %>% 
  group_by(content) %>% 
  summarise(
    acc_rate = sum(accuracy == 1) / n()
  ) %>% 
  knitr::kable()
```

Differences across content in the extent to which truth bias influenced accuracy 
are what you would expect given the differences in truth bias across content.
That is, when there is greater truth bias, there is better accuracy for truthful
messages. No surprises there.

```{r}
accuracy_long %>% 
  group_by(content, veracity) %>% 
  summarise(
    acc_rate = sum(accuracy == 1) / n()
  ) %>% 
  knitr::kable()
```

```{r}
accuracy_long %>% 
  group_by(content, confidence_condition, veracity) %>% 
  summarise(
    acc_rate = sum(accuracy == 1) / n()
  ) %>% 
  knitr::kable()
```

## Mean predicted accuracy

We can also calculate predicted accuracy rates for each condition based on the 
retained model.

```{r}
boot_ci_fixed_acc_inter_3
```


## Signal detection

Consistent with the analysis of the raw judgments, when we fit a linear model to
predict d' (a measure of discrimination) with sender confidence as a predictor, 
we see no evidence that sender confidence influences receiver accuracy.

```{r}
summary(sdt_model_acc)
```

# Does confidence predict accuracy?

To assess the extent to which confidence predicted accuracy (viz. are correct
judgments made more confidently?), we fit an additional logistic mixed effects 
model predicting accuracy with sender veracity, content type, and receiver 
confidence. We compared this model to the retained model predicting accuracy 
above. Adding recevier confidence offered no significant improvement to the 
model.

```{r}
lrt_confidence
```

For the sake of completeness, here is the main model output. As can be seen,
confidence did not appear to predict accuracy at all.

```{r}
summary(acc_conf_model_base)
```