-
Notifications
You must be signed in to change notification settings - Fork 1
/
CLIMR_analysis_notes.Rmd
100 lines (58 loc) · 6.6 KB
/
CLIMR_analysis_notes.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
---
title: "Construal Level International Multilab Replication (CLIMR) Project: Notes on the Analysis Plan"
author: "CLIMR Team"
date: "`r Sys.Date()`"
output:
html_document:
theme: darkly
toc: yes
toc_float: yes
pdf_document:
toc: yes
knit: (function(input_file, encoding) {
rmarkdown::render(input_file, encoding = encoding, output_dir = "./reports/")
})
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
# Overview of analysis plan
This document introduces background on the analysis plan for the CLIMR project.
For each replicated experiment, we will analyze the data with the following approach.
## Presence of the predicted effect
First, we will assess whether the estimated effect is significantly greater than 0. We will calculate a meta-analytic effect size estimate and corresponding 95% confidence interval for the effect of interest. If the CI includes only positive values and excludes 0, we will conclude that there is support for the existence of an effect.
We will calculate effect sizes and sampling variances for each lab's replication of the experiment. The method of effect size calculation is described below. We will then synthesize those estimates using a random effects meta-analysis.
## Heterogeneity
Second, we will assess the presence and magnitude of heterogeneity in the estimated effect sizes across contributing labs.
We will assess heterogeneity across labs using three methods: a $Q$ test, $I^2$, and $\tau$. The $Q$ test will be used to assess whether the variation in effect sizes exceeds what would be expected by random sampling error. To draw this inference, we will use a significance criterion of .05. We will use $I^2$ and $\tau$ to quantify this potential heterogeneity.
## Unstandardized effects
Third, to assist with interpretation, we will provide unstandardized effect estimates. For effects estimated with _d_, we will back-transform the standardized effect sizes and CI bounds into unstandardized mean differences. For effect estimated using log odds ratios (i.e., social distance; see below), we will instead calculate the overall difference in proportions between the near and distant conditions across all the data and construct a 95% confidence interval for that difference. Thus, we will be able to make statements about the size of the effects in the raw units of the dependent variables.
## Assessment of overall effect
Additionally, we will assess whether the overall effect across the experiments is significantly greater than zero. We will conduct a meta-analysis of the effect sizes of all the replicated experiments from the contributing labs. If the 95% confidence interval for the meta-analytic effect contains only positive values and excludes 0, we will conclude there is overall support for an effect consistent with the predictions of Construal Level Theory.
We will use a multi-level meta-analysis to assess the overall effect, with random variance components estimated for contributing labs and for the four experiments.
We will also assess the heterogeneity of the overall effect(s) using a $Q$ test, $I^2$ (overall, across labs, and across experiments), and $\sigma$ (across labs and across experiments).
# Designs
In the designs for all four experiments we are replicating, the comparison of interest is between the near and distant conditions (or trials).
## Temporal distance, Liberman & Trope (1998, Experiment 1)
Participants are presented with seven activities and asked to describe the activity with an open-ended response. The activities were either situated in the near future or the distant. Then, they completed the "levels of personal agency questionnaire" (i.e., the BIF).
### Original effect size
In the original study, the data for the BIF (total score) were analyzed using an independent samples _t_-test. No effect size was reported. Converting from the reported test statistics and assuming equal sample sizes (i.e., _n_ = 16 per group), the effect size was _d_ = 0.92, 95% CI [0.18, 1.66].
### Replication effect size
Following the same approach as the original study, we will calculate a total BIF score for each participant and calculate a standardized mean difference according to conventional methods:
$ d = \frac{M_1 - M_2}{SD_{pooled}} $
## Spatial distance, Fujita et al. (2006, Experiment 1)
### Original effect size
In the original study, the data for the BIF were analyzed using an independent samples _t_-test. An effect size _r_ was reported, _r_ = .27. Converting from the reported test statistics and assuming equal sample sizes (i.e., _n_ = 34 per group), the effect size was _d_ = 0.55, 95% CI [0.06, 1.03]. Converting from the reported effect size gives a similar estimate, _d_ = 0.56.
### Replication effect size
The effect size will be calculated according to the formula described above for the temporal distance effect.
## Social distance (paradigmatic replication)
The effect size will be calculated according to the formula described above for the temporal distance effect.
## Likelihood distance (paradigmatic replication)
The effect size will be calculated according to the formula described above for the temporal distance effect.
# Comprehension checks
We will include comprehension checks for the psychological distance manipulations. To supplement the main results, we will conduct analyses with the same approach described above but excluding data representing comprehension check failures.
For comprehension checks pertaining to specific stimuli (e.g., recognizing the time described in each of the four scenarios used in the temporal distance manipulation), we will exclude observations for which participants failed to respond correctly (but include observations from that same participant where they responded correctly). For comprehension checks pertaining to the stimuli generally (e.g., recognizing format of the cards presented in the social distance task), we will exclude data at the participant level.
In the manuscript, we will include a full report of the analyses with these data excluded if any of these conditions are met:
* If any of the 95% confidence intervals for the supplemental analyses (i.e., those excluding comprehension check failures) include or exclude zero, in a way that is inconsistent with the primary analyses (e.g., the supplemental analysis CI for temporal distance excludes zero but the primary analysis CI does not).
* If any of the 95% CIs for the supplemental analyses exclude the point estimates of the primary analyses.
If none of these conditions are met, we will store the analyses in supplemental material and briefly mention them in the main text, with a link to a document more fully reporting these results.