Statistics

Cluster-based permutation test

In order to draw statistical inferences at the group level, MVPAlab implements a non-parametric cluster-based permutation approach, as proposed by Stelzer [1] for fMRI studies. This method has been adapted to electroencephalography data and can be computed for different performance metrics: mean accuracy, area under de curve, F1 score, recall and precision.

Using a combined permutation and bootstrapping technique, the null distribution of the empirical decoding accuracy is obtained. By default, at the single-subject level, 100 randomly permuted accuracy maps are generated. Then, one of the previously calculated accuracy maps for each participant is randomly drawn. This selection is group-averaged and the procedure is repeated 10^5 times, generating 10^5 permuted group accuracy maps. Next, for each timepoint, the chance distribution of accuracy values is estimated. The above and below chance thresholds are determined (99.9th percentile of the right and left-tailed area of the distribution), which correspond to a very low probability of obtaining significant results by chance:

Then, clusters of time-points exceeding the previously calculated threshold in all the 10^5 permuted accuracy maps are collected, generating the normalized null distribution of cluster sizes. Finally, a correction for multiple comparisons (False Discovery Rate (FDR)) is applied at a cluster level to obtain the smallest cluster size to be considered significant.

Two different functions coded the beforementioned pipeline:

% Compute permutation maps: 
[permaps,cfg] = mvpalab_permaps(cfg,fv);

% Run statistical analysis:
stats = mvpalab_permtest(cfg,result,permaps);

First, the function mvpalab_permaps() computes the required permuted accuracy maps for each subject, randomly shuffling the original class labels. Then, mvpalab_permtest() generates the null distributions, determines the significance thresholds, collects significant clusters, computes cluster size distributions and corrects for multiple comparisons (FDR) to obtain the smallest cluster size to be considered significant. The variable stat is returned containing, among others, below and above chance significant clusters:

stats.clusters.sig   % Above chance clusters 
stats.clusters_.sig  % Below chance clusters

The default parameters for this analysis can be modified in the MVPAlab configuration file as follows:

cfg.stats.nper   = 100;
cfg.stats.nperg  = 1e5;
cfg.stats.pgroup = 99.9;
cfg.stats.pclust = 99.9;

cfg.stats.shownulldis = true;