-
Notifications
You must be signed in to change notification settings - Fork 1
/
training-and-evaluation.Rmd
323 lines (220 loc) · 12.3 KB
/
training-and-evaluation.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
---
title: "Training and evaluation of aggregation strategies"
author: "Andrzej Wójtowicz, Patryk Żywica"
output:
html_document:
toc: yes
pdf_document:
toc: yes
---
```{r cache=FALSE, echo=FALSE}
options(width=110)
knitr::opts_chunk$set(comment="")
knitr::read_chunk('training-and-evaluation.R')
```
Document generation date: `r Sys.time()`
# Executive summary
This document presents the process of training and evaluation of different aggregation operators and
thresholding strategies in medical diagnosis support under data incompleteness.
# Introduction
In the *Analytic datasets construction* document we described how to obtain two sets,
which consist of lists of interval-valued diagnoses contained in $[0,1]$:
* the first, where some patients' features are simulated to be missing,
* the second, where patients' features are missing due to decisions of physicians.
In the following document we describe how to choose and evaluate aggregation
strategies in a two-step scheme. Firstly, we utilize the dataset with simulated missing
values to optimize different aggregation strategies. Secondly, we use the dataset with
truly missing values to performance evaluation.
For sake of readibility, some code is externalized into separate R files.
```{r init, message=FALSE, warning=FALSE}
```
## Overview of supplementary files
### Initial configuration
An initial setup for the process is stored in `config.R` file. Besides setting
a seed for random procesures (`SEED` constant) and setting constants for
input/output files and directories (`*.DIR`/`*.FILE`/`*.LOCATION` constants),
there are some values which are important for the evaluation process.
As a measure of classification effectiveness we choose `PERFORMANCE.MEASURE`
= `r PERFORMANCE.MEASURE`. Since the lower values of this measure mean better performance,
`PERFORMANCE.MEASURE.DESC` = `r PERFORMANCE.MEASURE.DESC`.
To parallelize calculation process there will be used `THREADS` = `r THREADS` threads.
To estimate uncertainty performance of final models, we perform stratified bootstraping with `BOOTSTRAP_R` = `r BOOTSTRAP_R` replications.
### Diagnostic models
The uncertaintified diagnostic models are implemented in `methods.R` file and they
are breifly described in the *Analytic datasets construction* document.
### Aggregation operators and thresholding strategies
Aggregation operators and thresholding strategies are implemented in `aggregators.R`,
`aggregators-helpers.R` and `aggregators-optimize.R` files. A description of the
operators is beyond the technical scope of this document.
### Statistics
All statistical indicators are implemented in `stats.R` file. Most of them are standard
statistics: accuracy, sensitivity and specificity. These with suffix `.all` are
adjusted variations to extend statistic to three cases: malignant, benign, no diagnosis.
The statistic operates on six possible outcomes of the diagnosis:
* `TP` - true positive,
* `TN` - true negative,
* `FN` - false negative,
* `FP` - false positive,
* `N0` - no prediction when tumor was benign,
* `N1` - no prediction when tumor was malignant.
**Remark**: in reality, a classifier or an aggregation method may only output `NA` in case of no prediction; the distinction between `N0` and `N1` is introduced in the experiment only for the purpose of the performance evaluation.
A decisiveness means the ability to make a diagnosis ($0$ - a model never gives a prediction,
$1$ - a model always gives a prediction). A cost matrix results a sum of the six possible outcomes
with a given weights provided by an expert.
### Efficiacy calculations
Functions for efficacy calculation of the models and the aggregators are implementd in
`utils.R`. This calculation process will be described later in the document.
### Auxiliary functions
Debugging functions are implemented in `utils.R` file. A function for McNemar's test
is stored in `mcn-test.R` file.
# Loading the data
The datasets are stored in CSV files.
```{r read-datasets}
```
As it is written in the *Analytic datasets construction* document, the same structure
is used for both the training and test sets (`ds.training` and
`ds.test` data frames, respectively; in the previous step these data were stored in `training.data` and `test.data` variables). First $4$ columns of a data frame indicate the following:
1. `PatientId` - a patient's identifier in the original database,
2. `ObscureLevel` - a percentage of a patient's attributes with missing data,
3. `ObscureRepeat` - a number of the iteration for a given `ObscureLevel`
(only valid in simulation phase),
4. `MalignancyCharacter` - an actual diagnosis for a given patient.
Remaining columns contain lower and upper bounds of a given uncertaintified diagnostic model.
A sample overview of the datasets is following:
```{r show-datasets}
head(ds.training, n=3)
tail(ds.test, n=3)
```
# Setup for parallel calculations
Whole procedure is very time-consuming. In order to get results in reasonable
time, we make use of parallel calculations. In this step we prepare the environment to
be capable of such calculations (export variables to workers, etc.). All time-consuming
calculations are done by use of `usedLapply` function (either single-threaded or parallel).
```{r parallel-init}
```
# Training phase
In this step `ds.training` dataset is processed.
## Models
For the reference purposes we check how the diagnostic models, both original
and uncertaintified, perform on the simulated training set.
Firstly, for each patient we calculate outputs of each single diagnostic model. This is
achieved by serveral consequtive steps:
* `*.models.outcomes` for each row (patient) returns results of each `r length(METHODS)`
diagnostic model; if an upper bound is less than $0.5$, then a model predicts a benign tumor;
if lowe bound is greater or equal to $0.5$, then a model predicts a malignant tumor;
for each particular case an output can be one of the mentioned above outcomes (`TP`, `TN`, etc.).
* `aggregate.outcomes` combines the results of `*.models.outcomes`, namely for each obscuration
level and obscuration repeat it gathers outcomes of each model,
* `calculate.stats` takes the results of `aggregate.outcomes` and calculates performance
statistics with respect to the obscuration level and models.
Afterwards the results are converted into a long format by use of `melt` function. Finally,
a column is renamed to be more descriptive and three columns are added to latterly identify the
diagnostic models from aggregation operators.
The following code produces results for the original models:
```{r training-statistics-models-original}
```
The following code produces results for the uncertaintified models:
```{r training-statistics-models-uncertaintified}
```
## Aggregation operators and thresholding strategies
Secondly, similar procedure is performed for the aggregation operators:
* `outcomes.aggrs` data frame is constructed by the application of each aggregation operator
to the simulation data; `diags` is a vector of zeros and ones, which are converted to the mentioned
above six possible outcomes (`TP`, `TN`, etc.),
* `aggregate.outcomes` combines the results of `outcomes.aggrs`,
* `calculate.stats` takes the results of `aggregate.outcomes` and calculates performance
statistics with respect to the obscuration level and aggregation operators.
Afterwards the results are converted into a long format by use of `melt` function. Finally,
a column is renamed to be more descriptive and three columns are added to latterly identify the
aggregation operators.
```{r training-statistics-aggregators}
```
## Performance calculation
The simulation results for the models and aggregation operators are combined into one data frame.
```{r training-statistics-bind}
```
For a given statistic, the performance of each model and aggregation operator is calculated
as a mean value of the statistic over all considered levels of missing data.
Finally, three columns are added to identify the models and aggregation operators.
```{r training-statistics-performance-calculation}
```
The aggregation operators and thresholding strategies are optimised with regards to numerical
parameters. As the last step of the training phase, all statistics for the models and aggregation
strategies are combined.
```{r select-optimized-aggregators}
```
# Test phase
In this step `ds.test` datasets is processed.
After optimisation of the aggregation strategies on the training data, we will evaluate the
models and the aggregation operators on the data where patients' features are missing
due to decisions of a physician.
The procedure of performance calculation is very similar, hence the same functions will be used.
Because of relatively small `ds.test` dataset, this time we will not distinguish different
obscuration level - we will calculate statistics over all cases. To perform this operation,
we set the same missing data level for all patients (this step is made to achieve data
adjustment to the interface of the functions and it has no impact on final results).
```{r test-combine-obscuration-levels}
```
## Models
For the reference purposes we check how the diagnostic models, both original
and uncertaintified, perform on the test set.
As in the training phase, for each patient we calculate outputs of each single diagnostic model.
This is achieved by serveral consequtive steps:
* `*.models.outcomes` for each row (patient) returns results of each `r length(METHODS)`
diagnostic model; this can be one of the mentioned above outcomes (`TP`, `TN`, etc.).
* `aggregate.outcomes` combines the results of `*.models.outcomes`,
* `calculate.stats` takes the results of `aggregate.outcomes` and calculates performance
statistics with respect to the models.
Afterwards the results are converted into a long format by use of `melt` function. A column
is renamed to be more descriptive and three columns are added to identify the
diagnostic models from aggregation operators. Finally, the `ObscureLevel` column is removed, since
it is not used in the evaluation step.
The following code produces results for the original models:
```{r test-statistics-models-original}
```
The following code produces results for the uncertaintified models:
```{r test-statistics-models-uncertaintified}
```
Both steps perform stratified bootstraping to estimate performance uncertainty.
## Aggregation operators and thresholding strategies
Similar procedure is performed for the aggregation strategies:
* `outcomes.aggrs` data frame is constructed by the application of each aggregation operator
to the simulation data; `diags` is a vector of zeros and ones, which are converted to the mentioned
above six possible outcomes (`TP`, `TN`, etc.),
* `aggregate.outcomes` combines the results of `outcomes.aggrs`,
* `calculate.stats` takes the results of `aggregate.outcomes` and calculates performance
statistics with respect to the aggregation operators.
Afterwards the results are converted into a long format by use of `melt` function. A column
is renamed to be more descriptive and the `ObscureLevel` column is removed, since
it is not used in the evaluation step. Finally, three columns are added to identify the
aggregation operators.
```{r test-statistics-aggregators}
```
This step also performs stratified bootstraping to estimate performance uncertainty.
## Binding results
Eventually, we combine the statistics of the models and aggregation strategies
obtained on the test set.
```{r test-statistics-bind}
```
# Converting results for summary
To achieve better outlook on the results, several statistics are bound.
To statistics obtained on the test set we add those achieved on the training set.
```{r test-statistics-performance-bind-with-training}
```
Secondly, we convert statistics from long to wide format.
```{r convert-statistics-performance-to-wide-format}
```
As the last step of the test phase, we select a collection of promising aggregation
strategies which might be suitable to the medical criteria and implemented for further
performance validation. We also perform the McNemar's test among the selected aggregation
strategies and with relation to the uncertaintified models.
```{r aggregators-selection-and-statistical-tests}
```
# Clean up parallel calculations
When all calculations are done, we can close all connections to parallel workers.
```{r parallel-shutdown}
```
# Saving the results
Lastly, we save all results (and envirionment) to a binary R file.
```{r save-evaluation}
```