-
Notifications
You must be signed in to change notification settings - Fork 0
/
report.tex
470 lines (380 loc) · 18.3 KB
/
report.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
\documentclass{article}
\input{packages}
\input{acro}
\graphicspath{ {./images/} }
\begin{document}
\input{title}
\cleardoublepage
% TOC %
\pagenumbering{roman}
\tableofcontents
\newpage
\cleardoublepage
\listoffigures
\listoftables
\cleardoublepage
% This is a regular LaTeX comment
\pagenumbering{arabic}
\section{Introduction}
With the growing technological advances, the quantity of healthcare related data produced around the world increased exponentially \cite{choi_generating_2017,henry_adoption_2016}.
With this, the potential for harvesting this data also increases. The value locked within this data could help provide better healthcare with new information about diseases,
drugs, and preventive therapies. It can also help create better health information systems, meaning an overall better clinical practice \cite{ISI:000502534100049}. But for this to happen, the data must reach the capable hands at the right time.
But the release of clinical data has several barriers attached and rightly so. The leakage of patient’s privacy can break the confidence of the population in the healthcare professionals and institutions. Patient safety
and privacy should be kept at all costs and compromise should be kept to minimal values. However, the current mechanisms for privacy maintenance are very long, bureaucratic and time consuming, nationally \cite{comissao_nacional_protecao_de_dados_principios_2015} and internationally \cite{office_for_civil_rights_guidance_2013}. The current scenario and general methods for privacy safeguards are related with pseudo-anonymization techniques.
The removal of certain attributes, identifier modification, code grouping, or discretization are some of the methodologies. But not even these are totally safe \cite{el_emam_systematic_2011}.
Synthetic data appear as an alternative for clinical data sharing, promising great data utility with minimal privacy concerns. Synthetic data is data that is generated automatically through programmatic processes. This is especially impactful for the case at hand since synthetic data has no explicit connection with the original data.\\
This document is aimed for providing a technical overview of a synthetic/real dataset pair and evaluate the synthetically created dataset in terms of utility and privacy.
Not all columns of the original dataset will be assessed, since columns will be filtered by the synthetic columns, since synthesis mechanisms often reduce columns due to identifiers (not needed) and high null columns. \\
This report aims to facilitate the assessment of sharing synthetic data with third parties.
\section{Dataset Description}
The final dataset have the variables showed in table \ref{tab:desc}.\\
\begin{longtable}{lll}
\caption{Dataset Variables Description}\label{tab:desc}\\
\toprule
Columns & dtypes & Type \\
\midrule
\endhead
\midrule
\multicolumn{3}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
Age & float64 & Continuous \\
sex & object & Categorical \\
cp & object & Categorical \\
trestbps & float64 & Continuous \\
chol & float64 & Continuous \\
fbs & object & Categorical \\
restecg & object & Categorical \\
thalach & float64 & Continuous \\
exang & object & Categorical \\
oldpeak & float64 & Continuous \\
slope & object & Categorical \\
ca & object & Categorical \\
thal & object & Categorical \\
num & object & Categorical \\
\end{longtable}
\\
The removed columns were:\\
\begin{itemize}
\end{itemize}
\subsection{Brief Variable Overview}
The distribution of the variables is shown in table \ref{tab:realdatadesc}. The values are for numeric data only.\\
\\
\begin{longtable}{lrrrrr}
\caption{Original Data Description}\label{Realdatadesc}\\
\toprule
{} & mean & std & 25\% & 50\% & 75\% \\
\midrule
\endhead
\midrule
\multicolumn{6}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
Age & 54.4 & 9.0 & 48.0 & 56.0 & 61.0 \\
trestbps & 131.7 & 17.6 & 120.0 & 130.0 & 140.0 \\
chol & 246.7 & 51.8 & 211.0 & 241.0 & 275.0 \\
thalach & 149.6 & 22.9 & 133.5 & 153.0 & 166.0 \\
oldpeak & 1.0 & 1.2 & 0.0 & 0.8 & 1.6 \\
\end{longtable}
\\
\\
\begin{longtable}{lrrrrr}
\caption{Synthetic Data Description}\label{Synth data desc}\\
\toprule
{} & mean & std & 25\% & 50\% & 75\% \\
\midrule
\endhead
\midrule
\multicolumn{6}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
Age & 54.1 & 9.0 & 47.0 & 55.0 & 60.0 \\
trestbps & 131.8 & 17.0 & 120.0 & 130.0 & 140.0 \\
chol & 246.4 & 47.8 & 212.0 & 243.0 & 274.5 \\
thalach & 149.6 & 25.3 & 137.5 & 156.0 & 168.0 \\
oldpeak & 1.1 & 1.2 & 0.0 & 0.9 & 1.6 \\
\end{longtable}
\\
\\
\subsection{Null Overview}
Concerning null/empty cells for each column, the table \ref{tab:nullcomparasion} presents the comparison of the two datasets.\\
\begin{longtable}{lrr}
\caption{Null values Comparison}\label{tab:nullcomparison}\\
\toprule
Column & Real & Synthetic \\
\midrule
\endhead
\midrule
\multicolumn{3}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
Age & 0 & 0 \\
sex & 0 & 0 \\
cp & 0 & 0 \\
trestbps & 0 & 0 \\
chol & 0 & 0 \\
fbs & 0 & 0 \\
restecg & 0 & 0 \\
thalach & 0 & 0 \\
exang & 0 & 0 \\
oldpeak & 0 & 0 \\
slope & 0 & 0 \\
ca & 0 & 0 \\
thal & 0 & 0 \\
num & 0 & 0 \\
\end{longtable}
\\
For the rest of the reports, when null are removed or imputed, it shall be indicated.
\section{Utility evaluation}\\
This section aims tom assess the quality of the synthetic dataset by assessing the similarities with the original one. The subsection \ref{sec:visual} will be concerned with only visual methods and the following subsection \ref{sec:quant} with numerical comparison.\\
\subsection{Visual} \label{sec:visual}
This section is about visual methods for assessing the quality of the generated data.\\
\subsubsection{Heatmaps}
Heatmaps are concerning the correlations between all continuous variables. The more figure \ref{fig:heatmapreal} and figure \ref{fig:heatmapsynth} are alike, the more similar are the continuous variables relations between all columns.
\begin{figure}[h!]
\centering
\includegraphics[width=120mm]{heatmap_real}
\caption{Heatmap for Real Dataset}
\label{fig:heatmapreal}
\end{figure}
\\
\begin{figure}[h!]
\centering
\includegraphics[width=120mm]{heatmap_synth}
\caption{Heatmap for Synthetic Dataset}
\label{fig:heatmapsynth}
\end{figure}
\subsubsection{Plotting Categorical Variables}
Categorical variables will be plotted as count bar plots.
\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{images/categorical_plot_0.png}
\caption{Categorical Variable Plots 1}
\label{fig:categorical1}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{images/categorical_plot_1.png}
\caption{Categorical Variable Plots 2}
\label{fig:categorical2}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{images/categorical_plot_2.png}
\caption{Categorical Variable Plots 3}
\label{fig:categorical2}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{images/categorical_plot_3.png}
\caption{Categorical Variable Plots 4}
\label{fig:categorical3}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{images/categorical_plot_4.png}
\caption{Categorical Variable Plots 5}
\label{fig:categorical4}
\end{figure}
\subsubsection{Plotting Continuous Variables}
Continuous variables will be plotted as density plots. See figure \ref{fig:continuous}.
\begin{figure}[h!]
\centering
\includegraphics[width=\textwidth]{images/continuous_plot_0.png}
\caption{Categorical Variable Plots}
\label{fig:continuous}
\end{figure}
\subsubsection{Pairplot}
\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{pairplot}
\caption{PairPlot}
\label{fig:pairplot}
\end{figure}
\subsection{Quantitative Evaluation}\label{sec:quant}
This section hopes to bring more efficient methods of assessing the quality of a synthetic dataset. nowadays there are not unanimous metrics regarding this subject. For this reason, several different metrics are presented.\\
Columnwise metrics will assess how similar the equivalent columns are. The tablewise metrics will try to measure how the datasets as a whole are similar.
\subsubsection{Columnwise}
\paragraph{Kolmogorov–Smirnov test}
In statistics, the \textit{Kolmogorov–Smirnov} (\acrshort{kstest}) test is a nonparametric test of the equality of continuous, one-dimensional probability distributions that can be used to compare a sample with a reference probability distribution (one-sample K–S test), or to compare two samples (two-sample K–S test). It is named after Andrey Kolmogorov and Nikolai Smirnov.
The two-sample K–S test is one of the most useful and general nonparametric methods for comparing two samples, as it is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples \cite{kstest_wiki}. The results for the data are:\\
\begin{longtable}{lr}
\caption{K-S test Results}\label{tab:ks}\\
\toprule
{} & Value \\
\midrule
\endhead
\midrule
\multicolumn{2}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
Age & 0.97 \\
trestbps & 0.95 \\
chol & 0.98 \\
thalach & 0.95 \\
oldpeak & 0.93 \\
\end{longtable}
\paragraph{Chi-squared test}
This metric uses the chi-squared test to compare the distributions of the two categorical columns. It returns the resulting p-value so that a small value indicates that we can reject the null hypothesis (i.e. and suggests that the distributions are different). So higher p-value, chance of being similar is big, while small value indicates a small chance of being similar.\\
\begin{longtable}{lr}
\caption{Chi-Squared test Results}\label{tab:cs}\\
\toprule
{} & Value \\
\midrule
\endhead
\midrule
\multicolumn{2}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
ca & 1.00 \\
cp & 1.00 \\
exang & 0.96 \\
fbs & 0.92 \\
num & 1.00 \\
restecg & 1.00 \\
sex & 0.97 \\
slope & 0.99 \\
thal & 1.00 \\
\end{longtable}
\paragraph{Distance Metrics}
In mathematical statistics, the \textbf{Kullback–Leibler divergence} ( \acrshort{kl}), (also called relative entropy), is a measure of how one probability distribution is different from a second, reference probability distribution.\\
In probability theory and statistics, the \textbf{Jensen–Shannon divergence} (\acrshort{jsd}) is a method of measuring the similarity between two probability distributions. It is also known as information radius (IRad) or total divergence to the average.It is based on the Kullback–Leibler divergence, with some notable (and useful) differences, including that it is symmetric and it always has a finite value. The square root of the Jensen–Shannon divergence is a metric often referred to as Jensen-Shannon distance.
In mathematics, the \textbf{Wasserstein distance} (\acrshort{wass}) or Kantorovich–Rubinstein metric is a distance function defined between probability distributions on a given metric space M.
Intuitively, if each distribution is viewed as a unit amount of earth (soil) piled on M, the metric is the minimum "cost" of turning one pile into the other, which is assumed to be the amount of earth that needs to be moved times the mean distance it has to be moved. Because of this analogy, the metric is known in computer science as the earth mover's distance.
The name "Wasserstein distance" was coined by R. L. Dobrushin in 1970, after the Russian mathematician Leonid Vaseršteĭn who introduced the concept in 1969. Most English-language publications use the German spelling "Wasserstein" (attributed to the name "Vaseršteĭn" being of German origin). The values of different distance metrics are present in the table \ref{tab:distance}.\\
\caption{Distance Metrics Results Aggregated}\label{tab:distance}
\begin{longtable}{lrrrr}
\caption{Distance Metrics results}\label{tab:distance}\\
\toprule
{} & min & average & max & std \\
\midrule
\endhead
\midrule
\multicolumn{5}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
KL & 0.0000 & 0.0101 & 0.0535 & 0.0167 \\
JSD & 0.0023 & 0.0323 & 0.0564 & 0.0182 \\
Wass & 0.0022 & 0.0198 & 0.0363 & 0.0116 \\
Entropy & 0.0000 & 0.0101 & 0.0535 & 0.0167 \\
\end{longtable}
\subsubsection{Tablewise}
\paragraph{DiscreteKLDivergence}
The value is 0.964.\\
\paragraph{ContinuousKLDivergence}
The value is 0.679.\\
\paragraph{BNLikelihood}
This metric fits a Bayesian Network to the real data and then evaluates the average likelihood of the rows from the synthetic data on it.
The value is 0.0024167528547249175.\\
\paragraph{BNLogLikelihood}
This metric fits a Bayesian Network to the real data and then evaluates the average log likelihood of the rows from the synthetic data on it.
The value is -7.464182894178309.\\
\paragraph{GMLogLikelihood}
This metric fits multiple Gaussian Mixture models to the real data and then evaluates the average log likelihood of the synthetic data on them.
The value is -56938.28151424706.\\
\paragraph{Coverage Support}
Association rule mining is one of the important data mining tasks that is nowadays applied to solve different kinds of problems in weak formalization fields. The main goal of association rule mining is to find all rules in datasets that satisfy some basic requirements, such as minimum support and minimum confidence. It was initially proposed by Agrawal [3] to solve the market basket problem in transactional datasets and now it has developed to solve many other problems, such as classification, clustering, etc. Classification and prediction are two important tasks of data mining that can be used to extract the hidden regularities from the given datasets to form accurate models (classifiers). Classification approaches aim to build classifiers (models) to predict the class label of a future data object. Such analysis can help to provide us with a better understanding of the data comprehensively. Classification methods have been widely used in many real-world applications, such as customer relationship management [4], medical diagnosis [5] and industrial design [6].
The value for real on synthetic supporting rules: 0.9558620689655173 \\
The value for synthetic on real supporting rules: 0.9788583509513742 \\
\paragraph{Cross-validation}
For this evaluation, we will make cross-validation with several machine-learning algorithms.
The data will be trained on synthetic and then test on real and synthetic. This will help assess how plausible it is to deliver the synthetic data for data analysis. If the test values are similar, the synthetic dataset may be used as a proxy of the real.\\
On the other hand, training on real and testing on real and synthetic will help assess how the synthetic dataset correctly created the relationships between all variables.
For this purpose, there are 3 evaluation metrics, real on real, synthetic on synthetic and cross evaluation. Training with real and testing on synthetic and the other way around. For integer columns, mean absolute error was used, for categorical f1 (macro) was used (multiclass classification).\\
\begin{longtable}{lrrr}
\caption{Same Dataset score}\label{tab:samedatascore}\\
\toprule
{} & Real & Synthetic & Ratio \\
\midrule
\endhead
\midrule
\multicolumn{4}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
Age & 0.23 & 0.08 & 2.81 \\
sex & 0.58 & 0.63 & 0.93 \\
cp & 0.37 & 0.40 & 0.92 \\
trestbps & 0.05 & -0.00 & -26.13 \\
chol & 0.04 & -0.11 & -0.38 \\
fbs & 0.49 & 0.53 & 0.93 \\
restecg & 0.41 & 0.43 & 0.96 \\
thalach & 0.31 & 0.36 & 0.85 \\
exang & 0.63 & 0.64 & 0.98 \\
oldpeak & 0.36 & 0.38 & 0.92 \\
slope & 0.47 & 0.58 & 0.80 \\
ca & 0.26 & 0.24 & 1.05 \\
thal & 0.37 & 0.45 & 0.81 \\
num & 0.25 & 0.32 & 0.77 \\
\end{longtable}
\begin{longtable}{lrr}
\caption{Different dataset score}\label{tab:difdatascore}\\
\toprule
{} & Real-Synthetic & Synthetic-Real \\
\midrule
\endhead
\midrule
\multicolumn{3}{r}{{Continued on next page}} \\
\midrule
\endfoot
\bottomrule
\endlastfoot
Age & -7.61 & -7.31 \\
sex & 0.45 & 0.55 \\
cp & 0.26 & 0.21 \\
trestbps & -14.68 & -13.27 \\
chol & -37.95 & -38.67 \\
fbs & 0.46 & 0.46 \\
restecg & 0.48 & 0.36 \\
thalach & -18.27 & -20.81 \\
exang & 0.50 & 0.49 \\
oldpeak & -0.94 & -0.94 \\
slope & 0.36 & 0.32 \\
ca & 0.21 & 0.22 \\
thal & 0.27 & 0.35 \\
num & 0.19 & 0.23 \\
\end{longtable}
\section{Privacy evaluation}
This section aims to assess how a synthetic dataset provides protection regarding privacy breaks. It will be assessed how secure the dataset that was synthesized is.
\subsection{standalone metrics}
\textbf{duplicates}\\
Both datasets were merged and duplicates were tested. The value found as a result of this assessment is 0.\\
Then a new assessment was found for every columns but one. The result for number of duplicates of $N-1$ columns is 0.
\subsection{Membership Inference attacks}
For this purpose, we fixed a column for the attacker to known which was the age [IDADE\_MATERNA].
The remaining fields used were:
\begin{itemize}
\item Age
\item fbs
\item chol
\item restecg
\end{itemize}
\\
With this, we have a total of 0 equal results between datasets.
\subsection{distance metrics}
For this, we calculated the euclidian distance between records.
For the cosine distance between samples in Real Dataset and synthetic dataset, the value is 0.016277151371331865.
\section{Conclusion and final remarks}
...
% Biblio
\cleardoublepage
\printglossary[type=\acronymtype]
\cleardoublepage
\bibliographystyle{abbrv}
\bibliography{evaluationbiblio}
\end{document}