-
Notifications
You must be signed in to change notification settings - Fork 0
/
presentation.tex
233 lines (155 loc) · 17.1 KB
/
presentation.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
\environment envpresentation
\setlayerframed[footy][frame=off, x=.01\pagewidth, y=.0455\pageheight, align=flushleft, width=.88\pagewidth, height=.147\pageheight]%
{\color[white]{\setupbodyfont[52pt]\setupinterlinespace[line=0pt]\setupwhitespace[none]Using FAIMS Mobile for Field Data Recording \vfill}}
\setlayerframed[footy][frame=off, x=.01\pagewidth, y=.1465\pageheight, align=center]%
{\color[white]{\setupbodyfont[20pt]\setupinterlinespace[line=0pt]\setupwhitespace[none]Brian Ballsun-Stanton (Macquarie University), Jens Klump (CSIRO), Shawn Ross (Macquarie University)}}
\definebar[a][color=blue]
\starttext
\vfill
\setupcombinations[distance=10px, width=\textwidth, align=flushleft, style={\setupbodyfont[14pt]}]
\startcombination[4*2]%
{\externalfigure[noteBook1.jpg][width=.25\textwidth]}{Geosampling data was initially collected via notebook.}
{\externalfigure[noteBook2.jpg][width=.125\textwidth]\externalfigure[noteBook3.jpg][width=.125\textwidth]}{CSIRO Researchers then moved to pre-printed field notebooks for more accurate data entry. }
{\externalfigure[CSIROWireframe.pdf][width=.2\textwidth]}{Imported into FAIMS Mobile, the CSIRO workflow was deployed for multi-device field recording.}
{\setupcombinations[distance=5px, width=fit]
\startcombination[2*2]
{\externalfigure[dataSchema.png][width=.11\textwidth]}{Data Schema (XML)}
{\externalfigure[uiLogic.png][width=.11\textwidth]}{Logic (Beanshell)}
{\externalfigure[uiSchema.png][width=.11\textwidth]}{UI Schema (XML)}
\stopcombination}
{Three customisable primary files implement a model, view, and controller field data recording system.}
%nextrow
{\externalfigure[app2.png][width=.1\textwidth] \externalfigure[app3.png][width=.1\textwidth]}{After extensive testing, this recording system (\quote{module}) was deployed. These two screenshots from the app show multi-column formats, radiobuttons, checkboxes, and hierarchical dropdowns.}
{\externalfigure[establishingShot.jpg][width=.24\textwidth]}{Data was collected on multiple tablets in remote field locations, offline.}
{\externalfigure[faimsInTruck.jpg][width=.24\textwidth]}{FAIMS Mobile allows asynchronous work on tablets with eventual synchronisation, when the tablets connect to a local or online network. A local server, shown here, was mounted in a truck for fieldwork.}
{\externalfigure[exportData.png][width=.24\textwidth]}{After return to base, data was exported to shapefiles, a sqlite database, and CSVs. Upon export, all images were automatically renamed and tagged with metadata using information from the associated record.}
\stopcombination
\page
\setupcombinations[distance=10px, width=\textwidth, align={normal, hanging, hz}]
\startcombination[2*1]
{\externalfigure[dknf.pdf][width=.49\textwidth]}{
Records are defined in 3 logical tables\footnote. \quote{Rows} are defined in the ArchEntity table, which also holds cruical GIS data. \quote{Columns} are defined by the attribute and Ideal Entity tables. The Attribute table defines what attributes are possible, their names, and their list/export formats. The Ideal Entity table defines which attributes belong to which entity. By defining these tables in DML (Data {\em Manipulation} Language) rather than DDL (Data {\em Definition} Language), the structure of the database remains consistent. This consistent structure allows for significant query reuse and allows us to dynamically script the fields of a workflow {\em after} all the fundamental data interactions of FAIMS Mobile have been rewritten.
\page
\vfill
\vfill
{\tfx Image by Geoff Matheson.}
}
{\externalfigure[oss.pdf][width=.49\textwidth]}{
The three {\em sine qua non} open-source libraries of this project are:
\startitemize[joinedup, packed]
\item JavaRosa which allows us to parse XML into native android elements;
\item BeanShell which allows us to include a java-like scripting language;
\item Sqlite which is a supremely stable single-user database;
\stopitemize
By relying on stable, free, and public code we did not have to reinvent the wheel. We could, instead, focus on implementing our data schema and the novel features required by our environment.
}
\stopcombination
\page
\setuplist
[section]
[alternative=e, color=blue, style={\tfc}, prefix=yes, width={\textwidth}]
\vfill
{\tfc Quick Navigation:}
\placecontent
\vfill
{\tfb Abstract:}
FAIMS Mobile was originally developed for field data recording in archaeology. It has since been adapted for use in other disciplines.
The app is based on open-source software with open data formats to allow easy federation with other data formats. This presentation describes the internal architecture of FAIMS Mobile and how it can be used.
\vfill
\section{CSIRO Workflow}
FAIMS Mobile is an open-source software platform (comprising an android client and ruby server on ubuntu) funded by the Australian Research Council, the National eResearch Tools and Resources scheme (Australia), and six Australian universities. It is designed to provide a means of collecting rich, geospatial, and multi-media field data on multiple tablets with no network connectivity in the middle of nowhere. While originally intended to support archaeologists, FAIMS Mobile provided a sufficiently general field recording framework to allow for geochemical and biological sampling by multiple teams of CSIRO researchers.
\placefigure[here, none][fig1]{}{\externalfigure[noteBook1.jpg][width=.55\textwidth]}
Critical to the narrative is that the data design implemented by FAIMS Mobile existed first as data collected freehand in a field notebook. Field researchers have their own workflows to reflect methodologies and {\em imposition} of workflow changes, however slight, can render data incommensurate.
\page
\placefigure[here,none]{}{
\startcombination[2*1]
{\externalfigure[noteBook2.jpg][width=.35\textwidth]}{}
{\externalfigure[noteBook3.jpg][width=.35\textwidth]}{}
\stopcombination}
The next stage of field data collection entailed typesetting data collection forms based on the field notebook. These spiral bound data collection forms embodied the workflows of geosampling and critically moved the workflow from knowledge-in-the-head where the workflow was {\em present-at-hand} (i.e. after consideration of the tool, the researcher knew what data to enter next from their personal knowledge of the workflow) to {\em ready-to-hand.} The ready-to-hand workflow, embodied in these typeset forms, allowed the workflow to be embodied in the page: reducing errors, reducing cognitive load, and allowing the researcher to focus on the field research, instead of the methods of research.
\page
\startcombination[2*1]
{\externalfigure[CSIROWireframe.pdf][width=.58\textwidth]}{}
{\framed[width=.4\textwidth, height=\textheight, align={nothyphenated,verytolerant}, frame=off]{\setupwhitespace[big]Imported into FAIMS Mobile, the CSIRO workflow was deployed for the multi-device field recording. This workflow implements a waterfall methodology. Due to the requirements of offline field research and the asynchronous sync, {\em changes} to a module in the field are quite difficult and so we've embraced a waterfall model to get the design \quote{right} in the first place.
This wireframe diagram diagrams every attribute to appear in the module as well as the workflows of movement from screen to screen. It is both intended as a representational explainer as well as a design document. It allows everyone to understand the module to be in a way that reduces misunderstandings. This wireframe demonstrates how four different field collection methods: Water samples, rock samples, soil samples, and plant samples were all incorporated into the same data collection module.
\vfill}
}{}
\stopcombination
\page
\startcombination[2*2]
{\externalfigure[dataSchema.png][width=.49\textwidth]}{Data Schema (XML)}
{\externalfigure[uiLogic.png][width=.49\textwidth]}{Logic (Beanshell)}
{\externalfigure[uiSchema.png][width=.49\textwidth]}{UI Schema (XML)}
{\framed[width=.49\textwidth, height=.4\textheight, align={nothyphenated,verytolerant}, frame=off]{These are the primary files which implement a scriptable model (data schema), view (ui schema), and controller (ui logic) field data recording implementation. By scripting a field recording workflow, FAIMS Mobile itself can function on supporting field recording in network-degraded environments, and individual \quote{modules} (functioning sets of these scripts) can focus on implementing highly specific and customised workflows.}}{}
\stopcombination
\page
\startcombination[3*1]
{\externalfigure[app1.png][width=.39\textwidth]}{Multiple Ph/Eh measurements can be taken. We have support for a dynamic UI, so the number of measurements is finite, but unbounded. The \quote{best} measurement can then be written back into the parent entity.}
{\externalfigure[app2.png][width=.25\textwidth]}{Multi-column layouts are possible for \quote{tabular} data. The choice of the number of columns is mainly a choice of device size: smaller androids do not deal with many columns well.}
{\externalfigure[app3.png][width=.25\textwidth]}{We support checkboxes, drop-downs, and radio buttons. Dropdowns can have a hierarchical navigation, for rapid selection among hundreds of choices.}
\stopcombination
\blank
These are three screens from the FAIMS Mobile module showing some of the configurations of attributes suitable for this workflow. Of note, each attribute has annotation (the little pencil) and certainty (the little ?), allowing users, with a long-press, to \quote{write in the margins} of that attribute. The circled-i icon represents a description or information box, allowing descriptive or assistive text to appear for the attribute and any elements of the constrained vocabularies it has, thereby providing the benefit of a contextual field manual. Records autosave, meaning users can interact with the module in much the same way they do with paper and pencil.
\page
\startcombination[2*1]
{\externalfigure[establishingShot.jpg][width=.49\textwidth]}{}
{\externalfigure[faimsInTruck.jpg][width=.48\textwidth]}{Alistair White checking on the server mounted in his truck.}
\stopcombination
Data was collected on multiple tablets in the field. These tablets were offline to save battery. Different sampling teams (Water, rock, soil, and plant) all went out to rural Australia and were able to use this module to record data, take pictures, and take GIS points all in the same framework. The data, collected at the same time while tablets were offline, was able to sync after the tablets came back into server range. This synced data then resulted in a singular export, with all data present.
\page
\startcombination[2*1]
{\externalfigure[export.png][width=.49\textwidth]}{}
{\externalfigure[exportData.png][width=.49\textwidth]}{}
\stopcombination
After return to base, data exported (via customisable exporter) into shapefiles, a sqlite database, and CSVs. All pictures are renamed to the record they belong to and tagged with their record's metadata. This exporter (https://github.com/FAIMS/shapefileExport) is also completely customisable, up to and including running arbitrary linux programs. This one uses imagemagick and mogrify to properly export photos as well as spatialite-tool to export shapefiles.
The exporter is a shell script calling python scripts all hosted in their own github repository. This allows for arbitrary customisation on export to whatever format is appropriate. We use spatialite to export shapefiles and exiftool to write identifiers, authors, creation times, and the entire record they come from {\em to} each picture. By writing the exifdata to each picture, we ensure that the pictures, even if separated, retain their connection to the original record.
\section{FAIMS Internal Architecture}
\startcombination[2*1]
{\externalfigure[dknf.pdf][width=.49\textwidth]}{{\tfx Image by Geoff Matheson.}
Each attribute has four sub-attributes reflecting the needs of field data recording and can be multivalued if multiple rows share a timestamp. An attribute can comprise a set of:
\startitemize[packed, joinedup]
\item a constrained vocabulary (dropdowns, checkboxes, and radio buttons);
\item a unconstrained measurement;
\item an annotation (to represent a way of scribbling in the margins); and
\item a certainty (to otherwise rate uncertainty of the data reliability).
\stopitemize
By combining these in a single \quote{measurement.} Highly nuanced but machine readable data can be recorded in such a way as to fit the needs of the recording workflow.
}
{\framed[width=.49\textwidth, height=.49\textheight, align={nothyphenated,verytolerant}, frame=off]{\setupwhitespace[big]The fundamental innovation of FAIMS Mobile is the domain-key normal form append only datastore. Each record is identified by two naturally-unique identifiers: the user and the time of creation. Beyond that, every action is timestamped to allow for full histories and audit trails. Because every record and every action is timestamped and unique to a user, we can combine different versions of the database (i.e. those created on multiple devices over a week's trek through the Australian Outback) without any risk of clobbering edits or data. The most recent activity is \quote{true}.
Records are defined in 3 logical tables. \quote{Rows} are defined in the ArchEntity table, which also holds cruical GIS data. \quote{Columns} are defined by the attribute and Ideal Entity tables. The Attribute table defines what attributes are possible, their names, and their list/export formats. The Ideal Entity table defines which attributes belong to which entity. By defining these tables in DML (data manipulation language) rather than DDL (Data Definition Language), the structure of the database remains consistent. This structure allows for query reuse and allows us to script the fields of a workflow after all the fundamental data interactions of FAIMS Mobile have been written.
\vfill
}}{}
\stopcombination
\section{Append-Only Datastore}
\placefigure[here,none]{}{\externalfigure[eventLog.png][width=.70\textwidth]}
{\setupwhitespace[big]
This domain-key structure is also necessary to support the append only design. As each \quote{event} (insertion, updates, and \quote{deletion}) occupies its own row, we use GROUP BY and HAVING max(timestamp) to emit the latest versions of each attribute. Event uniqueness is guaranteed by UUID (user creation + time of creation due to the length limits of integers as primary keys in Sqlite), acting userid, and time of event. Therefore, by virtue of the need of having \quote{eventually consistent} data stores, we also have a complete action log for every record: it shows when each attribute was edited and by whom. This allows granular control and review of records, as individual attributes can be \quote{rolled back} to a more authoritative/correct state by users on the server.
The append-only design also protects against data-loss, as \quote{deletions} are merely a flag on the record which hides it from normal view. Thus, this database is designed to preserve user actions at all costs, allowing differences in datastores to be sent to the server and thereby distributed to all devices. This also has the virtue, so long as devices sync relatively often, of creating a complete backup of the datastore on every device, further armouring the database against mischance.}
\section{Open-source Heritage graph}
\startcombination[2*1]
{\externalfigure[oss.pdf][width=.5\textwidth]}{}
{\framed[width=.5\textwidth, height=.8\textheight, align={nothyphenated,verytolerant}, frame=off]{\setupwhitespace[big]%
FAIMS Mobile stands on the shoulders of giants. The only way an app this complex would be possible would be via the contributions of many open-source projects -- which is why our own work is released under the GPLv3. Crucial technologies include:
\startitemize[joinedup, packed]
\item JavaRosa which allows us to parse XML into native android elements;
\item NativeCSS which allows us to include some runtime stylings for elements;
\item BeanShell which allows us to dynamically include a java-like scripting language;
\item Spatialite which allows GIS operations inside the database;
\item Sqlite which is a supremely stable single-user database;
\item Ruby, Apache, Linux which allows us to write a sophisticated server running on a completely open source stack.
\stopitemize
Choices of the fundamental libraries were made in 2012, when the project was commissioned as part of NeCTAR (National eResearch Collaboration Tools and Resources project) eResearch tools initiative and the libraries remain solid (if slower than some newer iterations) for the next few years to come.
}}{}
\stopcombination
\page
~
\startitemize
\item To try this module, download FAIMS to your Android 4.4+ device, and download the CSIRO Geochemical Sample Collection module from the demo server.
\item For additional user guidelines, please see our \quote{cheatsheets} at: https://www.fedarch.org/resources/handouts.pdf
\item You can make your own data-collection modules for free via the user to developer documentation at https://www.fedarch.org/support/2.
\item Code for this module is available at: https://github.com/FAIMS/CSIRO-Geochemistry-Sampling
\item Source code for this presentation (compiled in \ConTeXt)~is available at: https://github.com/FAIMS/EGU2016-Presentation
\item If you have any questions, contact enquiries@fedarch.org.
\stopitemize
%\stopcombination
\stoptext