FuseLib library for multimodal fusion of multiple modalities

FuseLib provides two alternatives, one implemented in Matlab (multimodalFusionlib), providing several algorithms fusion algorithms and one in python (probabilisticFusionModel) consisting of a probabilistic fusion model. For the version in Matlab, the user can choose for a specific fusion algorithm from the available ones, or test all of them for finding the most suitable one. The early fusion is perfomed using the concatenation of all input features, corresponding to all the modalities, after a normalization step which rescales the features in the same interval. The late fusion is performed at the end of the processing pipeline, after training each modality independently using machine learning algorithms. The multimodal fusion function can be called using a set of parameters containing: the Excel file location, the number of modalities, the range of column indexes for each modality, the labels column index and the name of the output file. The input parameters file (multimodalFusion_parametersFile.txt) is optional and is useful in case the user wants to customize the different available options, otherwise default values will be used. Usage example: function [featureLevelFusionAccuracy, decisionLevelFusionAccuracy] = multimodalFusion (datasetPath, inputExcelFile, labelsColumn, numberOfModalities, rangeModalityFeatures, outputTxTFile) The results of the analysis will be stored in the outputTxTFile.txt, containing the accuracy for each modality, the best classification method, the results for early and late fusion, as well as the best fusion algorithm. The library is based on a matlab pattern recognition toolbox, PRTools, developed by the TUDelft Pattern Recognition Group and available at: http://37steps.com/software/, which needs to be downloaded and placed in the multimodalFusionlib folder. For the version in python, the fusion process is applied to the retrieved signals from several modalities (e.g. activity, medical data), while its output could be sent to the module responsible for taking decisions regarding the user’s health status or to the database where the user-specific profile is stored. The fusion process of the input information takes into account the reliability of each source of information, using a weighted probabilistic model, where each modality is weighted according to its contribution in the fusion scheme. The input to the FuseLib probabilistic model component consists of three json files, stored in the corresponding subfolders inside the input folder. We chose to use this configuration with 3 modalities, but the model can be used with more or less modalities, depending on the user’s needs. The example file for performing the analysis is called probabilisticFusionModel.py and contains as parameters: the list of user ids, the date of the analysis and the investigatedPeriodinDays. Finally, the results of the analysis are stored in the output folder in a file named MF_analysisdate.json, where the analysisdate is the actual date parameter. The file contains the considered events, their deviations and the computed probabilities for each condition, along with the conditional probabilities, for assessing which one was more important in influencing the final results.