Feat/aut974_2 Added options to SmartPeakCLI (#429)

* Add option to CLI - WIP * Added unit test, improved option parser * minor cleaning * Logging file setup * Override parameters * Added unit tests * improved help message * Added input feature/mzml directory options and fixed output feature directories * set report output dir * Handled relative path for input files * Using options lists to handle several parameter or input file overrides * Cleaning * Added unit tests * Factorized code to get parameter schema from workflow * Added unit tests for outputs directories * Updated documentation * Fixed Utilities test * Change n_thread parameter for Task_options_change_output_dirs * Update docs/guide/cli.rst Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com> * Update docs/guide/cli.rst Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com> * Update docs/guide/cli.rst Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com> * Update docs/guide/cli.rst Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com> * Update docs/guide/cli.rst Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com> * Update docs/guide/cli.rst Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com> * Update docs/guide/cli.rst Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com> Co-authored-by: Douglas McCloskey <dmccloskey@users.noreply.github.com>
AutoFlowResearch · Oct 2, 2021 · 9cde5d5 · 9cde5d5
1 parent a627883
commit 9cde5d5
Show file tree

Hide file tree

Showing 17 changed files with 656 additions and 86 deletions.
diff --git a/docs/guide/cli.rst b/docs/guide/cli.rst
@@ -39,7 +39,7 @@ After running a workflow, SmartPeak offers saving the result of the computation
 By default all possible reports will be exported, i.e. ``PivotTable`` and ``FeatureDB``. 
 However user can decide which report type to save using ``--report`` option. 
 It is still possible to specify both with this option, e.g. ``--report pivottable featuredb``.
-If ``--output`` path is not given, the files are written to current working directory.
+If ``--output-features`` path is not given, the files are written to current working directory.
 
 .. tabs::
 
@@ -61,7 +61,7 @@ If ``--output`` path is not given, the files are written to current working dire
 Output directory
 ~~~~~~~~~~~~~~~~
 
-In order to save files in specific location, set an ``--output`` path (relative or absolute).
+In order to save files in specific location, set an ``--output-features`` path (relative or absolute), or --reports-out-dir for the reports directories.
 
 .. tabs::
 
@@ -70,14 +70,16 @@ In order to save files in specific location, set an ``--output`` path (relative
         ./SmartPeakCLI `
             --load-session C:/data/GCMS_SIM_Unknowns/sequence.csv `
             --report featuredb `
-            --output C:/data/reports
+            --reports-out-dir $HOME/data/reports `
+            --output-features $HOME/data/output_features
 
     .. code-tab:: bash
 
         ./SmartPeakCLI \
             --load-session $HOME/data/GCMS_SIM_Unknowns/sequence.csv \
             --report featuredb \
-            --output $HOME/data/reports
+            --reports-out-dir $HOME/data/reports `
+            --output-features $HOME/data/output_features
 
 .. warning::
     Notice that if given a path to non-existing directory, SmartPeak will attempt to create one.
@@ -97,15 +99,17 @@ Values are space separated list, case insensitive.
             --load-session C:/data/GCMS_SIM_Unknowns/sequence.csv `
             --report featuredb `
             --report-sample-types standard blank qc `
-            --output C:/data/reports
+            --reports-out-dir $HOME/data/reports `
+            --output-features $HOME/data/output_features
 
     .. code-tab:: bash
 
         ./SmartPeakCLI \
             --load-session $HOME/data/GCMS_SIM_Unknowns/sequence.csv \
             --report featuredb \
             --report-sample-types standard blank qc \
-            --output $HOME/data/reports
+            --reports-out-dir $HOME/data/reports \
+            --output-features $HOME/data/output_features
 
 .. note::
     List of all possible sample types can be found here: :ref:`sample-types`.
@@ -128,15 +132,17 @@ The same rules applies, an example below.
             --load-session C:/data/GCMS_SIM_Unknowns/sequence.csv `
             --report featuredb `
             --report-metadata peak_area logSN RT
-            --output C:/data/reports
+            --reports-out-dir $HOME/data/reports `
+            --output-features $HOME/data/output_features
 
     .. code-tab:: bash
 
         ./SmartPeakCLI \
             --load-session $HOME/data/GCMS_SIM_Unknowns/sequence.csv \
             --report featuredb \
             --report-metadata peak_area logSN RT \
-            --output $HOME/data/reports
+            --reports-out-dir $HOME/data/reports \
+            --output-features $HOME/data/output_features
 
 .. note::
     List of all possible values can be found here: :ref:`metadata`.
@@ -166,7 +172,8 @@ The CLI tool allows to modify the list of commands inline with an option ``--wor
                         PICK_MRM_FEATURES `
                         SELECT_FEATURES `
                         STORE_FEATURES `
-            --output C:/data/reports
+            --reports-out-dir $HOME/data/reports `
+            --output-features $HOME/data/output_features
 
     .. code-tab:: bash
 
@@ -181,7 +188,8 @@ The CLI tool allows to modify the list of commands inline with an option ``--wor
                         PICK_MRM_FEATURES \
                         SELECT_FEATURES \
                         STORE_FEATURES \
-            --output $HOME/data/reports
+            --reports-out-dir $HOME/data/reports \
+            --output-features $HOME/data/output_features
 
 .. note::
     The full list of commands with short descriptions is included here: :ref:`workflow-commands`.
@@ -208,7 +216,8 @@ which forces SmartPeak to continue with the workflow even if one or more integri
             --report featuredb `
             --integrity SAMPLE COMP_GROUP `
             --allow-inconsistent `
-            --output C:/data/reports
+            --reports-out-dir $HOME/data/reports `
+            --output-features $HOME/data/output_features
 
     .. code-tab:: bash
 
@@ -217,7 +226,8 @@ which forces SmartPeak to continue with the workflow even if one or more integri
             --report featuredb \
             --integrity SAMPLE COMP_GROUP \
             --allow-inconsistent \
-            --output $HOME/data/reports
+            --reports-out-dir $HOME/data/reports \
+            --output-features $HOME/data/output_features
 
 .. note::
     Refer to list of integrity checks here: :ref:`integrity-checks`.
@@ -235,35 +245,34 @@ Full Command-Line Interface
     This parameter is optional. The default value is ''.
 
     -l    --load-session  (required)
-    The path to sequence.csv file
+    The path to a session file, or to a directory with a structured file organization (session file is preferred)
 
     -r    --report
     Specify a report type to export.
     This parameter is optional. The default value is '[ FeatureDB PivotTable ]'.
 
     -rt   --report-sample-types
-    A list of sample types to include in the report. 
-    The full list can be found in documentation https://smartpeak.readthedocs.io   
+    A list of sample types to include in the report. The full list can be found in the documentation https://smartpeak.readthedocs.io
     This parameter is optional. The default value is '[ ALL ]'.
 
     -rm   --report-metadata
-    A list of metadata to include in the report. 
-    The full list can be found in documentation https://smartpeak.readthedocs.io
+    A list of metadata to include in the report. The full list can be found in the documentation https://smartpeak.readthedocs.io
     This parameter is optional. The default value is '[ ALL ]'.
 
+    -ro   --reports-out-dir
+    An absolute or relative path to an output directory for reports. Overrides the default location which is the current working directory. SmartPeak will create the given directory if one does not exist.
+    This parameter is optional. The default value is '.'.
+
     -w    --workflow
-    The workflow sequence as a list of commands, e.g. LOAD_DATA MAP_CHROMATOGRAMS ... 
-    Overrides the workflow settings loaded from the sequence file (with option --load-session)
+    The workflow sequence as a list of commands, e.g. LOAD_DATA MAP_CHROMATOGRAMS ... Overrides the workflow settings loaded from the sequence file (with option --load-session)
     This parameter is optional. The default value is '[ ]'.
 
-    -i    --integrity
-    Specify which integrity checks to run, available are: SAMPLE, COMP, 
-    COMP_GROUP, IS and ALL (runs all listed).
+    -t    --integrity
+    Specify which integrity checks to run, available are: SAMPLE, COMP, COMP_GROUP, IS and ALL (runs all listed).
     This parameter is optional. The default value is '[ NONE ]'.
 
     -a    --allow-inconsistent
-    Given that any integrity checks were specified with '--integrity', 
-    this option allow to run workflow even if one or more check has failed.
+    Given that any integrity checks were specified with '--integrity', this option allow to run workflow even if one or more check has failed.
     This parameter is optional. The default value is '0'.
 
     -v    --verbose
@@ -274,12 +283,30 @@ Full Command-Line Interface
     By default the console output is colored, this flag disables colors.
     This parameter is optional. The default value is '0'.
 
+    -pg   --disable-progressbar
+    Progress bar allows tracks the progress of the entire workflow. This option disables the progress bar.
+    This parameter is optional. The default value is '0'.
+
     -ld   --log-dir
-    The path to the log directory. Given directory has to exist. Overrides the default 
-    location for the log file: https://smartpeak.readthedocs.io/en/latest/guide/guistart.html#logs
+    The path to the log directory. The given directory has to exist. Overrides the default location for the log file: https://smartpeak.readthedocs.io/en/latest/guide/guistart.html#logs
     This parameter is optional. The default value is ''.
 
-    -o    --output
-    An absolute or relative path to an output directory. Overrides the default location 
-    which is the current working directory. SmartPeak will create given directory if one does not exist.
-    This parameter is optional. The default value is '.'.
+    -o    --output-features
+    An absolute or relative path to an output directory. Overrides the default location which is the current working directory. SmartPeak will create the given directory if one does not exist.
+    This parameter is optional. The default value is './features'.
+
+    -i    --input-features
+    An absolute or relative path to the input features directory. Overrides the default location which is the current working directory.
+    This parameter is optional. The default value is './features'.
+
+    -z    --mzml
+    An absolute or relative path to the mzML directory. Overrides the default location which is the mzML folder under the current working directory. SmartPeak will create given directory if one does not exist.
+    This parameter is optional. The default value is './mzML'.
+
+    -f    --input-file
+    Override input file. Ex: -f featureQCComponents="./featureQCComponents_new.csv".
+    This parameter is optional. The default value is '[ ]'.
+
+    -p    --parameter
+    Override parameter. Ex: '-p MRMFeatureFinderScoring:TransitionGroupPicker:peak_integration=smoothed'.
+    This parameter is optional. The default value is '[ ]'.
diff --git a/src/smartpeak/include/SmartPeak/cli/ApplicationSettings.h b/src/smartpeak/include/SmartPeak/cli/ApplicationSettings.h
@@ -85,6 +85,12 @@ struct ApplicationSettings {
         const std::string& option, 
         std::string log_msg="");
 
+    /**
+     * Returns a pair key/value from one string "key1=value1"
+     */ 
+    static std::pair<std::string, std::string> get_key_value_from_option(
+      const std::string& option);
+
 public:
     /* options */
     std::string load_session;
@@ -98,7 +104,12 @@ struct ApplicationSettings {
     bool disable_colors;
     bool disable_progressbar;
     std::string log_dir; 
-    std::string out_dir;
+    std::string features_out_dir;
+    std::string features_in_dir;
+    std::vector<std::string> input_files;
+    std::vector<std::string> parameters;
+    std::string mzml_dir;
+    std::string reports_out_dir;
 
 public:
     void validate_report() const;

diff --git a/src/smartpeak/include/SmartPeak/core/ApplicationHandler.h b/src/smartpeak/include/SmartPeak/core/ApplicationHandler.h
@@ -108,6 +108,11 @@ namespace SmartPeak
     */
     bool isFileSaved(const std::string& file_id) const;
 
+    /**
+    * @brief returns Parameter Schema of the current Workflow
+    */
+    ParameterSet getWorkflowParameterSchema();
+
     std::filesystem::path sequence_pathname_;
     std::filesystem::path main_dir_                = ".";
     std::filesystem::path mzML_dir_;

diff --git a/src/smartpeak/include/SmartPeak/core/ApplicationProcessor.h b/src/smartpeak/include/SmartPeak/core/ApplicationProcessor.h
@@ -30,6 +30,7 @@
 #include <SmartPeak/iface/ISequenceSegmentProcessorObserver.h>
 #include <SmartPeak/iface/ISampleGroupProcessorObserver.h>
 #include <SmartPeak/core/ApplicationProcessorObservable.h>
+#include <SmartPeak/core/Parameters.h>
 #include <string>
 #include <vector>
 
@@ -89,9 +90,11 @@ namespace SmartPeak
     /* IFilePickerHandler */
     bool onFilePicked(const std::filesystem::path& filename, ApplicationHandler* application_handler) override;
 
-    std::optional<Filenames> filenames_;                            /// Pathnames to load
-    std::string      delimiter = ",";              /// String delimiter of the imported file
-    bool             checkConsistency = true;             /// Check consistency of data contained in files
+    std::optional<Filenames>    filenames_;           /// Pathnames to load - if not set, read it from session
+    std::optional<Filenames>    filenames_override_;  /// Pathnames override
+    std::optional<ParameterSet> parameters_override_; /// Parameters override
+    std::string      delimiter = ",";                 /// String delimiter of the imported file
+    bool             checkConsistency = true;         /// Check consistency of data contained in files
 
     LoadSession() = default;
     explicit LoadSession(ApplicationHandler& application_handler) : ApplicationProcessor(application_handler) {}
@@ -102,6 +105,10 @@ namespace SmartPeak
     /* IProcessorDescription */
     virtual std::string getName() const override { return "LOAD_SESSION"; }
     virtual std::string getDescription() const override { return "Load an existing session"; }
+
+  protected:
+    bool overrideFilenames();
+    bool overrideParameters();
   };
 
   struct SaveSession : ApplicationProcessor, IFilePickerHandler

diff --git a/src/smartpeak/include/SmartPeak/core/Filenames.h b/src/smartpeak/include/SmartPeak/core/Filenames.h
@@ -120,6 +120,11 @@ namespace SmartPeak
     */
     SessionDB& getSessionDB() { return session_db_; };
 
+    /**
+      @brief log content of this instance
+    */
+    void log() const;
+
   protected:
 
     struct FileName

diff --git a/src/smartpeak/include/SmartPeak/core/Parameters.h b/src/smartpeak/include/SmartPeak/core/Parameters.h
@@ -63,7 +63,7 @@ namespace SmartPeak
         {"type", "int"},
         {"value", "23"},
         {"description", "param1 description"},
-        {"tags", "tag1,tag2,tag3"}
+        {"tags", "tag1,tag2,tag3"},
         {"min","0"}, 
         {"max","10"}
       }

diff --git a/src/smartpeak/source/cli/ApplicationSettings.cpp b/src/smartpeak/source/cli/ApplicationSettings.cpp
@@ -44,10 +44,13 @@ void ApplicationSettings::define_options()
     m_parser.set_optional<std::vector<std::string>>("rm", "report-metadata", std::vector<std::string>{ "ALL" }, 
         "A list of metadata to include in the report. "
         "The full list can be found in documentation https://smartpeak.readthedocs.io");
-    m_parser.set_optional<std::vector<std::string>>("w", "workflow", std::vector<std::string>{}, 
+    m_parser.set_optional<std::string>("ro", "reports-out-dir", ".",
+      "An absolute or relative path to an output directory for reports. Overrides the default location which is the current working directory. "
+      "SmartPeak will create given directory if one does not exist.");
+    m_parser.set_optional<std::vector<std::string>>("w", "workflow", std::vector<std::string>{},
         "The workflow sequence as a list of commands, e.g. LOAD_DATA MAP_CHROMATOGRAMS ... "
         "Overrides the workflow settings loaded from the sequence file (with option --load-session)");
-    m_parser.set_optional<std::vector<std::string>>("i", "integrity", std::vector<std::string>{ "NONE" }, 
+    m_parser.set_optional<std::vector<std::string>>("t", "integrity", std::vector<std::string>{ "NONE" }, 
         "Specify which integrity checks to run, available are: SAMPLE, COMP, COMP_GROUP, IS and ALL (runs all listed).");
     m_parser.set_optional<bool>("a", "allow-inconsistent", false, 
         "Given that any integrity checks were specified with '--integrity', "
@@ -56,14 +59,23 @@ void ApplicationSettings::define_options()
         "Run SmartPeak in verbose mode, display more detailed information");
     m_parser.set_optional<bool>("d", "disable-colors", false, 
         "By default the console output is colored, this flag disables colors.");
-    m_parser.set_optional<bool>("p", "disable-progressbar", false, 
+    m_parser.set_optional<bool>("no-pg", "disable-progressbar", false, 
         "Progress bar allows to track the progress of the entire workflow. This option disables the progress bar.");
     m_parser.set_optional<std::string>("ld", "log-dir", "", 
         "The path to the log directory. Given directory has to exist. Overrides the default location for the log file: "
         "https://smartpeak.readthedocs.io/en/latest/guide/guistart.html#logs");
-    m_parser.set_optional<std::string>("o", "output", ".", 
+    m_parser.set_optional<std::string>("o", "output-features", "./features", 
         "An absolute or relative path to an output directory. Overrides the default location which is the current working directory. " 
         "SmartPeak will create given directory if one does not exist.");
+    m_parser.set_optional<std::string>("i", "input-features", "./features",
+        "An absolute or relative path to the input features directory. Overrides the default location which is the current working directory. ");
+    m_parser.set_optional<std::string>("z", "mzml", "./mzML",
+        "An absolute or relative path to the mzML directory. Overrides the default location which is the mzML folder under the current working directory. "
+        "SmartPeak will create given directory if one does not exist.");
+    m_parser.set_optional<std::vector<std::string>>("f", "input-file", {},
+        "Override input file. Ex: -f featureQCComponents=\"./featureQCComponents_new.csv\".");
+    m_parser.set_optional<std::vector<std::string>>("p", "parameter", {},
+        "Override parameter. Ex: '-p MRMFeatureFinderScoring:TransitionGroupPicker:peak_integration=smoothed'.");
     m_parser.run_and_exit_if_error();
 }
 
@@ -74,13 +86,18 @@ void ApplicationSettings::load_options()
     report_sample_types     = m_parser.get<std::vector<std::string>>("rt");
     report_metadata         = m_parser.get<std::vector<std::string>>("rm");
     workflow                = m_parser.get<std::vector<std::string>>("w");
-    integrity               = m_parser.get<std::vector<std::string>>("i");
+    integrity               = m_parser.get<std::vector<std::string>>("t");
     allow_inconsistent      = m_parser.get<bool>("a");
     verbose                 = m_parser.get<bool>("v");
     disable_colors          = m_parser.get<bool>("d");
-    disable_progressbar     = m_parser.get<bool>("p");
+    disable_progressbar     = m_parser.get<bool>("no-pg");
     log_dir                 = m_parser.get<std::string>("ld");
-    out_dir                 = m_parser.get<std::string>("o");
+    features_out_dir        = m_parser.get<std::string>("o");
+    features_in_dir         = m_parser.get<std::string>("i");
+    input_files             = m_parser.get<std::vector<std::string>>("f");
+    parameters              = m_parser.get<std::vector<std::string>>("p");
+    mzml_dir                = m_parser.get<std::string>("z");
+    reports_out_dir         = m_parser.get<std::string>("ro");
 }
 
 void ApplicationSettings::process_options()
@@ -119,12 +136,33 @@ bool ApplicationSettings::contains_option(
     return flag;
 }
 
+std::pair<std::string, std::string> ApplicationSettings::get_key_value_from_option(
+  const std::string& option)
+{
+  std::pair<std::string, std::string> key_value;
+  auto separator_pos = option.find("=");
+  if (separator_pos != option.npos)
+  {
+    std::string key = option.substr(0, separator_pos);
+    std::string value = option.substr((separator_pos + 1), option.size() - (separator_pos + 1));
+    if (!key.empty())
+    {
+      if ((value.size() > 1) && (value.front() == '\"') && (value.back() == '\"'))
+      {
+        value = value.substr(1, value.size() - 2);
+      }
+      key_value.first = key;
+      key_value.second = value;
+    }
+  };
+  return key_value;
+}
+
 void ApplicationSettings::validate_report() const
 {
     auto options = std::vector<std::string>{"FEATUREDB", "PIVOTTABLE", "ALL"};
     std::for_each(report.cbegin(), report.cend(), [&options](const auto& option) {
-        auto it = std::find(options.cbegin(), options.cend(), option);
-        if (it == std::cend(options))
+        if (std::find(options.cbegin(), options.cend(), option) == std::cend(options))
         {
             auto options_str = Utilities::join(options.cbegin(), options.cend(), ", ");
             throw std::invalid_argument(static_cast<std::ostringstream&&>(