diff --git a/.flake8 b/.flake8
new file mode 100644
index 0000000..7da1f96
--- /dev/null
+++ b/.flake8
@@ -0,0 +1,2 @@
+[flake8]
+max-line-length = 100
diff --git a/README.rst b/README.rst
index daf9c10..d0b5013 100644
--- a/README.rst
+++ b/README.rst
@@ -27,21 +27,28 @@
.. |Gitter| image:: https://badges.gitter.im/mlscorecheck.svg
.. _Gitter: https://app.gitter.im/#/room/!AmkvUevcfkobbwcNWS:gitter.im
-mlscorecheck: testing the consistency of binary classification performance scores
-*********************************************************************************
+
+mlscorecheck: testing the consistency of machine learning performance scores
+****************************************************************************
+
+.. contents::
+ :depth: 3
+
+Getting started
+===============
The purpose
-===========
+-----------
-Performance scores for binary classification are reported on a dataset and look suspicious (exceptionally high scores possibly due to typo, uncommon evaluation methodology, data leakage in preparation, incorrect use of statistics, etc.). With the tools implemented in the package ``mlscorecheck``, one can test if the reported performance scores are consistent with each other and the assumptions on the experimental setup up to the numerical uncertainty due to rounding/truncation/ceiling.
+Performance scores of a machine learning technique (binary/multiclass classification, regression) are reported on a dataset and look suspicious (exceptionally high scores possibly due to a typo, uncommon evaluation methodology, data leakage in preparation, incorrect use of statistics, etc.). With the tools implemented in the package ``mlscorecheck``, one can test if the reported performance scores are consistent with each other and the assumptions on the experimental setup up.
-Testing is as simple as the following example shows: the tested scores are inconsistent with a testset of 100 positive and 200 negative entries.
+Testing is as simple as the following example illustrated. Suppose the accuracy, sensitivity and specificity scores are reported for a binary classification testset consisting of p=100 and n=200 samples. All this information is supplied to the suitable test function and the result shows that that inconsistencies were identified: the scores could not be calculated from the confusion matrix of the testset:
.. code-block:: Python
- from mlscorecheck.check import check_1_testset_no_kfold_scores
+ from mlscorecheck.check.binary import check_1_testset_no_kfold
- result = check_1_testset_no_kfold_scores(testset={'p': 100, 'n': 200},
+ result = check_1_testset_no_kfold(testset={'p': 100, 'n': 200},
scores={'acc': 0.9567, 'sens': 0.8545, 'spec': 0.9734},
eps=1e-4)
result['inconsistency']
@@ -50,15 +57,16 @@ Testing is as simple as the following example shows: the tested scores are incon
The consistency tests are numerical and **not** statistical: if inconsistencies are identified, it means that either the assumptions on the experimental setup or the reported scores are incorrect.
Latest news
-===========
+-----------
-* the 1.0.0 version of the package is released
+* the 1.0.1 version of the package is released;
* the paper describing the numerical techniques is available as a preprint at: https://arxiv.org/abs/2310.12527
* the full documentation is available at: https://mlscorecheck.readthedocs.io/en/latest/
-* 10 test bundles including retina image processing datasets, preterm delivery prediction from electrohysterograms and skin lesion classification has been added.
+* 10 test bundles including retina image processing datasets, preterm delivery prediction from electrohysterograms and skin lesion classification has been added;
+* multiclass and regression tests added.
Citation
-========
+--------
If you use the package, please consider citing the following paper:
@@ -70,29 +78,20 @@ If you use the package, please consider citing the following paper:
year={2023}
}
-Contents
-========
-
-The contents of the repository:
-
-* ``mlscorecheck`` folder: the implementation of the consistency tests;
-* ``notebooks/illustration`` folder: the notebooks containing all working sample codes used throughout this README and the ReadTheDocs documentation;
-* ``notebooks/utils`` folder: utilities related to generate the algebraic solutions of the score functions, as well as the summary tables used for illustration;
-* ``tests`` folder: the unit and functional tests covering each line of code of the package.
-
Installation
-============
+------------
-The package has only basic requirements when used for consistency testing.
+The package has only basic requirements when used for consistency testing:
* ``numpy``
* ``pulp``
+* ``scikit-learn``
.. code-block:: bash
pip install numpy pulp
-In order to execute the tests, one also needs ``scikit-learn``, in order to test the computer algebra components or reproduce the algebraic solutions, either ``sympy`` or ``sage`` needs to be installed. The installation of ``sympy`` can be done in the usual way. To install ``sage`` in a ``conda`` environment, one needs to add the ``conda-forge`` channel first:
+In order to execute the unit tests for the computer algebra components or reproduce the algebraic solutions, either ``sympy`` or ``sage`` needs to be installed. The installation of ``sympy`` can be done in the usual way. To install ``sage`` in a ``conda`` environment, one needs to add the ``conda-forge`` channel first:
.. code-block:: bash
@@ -117,46 +116,25 @@ Alternatively, one can clone the latest version of the package from GitHub and i
Introduction
============
-Binary classification is one of the most fundamental tasks in machine learning. The evaluation of the performance of binary classification techniques, whether for original theoretical advancements or applications in specific fields, relies heavily on performance scores (https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers). Although reported performance scores are employed as primary indicators of research value, they often suffer from methodological problems, typos, and insufficient descriptions of experimental settings. These issues contribute to the replication crisis (https://en.wikipedia.org/wiki/Replication_crisis) and ultimately entire fields of research ([RV]_, [EHG]_). Even systematic reviews can suffer from using incomparable performance scores for ranking research papers [RV]_.
+The evaluation of the performance of machine learning techniques, whether for original theoretical advancements or applications in specific fields, relies heavily on performance scores (https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers). Although reported performance scores are employed as primary indicators of research value, they often suffer from methodological problems, typos, and insufficient descriptions of experimental settings. These issues contribute to the replication crisis (https://en.wikipedia.org/wiki/Replication_crisis) and ultimately entire fields of research ([RV]_, [EHG]_). Even systematic reviews can suffer from using incomparable performance scores for ranking research papers [RV]_.
-The majority of performance scores are calculated from the binary confusion matrix, or multiple confusion matrices aggregated across folds and/or datasets. For many commonly used experimental setups one can develop numerical techniques to test if there exists any confusion matrix (or matrices), compatible with the experiment and leading to the reported performance scores. This package implements such consistency tests for some common scenarios. We highlight that the developed tests cannot guarantee that the scores are surely calculated by some standards or a presumed evaluation protocol. However, *if the tests fail and inconsistencies are detected, it means that the scores are not calculated by the presumed protocols with certainty*. In this sense, the specificity of the test is 1.0, the inconsistencies being detected are inevitable.
+In practice, the performance scores cannot take any values independently, the scores reported for the same experiment are constrained by the experimental setup and need to express some internal consistency. For many commonly used experimental setups it is possible to develop numerical techniques to test if the scores could be the outcome of the presumed experiment on the presumed dataset. This package implements such consistency tests for some common experimental setups. We highlight that the developed tests cannot guarantee that the scores are surely calculated by some standards or a presumed evaluation protocol. However, *if the tests fail and inconsistencies are detected, it means that the scores are not calculated by the presumed protocols with certainty*. In this sense, the specificity of the test is 1.0, the inconsistencies being detected are inevitable.
For further information, see
* ReadTheDocs full documentation: https://mlscorecheck.readthedocs.io/en/latest/
* The preprint: https://arxiv.org/abs/2310.12527
-Use cases
-=========
+Preliminaries
+=============
+
+Requirements
+------------
In general, there are three inputs to the consistency testing functions:
* **the specification of the experiment**;
-* **the collection of available (reported) performance scores**: when aggregated performance scores (averages on folds or datasets) are reported, only accuracy (``acc``), sensitivity (``sens``), specificity (``spec``) and balanced accuracy (``bacc``) are supported; when cross-validation is not involved in the experimental setup, the list of supported scores reads as follows (with abbreviations in parentheses):
-
- * accuracy (``acc``),
- * sensitivity (``sens``),
- * specificity (``spec``),
- * positive predictive value (``ppv``),
- * negative predictive value (``npv``),
- * balanced accuracy (``bacc``),
- * f1(-positive) score (``f1``),
- * f1-negative score (``f1n``),
- * f-beta positive (``fbp``),
- * f-beta negative (``fbn``),
- * Fowlkes-Mallows index (``fm``),
- * unified performance measure (``upm``),
- * geometric mean (``gm``),
- * markedness (``mk``),
- * positive likelihood ratio (``lrp``),
- * negative likelihood ratio (``lrn``),
- * Matthews correlation coefficient (``mcc``),
- * bookmaker informedness (``bm``),
- * prevalence threshold (``pt``),
- * diagnostic odds ratio (``dor``),
- * Jaccard index (``ji``),
- * Cohen's kappa (``kappa``);
-
+* **the collection of available (reported) performance scores**;
* **the estimated numerical uncertainty**: the performance scores are usually shared with some finite precision, being rounded/ceiled/floored to ``k`` decimal places. The numerical uncertainty estimates the maximum difference of the reported score and its true value. For example, having the accuracy score 0.9489 published (4 decimal places), one can suppose that it is rounded, therefore, the numerical uncertainty is 0.00005 (10^(-4)/2). To be more conservative, one can assume that the score was ceiled or floored. In this case, the numerical uncertainty becomes 0.0001 (10^(-4)).
Specification of the experimental setup
@@ -230,27 +208,56 @@ A dataset and a folding constitute an *evaluation*, and many of the test functio
"folding": {"n_folds": 5, "n_repeats": 1,
"strategy": "stratified_sklearn"}}
-Testing the consistency of performance scores
----------------------------------------------
+A note on the *Score of Means* and *Mean of Scores* aggregations
+----------------------------------------------------------------
-Numerous experimental setups are supported by the package. In this section we go through them one by one giving some examples of possible use cases.
+When it comes to the aggregation of scores (either over multiple folds, multiple datasets or both), there are two approaches in the literature. In the *Mean of Scores* (MoS) scenario, the scores are calculated for each fold/dataset, and the mean of the scores is determined as the score characterizing the entire experiment. In the *Score of Means* (SoM) approach, first the overall confusion matrix is determined, and then the scores are calculated based on these total figures. The advantage of the MoS approach over SoM is that it is possible to estimate the standard deviation of the scores, however, its disadvantage is that the average of non-linear scores might be distorted and some score might become undefined on when the folds are extremely small (typically in the case of small and imbalanced data).
-We emphasize again, that the tests are designed to detect inconsistencies. If the resulting ``inconsistency`` flag is ``False``, the scores can still be calculated in non-standard ways. However, **if the resulting ``inconsistency`` flag is ``True``, it conclusively indicates that inconsistencies are detected, and the reported scores could not be the outcome of the presumed experiment**.
+The ``mlscorecheck`` package supports both approaches, however, by design, to increase awareness, different functions are provided for the different approaches, usually indicated by the '_mos' or '_som' suffixes in the function names.
-A note on the *Score of Means* and *Mean of Scores* aggregations
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The types of tests
+------------------
-When it comes to the aggregation of scores (either over multiple folds, multiple datasets or both), there are two approaches in the literature. In the *Mean of Scores* (MoS) scenario, the scores are calculated for each fold/dataset, and the mean of the scores is determined as the score characterizing the entire experiment. In the *Score of Means* (SoM) approach, first the overall confusion matrix is determined, and then the scores are calculated based on these total figures. The advantage of the MoS approach over SoM is that it is possible to estimate the standard deviation of the scores, however, its disadvantage is that the average of non-linear scores might be distorted and some score might become undefined on when the folds are extremely small (typically in the case of small and imbalanced data).
+The consistency tests can be grouped to three classes, and it is the problem and the experimental setup determining which internal implementation is applicable:
+
+- Exhaustive enumeration: primarily applied for binary and multiclass classification, when the scores are calculated from one single confusion matrix. The calculations are speeded up by interval computing techniques. These tests support all 20 performance scores of binary classification.
+- Linear programming: when averaging is involved in the calculation of performance scores, due to the non-linearity of most scores, the operation cannot be simplified and the extremely large parameter space prevents exhaustive enumeration. In these scenarios, linear integer programming is exploited. These tests usually support only the accuracy, sensitivity, specificity and balanced accuracy scores.
+- Checking the relation of scores: mainly used for regression, when the domain of the performance scores is continuous, preventing inference from the discrete values.
-The two types of tests
-^^^^^^^^^^^^^^^^^^^^^^
+Binary classification
+=====================
+
+Depending on the experimental setup, the consistency tests developed for binary classification problems support a variety of performance scores: when aggregated performance scores (averages on folds or datasets) are reported, only accuracy (``acc``), sensitivity (``sens``), specificity (``spec``) and balanced accuracy (``bacc``) are supported; when cross-validation is not involved in the experimental setup, the list of supported scores reads as follows (with abbreviations in parentheses):
+
+ * accuracy (``acc``),
+ * sensitivity (``sens``),
+ * specificity (``spec``),
+ * positive predictive value (``ppv``),
+ * negative predictive value (``npv``),
+ * balanced accuracy (``bacc``),
+ * f1(-positive) score (``f1``),
+ * f1-negative score (``f1n``),
+ * f-beta positive (``fbp``),
+ * f-beta negative (``fbn``),
+ * Fowlkes-Mallows index (``fm``),
+ * unified performance measure (``upm``),
+ * geometric mean (``gm``),
+ * markedness (``mk``),
+ * positive likelihood ratio (``lrp``),
+ * negative likelihood ratio (``lrn``),
+ * Matthews correlation coefficient (``mcc``),
+ * bookmaker informedness (``bm``),
+ * prevalence threshold (``pt``),
+ * diagnostic odds ratio (``dor``),
+ * Jaccard index (``ji``),
+ * Cohen's kappa (``kappa``)
-In the context of a single testset or a Score of Means (SoM) aggregation, which results in one confusion matrix, one can systematically iterate through all potential confusion matrices to assess whether any of them can generate the reported scores within the specified numerical uncertainty. To expedite this process, the test leverages interval arithmetic. The test supports the performance scores ``acc``, ``sens``, ``spec``, ``ppv``, ``npv``, ``bacc``, ``f1``, ``f1n``, ``fbp``, ``fbn``, ``fm``, ``upm``, ``gm``, ``mk``, ``lrp``, ``lrn``, ``mcc``, ``bm``, ``pt``, ``dor``, ``ji``, ``kappa``. Note that when the f-beta positive or f-beta negative scores are used, one also needs to specify the ``beta_positive`` or ``beta_negative`` parameters.
+The tests are designed to detect inconsistencies. If the resulting ``inconsistency`` flag is ``False``, the scores can still be calculated in non-standard ways. However, **if the resulting ``inconsistency`` flag is ``True``, it conclusively indicates that inconsistencies are detected, and the reported scores could not be the outcome of the presumed experiment**.
-With a MoS type of aggregation, only the averages of scores over folds or datasets are available. In this case, it is feasible to reconstruct fold-level or dataset-level confusion matrices for the linear scores ``acc``, ``sens``, ``spec`` and ``bacc`` using linear integer programming. These tests formulate a linear integer program based on the reported scores and the experimental setup, and check if the program is feasible to produce the reported values within the estimated numerical uncertainties.
+In the rest of the section, we illustrate some of the test functions, for further details and the full list of supported scenarios, see https://mlscorecheck.readthedocs.io/en/latest/.
1 testset with no k-fold
-^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------
A scenario like this is having one single test set to which classification is applied and the scores are computed from the resulting confusion matrix. For example, given a test image, which is segmented and the scores of the segmentation (as a binary classification of pixels) are calculated and reported.
@@ -258,13 +265,13 @@ In the example below, the scores are artificially generated to be consistent, an
.. code-block:: Python
- from mlscorecheck.check import check_1_testset_no_kfold_scores
+ from mlscorecheck.check.binary import check_1_testset_no_kfold
testset = {'p': 530, 'n': 902}
scores = {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
- result = check_1_testset_no_kfold_scores(testset=testset,
+ result = check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=1e-2)
result['inconsistency']
@@ -278,7 +285,7 @@ If one of the scores is altered, like accuracy is changed to 0.92, the configura
scores = {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
- result = check_1_testset_no_kfold_scores(testset=testset,
+ result = check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=1e-2)
result['inconsistency']
@@ -286,8 +293,8 @@ If one of the scores is altered, like accuracy is changed to 0.92, the configura
As the ``inconsistency`` flag shows, here inconsistencies were identified, there are no such ``tp`` and ``tn`` combinations which would end up with the reported scores. Either the assumption on the properties of the dataset, or the scores are incorrect.
-1 dataset with k-fold, mean-of-scores (MoS)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+1 dataset with k-fold, mean of scores (MoS)/score of means (SoM) aggregation
+----------------------------------------------------------------------------
This scenario is the most common in the literature. A classification technique is executed to each fold in a (repeated) k-fold scenario, the scores are calculated for each fold, and the average of the scores is reported with some numerical uncertainty due to rounding/ceiling/flooring. Because of the averaging, this test supports only the linear scores (``acc``, ``sens``, ``spec``, ``bacc``) which usually are among the most commonly reported scores. The test constructs a linear integer program describing the scenario with the true positive and true negative parameters of all folds and checks its feasibility.
@@ -295,14 +302,14 @@ In the example below, a consistent set of figures is tested:
.. code-block:: Python
- from mlscorecheck.check import check_1_dataset_known_folds_mos_scores
+ from mlscorecheck.check.binary import check_1_dataset_known_folds_mos
dataset = {'p': 126, 'n': 131}
folding = {'folds': [{'p': 52, 'n': 94}, {'p': 74, 'n': 37}]}
scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
- result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
+ result = check_1_dataset_known_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -316,7 +323,7 @@ If one of the scores is adjusted, for example, sensitivity is changed to 0.568,
.. code-block:: Python
scores = {'acc': 0.573, 'sens': 0.568, 'bacc': 0.662}
- result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
+ result = check_1_dataset_known_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -333,7 +340,7 @@ with a true positive and true negative configuration with the specified lower an
scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6, 'bacc': 0.1, 'f1': 0.95}
- result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
+ result = check_1_dataset_known_folds_mos(dataset=dataset,
folding=folding,
fold_score_bounds={'acc': (0.8, 1.0)},
scores=scores,
@@ -344,57 +351,21 @@ with a true positive and true negative configuration with the specified lower an
Note that in this example, although ``f1`` is provided, it is completely ignored as the aggregated tests work only for the four linear scores.
-1 dataset with kfold score-of-means (SoM)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-When the scores are calculated in the Score-of-Means (SoM) manner in a k-fold scenario, it means that the total confusion matrix of all folds is calculated first, and then the score formulas are applied to it. The only difference compared to the "1 testset no kfold" scenario is that the number of repetitions of the k-fold scheme multiples the ``p`` and ``n`` statistics of the dataset, but the actual structure of the folds is irrelevant. The result of the analysis is structured similarly to the "1 testset no kfold" case.
-
-For example, testing a consistent scenario:
-
-.. code-block:: Python
-
- from mlscorecheck.check import check_1_dataset_som_scores
+Similar tests are provided for the SoM aggregation as well, for further details see https://mlscorecheck.readthedocs.io/en/latest/.
- dataset = {'dataset_name': 'common_datasets.monk-2'}
- folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}
-
- scores = {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667,
- 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}
-
- result = check_1_dataset_som_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-3)
- result['inconsistency']
- # False
-
-If one of the scores is adjusted, for example, negative predictive value is changed to 0.754, the configuration becomes inconsistent:
-
-.. code-block:: Python
-
- scores = {'spec': 0.668, 'npv': 0.754, 'ppv': 0.667,
- 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}
-
- result = check_1_dataset_som_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-3)
- result['inconsistency']
- # True
-
-n testsets without k-folding, SoM over the testsets
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+n testsets without k-fold, SoM/MoS aggregation
+----------------------------------------------
In this scenario there are n different testsets, the classifier is evaluated on each testsets, and the scores are aggregated by the SoM aggregation. This scenario is similar to the "1 dataset k-fold SoM" case, except the scores are aggregated over testsets rather than folds. The output of the test is structured similarly as in the "1 dataset k-fold SoM" case. In the following example, a consistent case is tested.
.. code-block:: Python
- from mlscorecheck.check import check_n_testsets_som_no_kfold_scores
+ from mlscorecheck.check.binary import check_n_testsets_som_no_kfold
testsets = [{'p': 405, 'n': 223}, {'p': 3, 'n': 422}, {'p': 109, 'n': 404}]
scores = {'acc': 0.4719, 'npv': 0.6253, 'f1p': 0.3091}
- results = check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ results = check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=0.0001)
results["inconsistency"]
@@ -406,130 +377,23 @@ If one of the scores is slightly adjusted, for example, ``npv`` changed to 0.626
scores['npv'] = 0.6263
- results = check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ results = check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=0.0001)
results["inconsistency"]
# True
-n testsets without k-folding, MoS over the testsets
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-This scenario is analogous to the "n testsets without k-folding, SoM" scenario, except the aggregation over the testsets is carried out with the MoS approach. The output is structured similarly to the output of the "1 dataset k-fold MoS" scenario. In the first example, a feasible scenario is tested.
-
-.. code-block:: Python
+Similar tests are provided for the MoS aggregation as well, for further details see https://mlscorecheck.readthedocs.io/en/latest/.
- from mlscorecheck.check import check_n_testsets_mos_no_kfold_scores
- testsets = [{'p': 349, 'n': 50},
- {'p': 478, 'n': 323},
- {'p': 324, 'n': 83},
- {'p': 123, 'n': 145}]
-
- scores = {'acc': 0.6441, 'sens': 0.6706, 'spec': 0.3796, 'bacc': 0.5251}
- results = check_n_testsets_mos_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=0.0001)
- results["inconsistency"]
- # False
-
-If one of the scores is slightly adjusted, for example, ``sens`` is updated to 0.6756, the configuration becomes infeasible.
-
-.. code-block:: Python
-
- scores['sens'] = 0.6756
-
- results = check_n_testsets_mos_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=0.0001)
- results["inconsistency"]
- # True
-
-n datasets with k-folds, SoM over datasets and SoM over folds
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Again, the scenario is similar to the "1 dataset k-fold SoM" scenario, except there is another level of aggregation over datasets, and one single confusion matrix is determined for the entire experiment and the scores are calculated from that. In this scenario a list of evaluations need to be specified. The output of the test is structured similarly as in the "1 dataset k-fold SoM" case, there is a top level ``inconsistency`` flag indicating if inconsistency has been detected. In the following example, a consistent case is prepared with two datasets.
-
-.. code-block:: Python
-
- from mlscorecheck.check import check_n_datasets_som_kfold_som_scores
-
- evaluation0 = {'dataset': {'p': 389, 'n': 630},
- 'folding': {'n_folds': 5, 'n_repeats': 2,
- 'strategy': 'stratified_sklearn'}}
- evaluation1 = {'dataset': {'dataset_name': 'common_datasets.saheart'},
- 'folding': {'n_folds': 5, 'n_repeats': 2,
- 'strategy': 'stratified_sklearn'}}
- evaluations = [evaluation0, evaluation1]
-
- scores = {'acc': 0.631, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}
-
- result = check_n_datasets_som_kfold_som_scores(scores=scores,
- evaluations=evaluations,
- eps=1e-3)
- result['inconsistency']
- # False
-
-However, if one of the scores is adjusted a little, like accuracy is changed to 0.731, the configuration becomes inconsistent:
-
-.. code-block:: Python
-
- scores = {'acc': 0.731, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}
-
- result = check_n_datasets_som_kfold_som_scores(scores=scores,
- evaluations=evaluations,
- eps=1e-3)
- result['inconsistency']
- # True
-
-n datasets with k-folds, MoS over datasets and SoM over folds
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-This scenario is about performance scores calculated for each dataset individually by the SoM aggregation in any k-folding strategy, and then the scores are aggregated across the datasets in the MoS manner. Because of the overall averaging, one cannot do inference about the non-linear scores, only the four linear scores are supported (``acc``, ``sens``, ``spec``, ``bacc``), and the scores are checked by linear programming. Similarly as before, the specification of a list of evaluations is needed. In the following example a consistent scenario is tested, with score bounds also specified on the datasets:
-
-.. code-block:: Python
-
- from mlscorecheck.check import check_n_datasets_mos_kfold_som_scores
-
- evaluation0 = {'dataset': {'p': 39, 'n': 822},
- 'folding': {'n_folds': 5, 'n_repeats': 3,
- 'strategy': 'stratified_sklearn'}}
- evaluation1 = {'dataset': {'dataset_name': 'common_datasets.winequality-white-3_vs_7'},
- 'folding': {'n_folds': 5, 'n_repeats': 3,
- 'strategy': 'stratified_sklearn'}}
- evaluations = [evaluation0, evaluation1]
-
- scores = {'acc': 0.312, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
-
- result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,
- dataset_score_bounds={'acc': (0.0, 0.5)},
- eps=1e-4,
- scores=scores)
- result['inconsistency']
- # False
-
-However, if one of the scores is adjusted a little (accuracy changed to 0.412 and the score bounds also changed), the configuration becomes infeasible:
-
-.. code-block:: Python
-
- scores = {'acc': 0.412, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
- result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,
- dataset_score_bounds={'acc': (0.5, 1.0)},
- eps=1e-4,
- scores=scores)
- result['inconsistency']
- # True
-
-The output is structured similarly to the '1 dataset k-folds MoS' case, one can query the status of the solver by the key ``lp_status`` and the actual configuration of the variables by the ``lp_configuration`` key. If there are hints on the minimum and maximum scores across the datasets, one can add those bounds through the ``dataset_score_bounds`` parameter to strengthen the test.
-
-n datasets with k-folds, MoS over datasets and MoS over folds
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+n datasets with k-fold, MoS over datasets and MoS over folds
+------------------------------------------------------------
In this scenario, scores are calculated in the MoS manner for each dataset, and then aggregated again across the datasets. Again, because of the averaging, only the four linear scores (``acc``, ``sens``, ``spec``, ``bacc``) are supported. In the following example a consistent scenario is checked with three datasets and without score bounds specified at any level:
.. code-block:: Python
- from mlscorecheck.check import check_n_datasets_mos_known_folds_mos_scores
+ from mlscorecheck.check.binary import check_n_datasets_mos_known_folds_mos
evaluation0 = {'dataset': {'p': 118, 'n': 95},
'folding': {'folds': [{'p': 22, 'n': 23}, {'p': 96, 'n': 72}]}}
@@ -539,7 +403,7 @@ In this scenario, scores are calculated in the MoS manner for each dataset, and
scores = {'acc': 0.61, 'sens': 0.709, 'spec': 0.461, 'bacc': 0.585}
- result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,
+ result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
result['inconsistency']
@@ -551,7 +415,7 @@ Again, the details of the analysis are accessible under the ``lp_status`` and ``
scores = {'acc': 0.71, 'sens': 0.709, 'spec': 0.461}
- result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,
+ result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
result['inconsistency']
@@ -559,10 +423,7 @@ Again, the details of the analysis are accessible under the ``lp_status`` and ``
If there are hints on the minimum and maximum scores across the datasets, one can add those bounds through the ``dataset_score_bounds`` parameter to strengthen the test.
-Not knowing the mode of aggregation
------------------------------------
-
-The biggest challenge with aggregated scores is that the ways of aggregation at the dataset and experiment level are rarely disclosed explicitly. Even in this case the tools presented in the previous section can be used since there are hardly any further ways of meaningful averaging than (MoS on folds, MoS on datasets), (SoM on folds, MoS on datasets), (SoM on folds, SoM on datasets), hence, if a certain set of scores is inconsistent with each of these possibilities, one can safely say that the results do not satisfy the reasonable expectations.
+Similar tests are provided for the SoM aggregation and the mixing of MoS and SoM aggregation, as well, for further details see https://mlscorecheck.readthedocs.io/en/latest/.
Not knowing the k-folding scheme
--------------------------------
@@ -573,14 +434,14 @@ Given a dataset and knowing that k-fold cross-validation was applied with MoS ag
.. code-block:: Python
- from mlscorecheck.check import check_1_dataset_unknown_folds_mos_scores
+ from mlscorecheck.check.binary import check_1_dataset_unknown_folds_mos
dataset = {'p': 126, 'n': 131}
folding = {'n_folds': 2, 'n_repeats': 1}
scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
- result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,
+ result = check_1_dataset_unknown_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -593,7 +454,7 @@ If the balanced accuracy score is adjusted to 0.862, the configuration becomes i
scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.862}
- result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,
+ result = check_1_dataset_unknown_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -606,7 +467,7 @@ The following scenario is similar in the sense that MoS aggregation is applied t
.. code-block:: Python
- from mlscorecheck.check import check_n_datasets_mos_unknown_folds_mos_scores
+ from mlscorecheck.check.binary import check_n_datasets_mos_unknown_folds_mos
evaluation0 = {'dataset': {'p': 13, 'n': 73},
'folding': {'n_folds': 4, 'n_repeats': 1}}
@@ -616,7 +477,7 @@ The following scenario is similar in the sense that MoS aggregation is applied t
scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.343}
- result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,
+ result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
result['inconsistency']
@@ -628,12 +489,134 @@ The setup is consistent. However, if the balanced accuracy is changed to 0.9, th
scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.9}
- result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,
+ result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
result['inconsistency']
# True
+Not knowing the mode of aggregation
+-----------------------------------
+
+The biggest challenge with aggregated scores is that the ways of aggregation at the dataset and experiment level are rarely disclosed explicitly. Even in this case the tools presented in the previous section can be used since there are hardly any further ways of meaningful averaging than (MoS on folds, MoS on datasets), (SoM on folds, MoS on datasets), (SoM on folds, SoM on datasets), hence, if a certain set of scores is inconsistent with each of these possibilities, one can safely say that the results do not satisfy the reasonable expectations.
+
+Multiclass classification
+=========================
+
+In multiclass classification scenarios single testsets and k-fold cross-validation on a single dataset are supported with both the micro-averaging and macro-averaging aggregation strategies. The list of supported scores depends on the experimental setup, when applicable, all 20 scores listed for binary classification are supported, when the test operates in terms of linear programming, only accuracy, sensitivity, specificity and balanced accuracy are supported.
+
+A note on micro and macro-averaging
+-----------------------------------
+
+In a multiclass scenario, the commonly used approach is for measuring the performance of a classification technique is to calculate the micro or macro-averaged scores. In the micro-averaging approach, the confusion matrices of the individual classes are aggregated, and the scores are calculated from the aggregated confusion matrix. In the macro-averaging approach, the scores are calculated for each class, and the average of the scores is reported. The micro-averaging approach is more robust to class imbalance, however, it is not possible to estimate the standard deviation of the scores. The macro-averaging approach is more sensitive to class imbalance, but it is possible to estimate the standard deviation of the scores.
+
+1 testset, no k-fold, micro/macro-averaging
+-------------------------------------------
+
+In this scenario, we suppose there is a multiclass classification testset and the class level scores on the testset are aggregated by micro-averaging. The test is based on exhaustive enumeration, so all 20 performance scores are supported. In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ from mlscorecheck.check.multiclass import check_1_testset_no_kfold_micro
+
+ testset = {0: 10, 1: 100, 2: 80}
+ scores = {'acc': 0.5158, 'sens': 0.2737, 'spec': 0.6368,
+ 'bacc': 0.4553, 'ppv': 0.2737, 'npv': 0.6368}
+ results = check_1_testset_no_kfold_micro(testset=testset,
+ scores=scores,
+ eps=1e-4)
+ results['inconsistency']
+ # False
+
+As the test confirms, the setup is consistent. However, if one of the scores is adjusted a little, for example, accuracy is changed to 0.5258, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ scores['acc'] = 0.5258
+ results = check_1_testset_no_kfold_micro(testset=testset,
+ scores=scores,
+ eps=1e-4)
+ results['inconsistency']
+ # True
+
+Similar functionality is provided for macro-averaging, for further details see https://mlscorecheck.readthedocs.io/en/latest/.
+
+1 dataset, known k-folds, SoM/MoS aggregation, micro/macro-averaging
+--------------------------------------------------------------------
+
+In this scenario, we assume there is a multiclass classification dataset, which is evaluated in a k-fold cross-validation scenario, the class level scores are calculated by micro-averaging, and the fold level results are aggregated in the score of means fashion. The test is based on exhaustive enumeration, therefore, all 20 performance scores are supported.
+
+In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_micro
+
+ dataset = {0: 86, 1: 96, 2: 59, 3: 105}
+ folding = {'folds': [{0: 43, 1: 48, 2: 30, 3: 52}, {0: 43, 1: 48, 2: 29, 3: 53}]}
+ scores = {'acc': 0.6272, 'sens': 0.2543, 'spec': 0.7514, 'f1p': 0.2543}
+
+ result = check_1_dataset_known_folds_som_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ result['inconsistency']
+ # False
+
+As the test confirms, the scenario is feasible. However, if one of the scores is adjusted a little, for example, sensitivity is changed to 0.2553, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ scores['sens'] = 0.2553
+ result = check_1_dataset_known_folds_som_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ result['inconsistency']
+ # True
+
+Similar functionality is provided for mean of scores aggregation and macro averaging, for further details see https://mlscorecheck.readthedocs.io/en/latest/.
+
+Regression
+==========
+
+From the point of view of consistency testing, regression is the hardest problem as the predictions can produce any performance scores. The tests implemented in the package allow testing the relation of the *mean squared error* (``mse``), *root mean squared error* (``rmse``), *mean average error* (``mae``) and *r^2 scores* (``r2``).
+
+1 testset, no k-fold
+--------------------
+
+In this scenario, we assume there is a regression testset, and the performance scores are calculated on the testset.
+
+In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ from mlscorecheck.check.regression import check_1_testset_no_kfold
+
+ var = 0.0831619 # the variance of the target values in the testset
+ n_samples = 100
+ scores = {'mae': 0.0254, 'r2': 0.9897}
+
+ result = check_1_testset_no_kfold(var=var,
+ n_samples=n_samples,
+ scores=scores,
+ eps=1e-4)
+ result['inconsistency']
+ # False
+
+As the results show, there is no inconsistency detected. However, if the mae score is adjusted slightly to 0.03, the configuration becomes inconsistent:
+
+.. code-block:: Python
+
+ scores['mae'] = 0.03
+ result = check_1_testset_no_kfold(var=var,
+ n_samples=n_samples,
+ scores=scores,
+ eps=1e-4)
+ result['inconsistency']
+ # True
+
+
Test bundles
============
@@ -660,7 +643,7 @@ Contribution
We kindly encourage any experts to provide further, field specific dataset and experiment specifications and test bundles to facilitate the reporting of clean and reproducible results in any field related to binary classification!
References
-**********
+==========
.. [RV] Kovács, G. and Fazekas, A.: "A new baseline for retinal vessel segmentation: Numerical identification and correction of methodological inconsistencies affecting 100+ papers", Medical Image Analysis, 2022(1), pp. 102300
diff --git a/docs/00_introduction.rst b/docs/00_introduction.rst
index a3e0e25..cd8f024 100644
--- a/docs/00_introduction.rst
+++ b/docs/00_introduction.rst
@@ -2,23 +2,35 @@
:file: ga4.html
Introduction
-=============
+============
The purpose
-----------
-Performance scores for binary classification are reported on a dataset and look suspicious (exceptionally high scores possibly due to typo, uncommon evaluation methodology, data leakage in preparation, incorrect use of statistics, etc.). With the tools implemented in the package ``mlscorecheck``, one can test if the reported performance scores are consistent with each other and the assumptions on the experimental setup up to the numerical uncertainty due to rounding/truncation/ceiling.
+Performance scores of a machine learning technique (binary/multiclass classification, regression) are reported on a dataset and look suspicious (exceptionally high scores possibly due to a typo, uncommon evaluation methodology, data leakage in preparation, incorrect use of statistics, etc.). With the tools implemented in the package ``mlscorecheck``, one can test if the reported performance scores are consistent with each other and the assumptions on the experimental setup up.
+
+Testing is as simple as the following example illustrated. Suppose the accuracy, sensitivity and specificity scores are reported for a binary classification testset consisting of p=100 and n=200 samples. All this information is supplied to the suitable test function and the result shows that that inconsistencies were identified: the scores could not be calculated from the confusion matrix of the testset:
+
+.. code-block:: Python
+
+ from mlscorecheck.check.binary import check_1_testset_no_kfold
+
+ result = check_1_testset_no_kfold(testset={'p': 100, 'n': 200},
+ scores={'acc': 0.9567, 'sens': 0.8545, 'spec': 0.9734},
+ eps=1e-4)
+ result['inconsistency']
+ # True
-The consistency tests are numerical and **not** statistical: if inconsistencies are identified, it means that either the assumptions on the evaluation protocol or the reported scores are incorrect.
+The consistency tests are numerical and **not** statistical: if inconsistencies are identified, it means that either the assumptions on the experimental setup or the reported scores are incorrect.
In more detail
--------------
-Binary classification is one of the most fundamental tasks in machine learning. The evaluation of the performance of binary classification techniques, whether for original theoretical advancements or applications in specific fields, relies heavily on performance scores (https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers). Although reported performance scores are employed as primary indicators of research value, they often suffer from methodological problems, typos, and insufficient descriptions of experimental settings. These issues contribute to the replication crisis (https://en.wikipedia.org/wiki/Replication_crisis) and ultimately entire fields of research ([RV]_, [EHG]_). Even systematic reviews can suffer from using incomparable performance scores for ranking research papers [RV]_.
+The evaluation of the performance of machine learning techniques, whether for original theoretical advancements or applications in specific fields, relies heavily on performance scores (https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers). Although reported performance scores are employed as primary indicators of research value, they often suffer from methodological problems, typos, and insufficient descriptions of experimental settings. These issues contribute to the replication crisis (https://en.wikipedia.org/wiki/Replication_crisis) and ultimately entire fields of research ([RV]_, [EHG]_). Even systematic reviews can suffer from using incomparable performance scores for ranking research papers [RV]_.
-The majority of performance scores are calculated from the binary confusion matrix, or multiple confusion matrices aggregated across folds and/or datasets. For many commonly used experimental setups one can develop numerical techniques to test if there exists any confusion matrix (or matrices), compatible with the experiment and leading to the reported performance scores. This package implements such consistency tests for some common scenarios. We highlight that the developed tests cannot guarantee that the scores are surely calculated by some standards or a presumed evaluation protocol. However, *if the tests fail and inconsistencies are detected, it means that the scores are not calculated by the presumed protocols with certainty*. In this sense, the specificity of the test is 1.0, the inconsistencies being detected are inevitable.
+In practice, the performance scores cannot take any values independently, the scores reported for the same experiment are constrained by the experimental setup and need to express some internal consistency. For many commonly used experimental setups it is possible to develop numerical techniques to test if the scores could be the outcome of the presumed experiment on the presumed dataset. This package implements such consistency tests for some common experimental setups. We highlight that the developed tests cannot guarantee that the scores are surely calculated by some standards or a presumed evaluation protocol. However, *if the tests fail and inconsistencies are detected, it means that the scores are not calculated by the presumed protocols with certainty*. In this sense, the specificity of the test is 1.0, the inconsistencies being detected are inevitable.
-For further information, see the preprint:
+For further information, see the preprint: https://arxiv.org/abs/2310.12527
Citation
========
@@ -28,16 +40,18 @@ If you use the package, please consider citing the following paper:
.. code-block:: BibTex
@article{mlscorecheck,
- author={Gy\"orgy Kov\'acs and Attila Fazekas},
- title={Checking the internal consistency of reported performance scores in binary classification},
- year={2023}
+ author={Attila Fazekas and Gy\"orgy Kov\'acs},
+ title={Testing the Consistency of Performance Scores Reported for Binary Classification Problems},
+ year={2023}
}
Latest news
===========
-* the 0.1.0 version of the package is released
-* the paper describing the implemented techniques is available as a preprint at: https://arxiv.org/abs/2310.12527
+* the 1.0.1 version of the package is released;
+* the paper describing the numerical techniques is available as a preprint at: https://arxiv.org/abs/2310.12527
+* 10 test bundles including retina image processing datasets, preterm delivery prediction from electrohysterograms and skin lesion classification has been added;
+* multiclass and regression tests added.
Installation
============
@@ -45,16 +59,17 @@ Installation
Requirements
------------
-The package has only basic requirements when used for consistency testing.
+The package has only basic requirements when used for consistency testing:
* ``numpy``
* ``pulp``
+* ``scikit-learn``
.. code-block:: bash
> pip install numpy pulp
-In order to execute the tests, one also needs ``scikit-learn``, in order to test the computer algebra components or reproduce the algebraic solutions, either ``sympy`` or ``sage`` needs to be installed. The installation of ``sympy`` can be done in the usual way. To install ``sage`` in a ``conda`` environment, one needs to add the ``conda-forge`` channel first:
+In order to execute the unit tests for the computer algebra components or reproduce the algebraic solutions, either ``sympy`` or ``sage`` needs to be installed. The installation of ``sympy`` can be done in the usual way. To install ``sage`` in a ``conda`` environment, one needs to add the ``conda-forge`` channel first:
.. code-block:: bash
diff --git a/docs/01a_requirements.rst b/docs/01a_requirements.rst
index ca9e518..fd18813 100644
--- a/docs/01a_requirements.rst
+++ b/docs/01a_requirements.rst
@@ -1,32 +1,98 @@
+Preliminaries
+=============
+
Requirements
-************
+------------
In general, there are three inputs to the consistency testing functions:
* **the specification of the experiment**;
-* **the collection of available (reported) performance scores**: when aggregated performance scores (averages on folds or datasets) are reported, only accuracy (``acc``), sensitivity (``sens``), specificity (``spec``) and balanced accuracy (``bacc``) are supported; when cross-validation is not involved in the experimental setup, the list of supported scores reads as follows (with abbreviations in parentheses):
-
- * accuracy (``acc``),
- * sensitivity (``sens``),
- * specificity (``spec``),
- * positive predictive value (``ppv``),
- * negative predictive value (``npv``),
- * balanced accuracy (``bacc``),
- * f1(-positive) score (``f1``),
- * f1-negative score (``f1n``),
- * f-beta positive (``fbp``),
- * f-beta negative (``fbn``),
- * Fowlkes-Mallows index (``fm``),
- * unified performance measure (``upm``),
- * geometric mean (``gm``),
- * markedness (``mk``),
- * positive likelihood ratio (``lrp``),
- * negative likelihood ratio (``lrn``),
- * Matthews correlation coefficient (``mcc``),
- * bookmaker informedness (``bm``),
- * prevalence threshold (``pt``),
- * diagnostic odds ratio (``dor``),
- * Jaccard index (``ji``),
- * Cohen's kappa (``kappa``);
-
+* **the collection of available (reported) performance scores**;
* **the estimated numerical uncertainty**: the performance scores are usually shared with some finite precision, being rounded/ceiled/floored to ``k`` decimal places. The numerical uncertainty estimates the maximum difference of the reported score and its true value. For example, having the accuracy score 0.9489 published (4 decimal places), one can suppose that it is rounded, therefore, the numerical uncertainty is 0.00005 (10^(-4)/2). To be more conservative, one can assume that the score was ceiled or floored. In this case, the numerical uncertainty becomes 0.0001 (10^(-4)).
+
+Specification of the experimental setup
+---------------------------------------
+
+In this subsection, we illustrate the various ways the experimental setup can be specified.
+
+Specification of one testset or dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are multiple ways to specify datasets and entire experiments consisting of multiple datasets evaluated in differing ways of cross-validations.
+
+A simple binary classification testset consisting of ``p`` positive samples (usually labelled 1) and ``n`` negative samples (usually labelled 0) can be specified as
+
+.. code-block:: Python
+
+ testset = {"p": 10, "n": 20}
+
+One can also specify a commonly used dataset by its name and the package will look up the ``p`` and ``n`` counts of the datasets from its internal registry (based on the representations in the ``common-datasets`` package):
+
+.. code-block:: Python
+
+ dataset = {"dataset_name": "common_datasets.ADA"}
+
+To see the list of supported datasets and corresponding counts, issue
+
+.. code-block:: Python
+
+ from mlscorecheck.experiments import dataset_statistics
+ print(dataset_statistics)
+
+Specification of a folding
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The specification of foldings is needed when the scores are computed in cross-validation scenarios. We distinguish two main cases: in the first case, the number of positive and negative samples in the folds are known, or can be derived from the attributes of the dataset (for example, by stratification); in the second case, the statistics of the folds are not known, but the number of folds and potential repetitions are known.
+
+In the first case, when the folds are known, one can specify them by listing them:
+
+.. code-block:: Python
+
+ folding = {"folds": [{"p": 5, "n": 10},
+ {"p": 4, "n": 10},
+ {"p": 5, "n": 10}]}
+
+This folding can represent the evaluation of a dataset with 14 positive and 30 negative samples in a 3-fold stratified cross-validation scenario.
+
+Knowing that the folding is derived by some standard stratification techniques, one can just specify the parameters of the folding:
+
+.. code-block:: Python
+
+ folding = {"n_folds": 3, "n_repeats": 1, "strategy": "stratified_sklearn"}
+
+In this specification, it is assumed that the samples are distributed into the folds according to the ``sklearn`` stratification implementation.
+
+Finally, if neither the folds nor the folding strategy is known, one can simply specify the folding with its parameters (assuming a repeated k-fold scheme):
+
+.. code-block:: Python
+
+ folding = {"n_folds": 3, "n_repeats": 2}
+
+Note that not all consistency testing functions support the latter case (not knowing the exact structure of the folds).
+
+Specification of an evaluation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A dataset and a folding constitute an *evaluation*, and many of the test functions take evaluations as parameters describing the scenario:
+
+.. code-block:: Python
+
+ evaluation = {"dataset": {"p": 10, "n": 50},
+ "folding": {"n_folds": 5, "n_repeats": 1,
+ "strategy": "stratified_sklearn"}}
+
+A note on the *Score of Means* and *Mean of Scores* aggregations
+----------------------------------------------------------------
+
+When it comes to the aggregation of scores (either over multiple folds, multiple datasets or both), there are two approaches in the literature. In the *Mean of Scores* (MoS) scenario, the scores are calculated for each fold/dataset, and the mean of the scores is determined as the score characterizing the entire experiment. In the *Score of Means* (SoM) approach, first the overall micro-figures (e.g. the overall confusion matrix in classification, the overall squared error in regression) are determined, and then the scores are calculated based on these total figures. The advantage of the MoS approach over SoM is that it is possible to estimate the standard deviation of the scores, however, its disadvantage is that the average of non-linear scores might be distorted and some score might become undefined on when the folds are extremely small (typically in the case of small and imbalanced data).
+
+The ``mlscorecheck`` package supports both approaches, however, by design, to increase awareness, different functions are provided for the different approaches, usually indicated by the '_mos' or '_som' suffixes in the function names.
+
+The types of tests
+------------------
+
+The consistency tests can be grouped to three classes, and it is the problem and the experimental setup determining which internal implementation is applicable:
+
+- Exhaustive enumeration: primarily applied for binary and multiclass classification, when the scores are calculated from one single confusion matrix. The calculations are speeded up by interval computing techniques. These tests support all 20 performance scores of binary classification.
+- Linear programming: when averaging is involved in the calculation of performance scores, due to the non-linearity of most scores, the operation cannot be simplified and the extremely large parameter space prevents exhaustive enumeration. In these scenarios, linear integer programming is exploited. These tests usually support only the accuracy, sensitivity, specificity and balanced accuracy scores.
+- Checking the relation of scores: mainly used for regression, when the domain of the performance scores is continuous, preventing inference from the discrete values.
diff --git a/docs/01b_specifying_setup.rst b/docs/01b_specifying_setup.rst
deleted file mode 100644
index a107d8a..0000000
--- a/docs/01b_specifying_setup.rst
+++ /dev/null
@@ -1,70 +0,0 @@
-Specification of the experimental setup
----------------------------------------
-
-In this subsection, we illustrate the various ways the experimental setup can be specified.
-
-Specification of one testset or dataset
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-There are multiple ways to specify datasets and entire experiments consisting of multiple datasets evaluated in differing ways of cross-validations.
-
-A simple binary classification testset consisting of ``p`` positive samples (usually labelled 1) and ``n`` negative samples (usually labelled 0) can be specified as
-
-.. code-block:: Python
-
- testset = {"p": 10, "n": 20}
-
-One can also specify a commonly used dataset by its name and the package will look up the ``p`` and ``n`` counts of the datasets from its internal registry (based on the representations in the ``common-datasets`` package):
-
-.. code-block:: Python
-
- dataset = {"dataset_name": "common_datasets.ADA"}
-
-To see the list of supported datasets and corresponding counts, issue
-
-.. code-block:: Python
-
- from mlscorecheck.experiments import dataset_statistics
- print(dataset_statistics)
-
-Specification of a folding
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The specification of foldings is needed when the scores are computed in cross-validation scenarios. We distinguish two main cases: in the first case, the number of positive and negative samples in the folds are known, or can be derived from the attributes of the dataset (for example, by stratification); in the second case, the statistics of the folds are not known, but the number of folds and potential repetitions are known.
-
-In the first case, when the folds are known, one can specify them by listing them:
-
-.. code-block:: Python
-
- folding = {"folds": [{"p": 5, "n": 10},
- {"p": 4, "n": 10},
- {"p": 5, "n": 10}]}
-
-This folding can represent the evaluation of a dataset with 14 positive and 30 negative samples in a 3-fold stratified cross-validation scenario.
-
-Knowing that the folding is derived by some standard stratification techniques, one can just specify the parameters of the folding:
-
-.. code-block:: Python
-
- folding = {"n_folds": 3, "n_repeats": 1, "strategy": "stratified_sklearn"}
-
-In this specification, it is assumed that the samples are distributed into the folds according to the ``sklearn`` stratification implementation.
-
-Finally, if neither the folds nor the folding strategy is known, one can simply specify the folding with its parameters (assuming a repeated k-fold scheme):
-
-.. code-block:: Python
-
- folding = {"n_folds": 3, "n_repeats": 2}
-
-Note that not all consistency testing functions support the latter case (not knowing the exact structure of the folds).
-
-Specification of an evaluation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A dataset and a folding constitute an *evaluation*, and many of the test functions take evaluations as parameters describing the scenario:
-
-.. code-block:: Python
-
- evaluation = {"dataset": {"p": 10, "n": 50},
- "folding": {"n_folds": 5, "n_repeats": 1,
- "strategy": "stratified_sklearn"}}
diff --git a/docs/01c_consistency_checking.rst b/docs/01c_consistency_checking.rst
index d04ad88..33ccb9a 100644
--- a/docs/01c_consistency_checking.rst
+++ b/docs/01c_consistency_checking.rst
@@ -1,21 +1,32 @@
-Testing the consistency of performance scores
----------------------------------------------
-
-Numerous experimental setups are supported by the package. In this section we go through them one by one giving some examples of possible use cases.
-
-We emphasize again, that the tests are designed to detect inconsistencies. If the resulting ``inconsistency`` flag is ``False``, the scores can still be calculated in non-standard ways. However, **if the resulting ``inconsistency`` flag is ``True``, it conclusively indicates that inconsistencies are detected, and the reported scores could not be the outcome of the presumed experiment**.
-
-A note on the *Score of Means* and *Mean of Scores* aggregations
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-When it comes to the aggregation of scores (either over multiple folds, multiple datasets or both), there are two approaches in the literature. In the *Mean of Scores* (MoS) scenario, the scores are calculated for each fold/dataset, and the mean of the scores is determined as the score characterizing the entire experiment. In the *Score of Means* (SoM) approach, first the overall confusion matrix is determined, and then the scores are calculated based on these total figures. The advantage of the MoS approach over SoM is that it is possible to estimate the standard deviation of the scores, however, its disadvantage is that the average of non-linear scores might be distorted and some score might become undefined on when the folds are extremely small (typically in the case of small and imbalanced data).
-
-The two types of tests
-^^^^^^^^^^^^^^^^^^^^^^
-
-In the context of a single testset or a Score of Means (SoM) aggregation, which results in one confusion matrix, one can systematically iterate through all potential confusion matrices to assess whether any of them can generate the reported scores within the specified numerical uncertainty. To expedite this process, the test leverages interval arithmetic. The test supports the performance scores ``acc``, ``sens``, ``spec``, ``ppv``, ``npv``, ``bacc``, ``f1``, ``f1n``, ``fbp``, ``fbn``, ``fm``, ``upm``, ``gm``, ``mk``, ``lrp``, ``lrn``, ``mcc``, ``bm``, ``pt``, ``dor``, ``ji``, ``kappa``. Note that when the f-beta positive or f-beta negative scores are used, one also needs to specify the ``beta_positive`` or ``beta_negative`` parameters.
-
-With a MoS type of aggregation, only the averages of scores over folds or datasets are available. In this case, it is feasible to reconstruct fold-level or dataset-level confusion matrices for the linear scores ``acc``, ``sens``, ``spec`` and ``bacc`` using linear integer programming. These tests formulate a linear integer program based on the reported scores and the experimental setup, and check if the program is feasible to produce the reported values within the estimated numerical uncertainties.
+Binary classification
+=====================
+
+Depending on the experimental setup, the consistency tests developed for binary classification problems support a variety of performance scores: when aggregated performance scores (averages on folds or datasets) are reported, only accuracy (``acc``), sensitivity (``sens``), specificity (``spec``) and balanced accuracy (``bacc``) are supported; when cross-validation is not involved in the experimental setup, the list of supported scores reads as follows (with abbreviations in parentheses):
+
+ * accuracy (``acc``),
+ * sensitivity (``sens``),
+ * specificity (``spec``),
+ * positive predictive value (``ppv``),
+ * negative predictive value (``npv``),
+ * balanced accuracy (``bacc``),
+ * f1(-positive) score (``f1``),
+ * f1-negative score (``f1n``),
+ * f-beta positive (``fbp``),
+ * f-beta negative (``fbn``),
+ * Fowlkes-Mallows index (``fm``),
+ * unified performance measure (``upm``),
+ * geometric mean (``gm``),
+ * markedness (``mk``),
+ * positive likelihood ratio (``lrp``),
+ * negative likelihood ratio (``lrn``),
+ * Matthews correlation coefficient (``mcc``),
+ * bookmaker informedness (``bm``),
+ * prevalence threshold (``pt``),
+ * diagnostic odds ratio (``dor``),
+ * Jaccard index (``ji``),
+ * Cohen's kappa (``kappa``)
+
+The tests are designed to detect inconsistencies. If the resulting ``inconsistency`` flag is ``False``, the scores can still be calculated in non-standard ways. However, **if the resulting ``inconsistency`` flag is ``True``, it conclusively indicates that inconsistencies are detected, and the reported scores could not be the outcome of the presumed experiment**.
1 testset with no k-fold
^^^^^^^^^^^^^^^^^^^^^^^^
@@ -26,13 +37,13 @@ In the example below, the scores are artificially generated to be consistent, an
.. code-block:: Python
- >>> from mlscorecheck.check import check_1_testset_no_kfold_scores
+ >>> from mlscorecheck.check.binary import check_1_testset_no_kfold
>>> testset = {'p': 530, 'n': 902}
>>> scores = {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
- >>> result = check_1_testset_no_kfold_scores(testset=testset,
+ >>> result = check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=1e-2)
>>> result['inconsistency']
@@ -46,7 +57,7 @@ If one of the scores is altered, like accuracy is changed to 0.92, the configura
>>> scores = {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
- >>> result = check_1_testset_no_kfold_scores(testset=testset,
+ >>> result = check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=1e-2)
>>> result['inconsistency']
@@ -63,14 +74,14 @@ In the example below, a consistent set of figures is tested:
.. code-block:: Python
- >>> from mlscorecheck.check import check_1_dataset_known_folds_mos_scores
+ >>> from mlscorecheck.check.binary import check_1_dataset_known_folds_mos
>>> dataset = {'p': 126, 'n': 131}
>>> folding = {'folds': [{'p': 52, 'n': 94}, {'p': 74, 'n': 37}]}
>>> scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
- >>> result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
+ >>> result = check_1_dataset_known_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -84,7 +95,7 @@ If one of the scores is adjusted, for example, sensitivity is changed to 0.568,
.. code-block:: Python
>>> scores = {'acc': 0.573, 'sens': 0.568, 'bacc': 0.662}
- >>> result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
+ >>> result = check_1_dataset_known_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -101,7 +112,7 @@ with a true positive and true negative configuration with the specified lower an
>>> scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6, 'bacc': 0.1, 'f1': 0.95}
- >>> result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
+ >>> result = check_1_dataset_known_folds_mos(dataset=dataset,
folding=folding,
fold_score_bounds={'acc': (0.8, 1.0)},
scores=scores,
@@ -121,7 +132,7 @@ For example, testing a consistent scenario:
.. code-block:: Python
- >>> from mlscorecheck.check import check_1_dataset_som_scores
+ >>> from mlscorecheck.check.binary import check_1_dataset_kfold_som
>>> dataset = {'dataset_name': 'common_datasets.monk-2'}
>>> folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}
@@ -129,7 +140,7 @@ For example, testing a consistent scenario:
>>> scores = {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667,
'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}
- >>> result = check_1_dataset_som_scores(dataset=dataset,
+ >>> result = check_1_dataset_kfold_som(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -143,7 +154,7 @@ If one of the scores is adjusted, for example, negative predictive value is chan
>>> {'spec': 0.668, 'npv': 0.754, 'ppv': 0.667,
'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}
- >>> result = check_1_dataset_som_scores(dataset=dataset,
+ >>> result = check_1_dataset_kfold_som(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -157,12 +168,12 @@ In this scenario there are n different testsets, the classifier is evaluated on
.. code-block:: Python
- from mlscorecheck.check import check_n_testsets_som_no_kfold_scores
+ from mlscorecheck.check.binary import check_n_testsets_som_no_kfold
testsets = [{'p': 405, 'n': 223}, {'p': 3, 'n': 422}, {'p': 109, 'n': 404}]
scores = {'acc': 0.4719, 'npv': 0.6253, 'f1p': 0.3091}
- results = check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ results = check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=0.0001)
results["inconsistency"]
@@ -174,7 +185,7 @@ If one of the scores is slightly adjusted, for example, ``npv`` changed to 0.626
scores['npv'] = 0.6263
- results = check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ results = check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=0.0001)
results["inconsistency"]
@@ -187,7 +198,7 @@ This scenario is analogous to the "n testsets without k-folding, SoM" scenario,
.. code-block:: Python
- from mlscorecheck.check import check_n_testsets_mos_no_kfold_scores
+ from mlscorecheck.check.binary import check_n_testsets_mos_no_kfold
testsets = [{'p': 349, 'n': 50},
{'p': 478, 'n': 323},
@@ -195,7 +206,7 @@ This scenario is analogous to the "n testsets without k-folding, SoM" scenario,
{'p': 123, 'n': 145}]
scores = {'acc': 0.6441, 'sens': 0.6706, 'spec': 0.3796, 'bacc': 0.5251}
- results = check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ results = check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=0.0001)
results["inconsistency"]
@@ -207,7 +218,7 @@ If one of the scores is slightly adjusted, for example, ``sens`` is updated to 0
scores['sens'] = 0.6756
- results = check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ results = check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=0.0001)
results["inconsistency"]
@@ -220,7 +231,7 @@ Again, the scenario is similar to the "1 dataset k-fold SoM" scenario, except th
.. code-block:: Python
- >>> from mlscorecheck.check import check_n_datasets_som_kfold_som_scores
+ >>> from mlscorecheck.check.binary import check_n_datasets_som_kfold_som
>>> evaluation0 = {'dataset': {'p': 389, 'n': 630},
'folding': {'n_folds': 5, 'n_repeats': 2,
@@ -232,7 +243,7 @@ Again, the scenario is similar to the "1 dataset k-fold SoM" scenario, except th
>>> scores = {'acc': 0.631, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}
- >>> result = check_n_datasets_som_kfold_som_scores(scores=scores,
+ >>> result = check_n_datasets_som_kfold_som(scores=scores,
evaluations=evaluations,
eps=1e-3)
>>> result['inconsistency']
@@ -244,7 +255,7 @@ However, if one of the scores is adjusted a little, like accuracy is changed to
>>> scores = {'acc': 0.731, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}
- >>> result = check_n_datasets_som_kfold_som_scores(scores=scores,
+ >>> result = check_n_datasets_som_kfold_som(scores=scores,
evaluations=evaluations,
eps=1e-3)
>>> result['inconsistency']
@@ -257,7 +268,7 @@ This scenario is about performance scores calculated for each dataset individual
.. code-block:: Python
- >>> from mlscorecheck.check import check_n_datasets_mos_kfold_som_scores
+ >>> from mlscorecheck.check.binary import check_n_datasets_mos_kfold_som
>>> evaluation0 = {'dataset': {'p': 39, 'n': 822},
'folding': {'n_folds': 5, 'n_repeats': 3,
@@ -269,7 +280,7 @@ This scenario is about performance scores calculated for each dataset individual
>>> scores = {'acc': 0.312, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
- >>> result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,
+ >>> result = check_n_datasets_mos_kfold_som(evaluations=evaluations,
dataset_score_bounds={'acc': (0.0, 0.5)},
eps=1e-4,
scores=scores)
@@ -281,7 +292,7 @@ However, if one of the scores is adjusted a little (accuracy changed to 0.412 an
.. code-block:: Python
>>> scores = {'acc': 0.412, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
- >>> result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,
+ >>> result = check_n_datasets_mos_kfold_som(evaluations=evaluations,
dataset_score_bounds={'acc': (0.5, 1.0)},
eps=1e-4,
scores=scores)
@@ -297,7 +308,7 @@ In this scenario, scores are calculated in the MoS manner for each dataset, and
.. code-block:: Python
- >>> from mlscorecheck.check import check_n_datasets_mos_known_folds_mos_scores
+ >>> from mlscorecheck.check.binary import check_n_datasets_mos_known_folds_mos
>>> evaluation0 = {'dataset': {'p': 118, 'n': 95},
'folding': {'folds': [{'p': 22, 'n': 23}, {'p': 96, 'n': 72}]}}
@@ -307,7 +318,7 @@ In this scenario, scores are calculated in the MoS manner for each dataset, and
>>> scores = {'acc': 0.61, 'sens': 0.709, 'spec': 0.461, 'bacc': 0.585}
- >>> result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,
+ >>> result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
>>> result['inconsistency']
@@ -319,7 +330,7 @@ Again, the details of the analysis are accessible under the ``lp_status`` and ``
>>> scores = {'acc': 0.71, 'sens': 0.709, 'spec': 0.461}
- >>> result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,
+ >>> result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
>>> result['inconsistency']
@@ -341,14 +352,14 @@ Given a dataset and knowing that k-fold cross-validation was applied with MoS ag
.. code-block:: Python
- >>> from mlscorecheck.check import check_1_dataset_unknown_folds_mos_scores
+ >>> from mlscorecheck.check.binary import check_1_dataset_unknown_folds_mos
>>> dataset = {'p': 126, 'n': 131}
>>> folding = {'n_folds': 2, 'n_repeats': 1}
>>> scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
- >>> result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,
+ >>> result = check_1_dataset_unknown_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -361,7 +372,7 @@ If the balanced accuracy score is adjusted to 0.862, the configuration becomes i
>>> scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.862}
- >>> result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,
+ >>> result = check_1_dataset_unknown_folds_mos(dataset=dataset,
folding=folding,
scores=scores,
eps=1e-3)
@@ -374,7 +385,7 @@ The following scenario is similar in the sense that MoS aggregation is applied t
.. code-block:: Python
- >>> from mlscorecheck.check import check_n_datasets_mos_unknown_folds_mos_scores
+ >>> from mlscorecheck.check.binary import check_n_datasets_mos_unknown_folds_mos
>>> evaluation0 = {'dataset': {'p': 13, 'n': 73},
'folding': {'n_folds': 4, 'n_repeats': 1}}
@@ -384,7 +395,7 @@ The following scenario is similar in the sense that MoS aggregation is applied t
>>> scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.343}
- >>> result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,
+ >>> result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
>>> result['inconsistency']
@@ -396,8 +407,241 @@ The setup is consistent. However, if the balanced accuracy is changed to 0.9, th
>>> scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.9}
- >>> result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,
+ >>> result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,
scores=scores,
eps=1e-3)
>>> result['inconsistency']
# True
+
+Multiclass classification
+=========================
+
+In multiclass classification scenarios single testsets and k-fold cross-validation on a single dataset are supported with both the micro-averaging and macro-averaging aggregation strategies. The list of supported scores depends on the experimental setup, when applicable, all 20 scores listed for binary classification are supported, when the test operates in terms of linear programming, only accuracy, sensitivity, specificity and balanced accuracy are supported.
+
+1 testset, no k-fold, micro-averaging
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this scenario, we suppose there is a multiclass classification testset and the class level scores on the testset are aggregated by micro-averaging. The test is based on exhaustive enumeration, so all 20 performance scores are supported. In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ >>> from mlscorecheck.check.multiclass import check_1_testset_no_kfold_micro
+
+ >>> testset = {0: 10, 1: 100, 2: 80}
+ >>> scores = {'acc': 0.5158, 'sens': 0.2737, 'spec': 0.6368,
+ 'bacc': 0.4553, 'ppv': 0.2737, 'npv': 0.6368}
+ >>> results = check_1_testset_no_kfold_micro(testset=testset,
+ scores=scores,
+ eps=1e-4)
+ >>> results['inconsistency']
+ # False
+
+As the test confirms, the setup is consistent. However, if one of the scores is adjusted a little, for example, accuracy is changed to 0.5258, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ >>> scores['acc'] = 0.5258
+ >>> results = check_1_testset_no_kfold_micro(testset=testset,
+ scores=scores,
+ eps=1e-4)
+ >>> results['inconsistency']
+ # True
+
+1 testset, no k-fold, macro-averaging
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This scenario is similar to the previous one, except the scores are aggregated by macro-averaging. The test is based on linear programming, so only accuracy, sensitivity, specificity and balanced accuracy are supported. In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ >>> from mlscorecheck.check.multiclass import check_1_testset_no_kfold_macro
+
+ >>> testset = {0: 10, 1: 100, 2: 80}
+ >>> scores = {'acc': 0.6, 'sens': 0.3417, 'spec': 0.6928, 'f1p': 0.3308}
+ >>> results = check_1_testset_no_kfold_macro(scores=scores, testset=testset, eps=1e-4)
+ >>> results['inconsistency']
+ # False
+
+As the test confirms, the configuration shows no inconsistency. However, if one of the scores is adjusted a little, for example, accuracy is changed to 0.601, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ >>> scores['acc'] = 0.601
+ >>> results = check_1_testset_no_kfold_macro(scores=scores, testset=testset, eps=1e-4)
+ >>> results['inconsistency']
+ # True
+
+1 dataset, known k-folds, score of means aggregation, micro-averaging
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this scenario, we assume there is a multiclass classification dataset, which is evaluated in a k-fold cross-validation scenario, the class level scores are calculated by micro-averaging, and the fold level results are aggregated in the score of means fashion. The test is based on exhaustive enumeration, therefore, all 20 performance scores are supported.
+
+In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_micro
+
+ >>> dataset = {0: 86, 1: 96, 2: 59, 3: 105}
+ >>> folding = {'folds': [{0: 43, 1: 48, 2: 30, 3: 52}, {0: 43, 1: 48, 2: 29, 3: 53}]}
+ >>> scores = {'acc': 0.6272, 'sens': 0.2543, 'spec': 0.7514, 'f1p': 0.2543}
+
+ >>> result = check_1_dataset_known_folds_som_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+As the test confirms, the scenario is feasible. However, if one of the scores is adjusted a little, for example, sensitivity is changed to 0.2553, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ >>> scores['sens'] = 0.2553
+ >>> result = check_1_dataset_known_folds_som_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+
+1 dataset, known k-folds, score of means aggregation, macro-averaging
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this scenario, we assume there is a multiclass classification dataset, which is evaluated in a k-fold cross-validation scenario, the class level scores are calculated by macro-averaging, and the fold level results are aggregated in the score of means fashion. The test is based on linear programming, thus, only accuracy, sensitivity, specificity and balanced accuracy are supported.
+
+In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_macro
+
+ >>> dataset = {0: 129, 1: 81, 2: 135}
+ >>> folding = {'n_folds': 2, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
+ >>> scores = {'acc': 0.5662, 'sens': 0.3577, 'spec': 0.6767, 'f1p': 0.3481}
+
+ >>> result = check_1_dataset_known_folds_som_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+As the results show, no inconsistency has been identified. However, if accuracy is changed to 0.6662, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ >>> scores['acc'] = 0.6662
+ >>> result = check_1_dataset_known_folds_som_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+
+1 dataset, known k-folds, mean of scores aggregation, micro-averaging
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this scenario, we assume there is a multiclass classification dataset, which is evaluated in a k-fold cross-validation scenario, the class level scores are calculated by micro-averaging, and the fold level results are aggregated in the mean of scores fashion. The test is based on linear programming, thus, only accuracy, sensitivity, specificity and balanced accuracy are supported.
+
+In the first example, an artificially generated, consistent scenario is tested:
+
+.. code-block:: Python
+
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_micro
+
+ >>> dataset = {0: 66, 1: 178, 2: 151}
+ >>> folding = {'folds': [{0: 33, 1: 89, 2: 76}, {0: 33, 1: 89, 2: 75}]}
+ >>> scores = {'acc': 0.5646, 'sens': 0.3469, 'spec': 0.6734, 'f1p': 0.3469}
+
+ >>> result = check_1_dataset_known_folds_mos_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+As the results show, inconsistencies have not been identified. However, if one of the scores, say, accuracy is adjusted to 0.5746, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ >>> scores['acc'] = 0.5746
+ >>> result = check_1_dataset_known_folds_mos_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+
+1 dataset, known k-folds, mean of scores aggregation, macro-averaging
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the last multiclass scenario, we assume there is a multiclass classification dataset, which is evaluated in a k-fold cross-validation scenario, the class level scores are calculated by macro-averaging, and the fold level results are aggregated in the mean of scores fashion. The test is based on linear programming, thus, only accuracy, sensitivity, specificity and balanced accuracy are supported.
+
+In the first example, an artificially generated, consistent scenario is tested:
+
+.. code-block:: Python
+
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_macro
+
+ >>> dataset = {0: 149, 1: 118, 2: 83, 3: 154}
+ >>> folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
+ >>> scores = {'acc': 0.626, 'sens': 0.2483, 'spec': 0.7509, 'f1p': 0.2469}
+
+ >>> result = check_1_dataset_known_folds_mos_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+As the results show, there are no inconsistencies in the configuration. However, if accuracy is changed to 0.656, the configuration becomes infeasible:
+
+.. code-block:: Python
+
+ >>> scores['acc'] = 0.656
+ >>> result = check_1_dataset_known_folds_mos_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+
+Regression
+==========
+
+From the point of view of consistency testing, regression is the hardest problem as the predictions can produce any performance scores. The tests implemented in the package allow testing the relation of the *mean squared error* (``mse``), *root mean squared error* (``rmse``), *mean average error* (``mae``) and *r^2 scores* (``r2``).
+
+1 testset, no k-fold
+^^^^^^^^^^^^^^^^^^^^
+
+In this scenario, we assume there is a regression testset, and the performance scores are calculated on the testset.
+
+In the first example, we test an artificially generated, consistent scenario:
+
+.. code-block:: Python
+
+ >>> from mlscorecheck.check.regression import check_1_testset_no_kfold
+
+ >>> var = 0.0831619 # the variance of the target values in the testset
+ >>> n_samples = 100
+ >>> scores = {'mae': 0.0254, 'r2': 0.9897}
+
+ >>> result = check_1_testset_no_kfold(var=var,
+ n_samples=n_samples,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+As the results show, there is no inconsistency detected. However, if the mae score is adjusted slightly to 0.03, the configuration becomes inconsistent:
+
+.. code-block:: Python
+
+ >>> scores['mae'] = 0.03
+ >>> result = check_1_testset_no_kfold(var=var,
+ n_samples=n_samples,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
diff --git a/docs/02b_retina.rst b/docs/02b_retina.rst
index ff9a3f9..f2db8cb 100644
--- a/docs/02b_retina.rst
+++ b/docs/02b_retina.rst
@@ -12,7 +12,7 @@ The first function enables the testing of performance scores reported for specif
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_drive_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_drive_vessel_image
>>> scores = {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}
>>> identifier = '01'
>>> k = 4
@@ -29,7 +29,7 @@ A similar functionality is provided for the aggregated scores calculated on the
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_drive_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_drive_vessel_aggregated
>>> scores = {'acc': 0.9494, 'sens': 0.7450, 'spec': 0.9793}
>>> k = 4
>>> results = check_drive_vessel_aggregated(scores=scores,
@@ -56,7 +56,7 @@ In the first example, we test the consistency of scores provided for the image '
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_stare_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_stare_vessel_image
>>> img_identifier = 'im0235'
>>> scores = {'acc': 0.4699, 'npv': 0.8993, 'f1p': 0.134}
>>> results = check_stare_vessel_image(image_identifier=img_identifier,
@@ -70,7 +70,7 @@ In the next example, we illustrate the consistency testing of performance scores
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_stare_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_stare_vessel_aggregated
>>> scores = {'acc': 0.4964, 'sens': 0.5793, 'spec': 0.4871, 'bacc': 0.5332}
>>> results = check_stare_vessel_aggregated(imageset='all',
annotator='ah',
@@ -90,7 +90,7 @@ In the first example, we illustrate the consistency testing of scores reported f
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_hrf_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_hrf_vessel_image
>>> scores = {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}
>>> identifier = '13_h'
>>> k = 4
@@ -106,7 +106,7 @@ In the next example, we illustrate the consistency testing of scores aggregated
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_hrf_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_hrf_vessel_aggregated
>>> scores = {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}
>>> k = 4
>>> results = check_hrf_vessel_aggregated(scores=scores,
@@ -130,7 +130,7 @@ In the first example, we illustrate the consistency testing of scores reported f
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_chasedb1_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_chasedb1_vessel_image
>>> img_identifier = '11R'
>>> scores = {'acc': 0.4457, 'sens': 0.0051, 'spec': 0.4706}
>>> results = check_chasedb1_vessel_image(image_identifier=img_identifier,
@@ -146,7 +146,7 @@ The next example illustrates the consistency testing of scores aggregated over a
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_chasedb1_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_chasedb1_vessel_aggregated
>>> scores = {'acc': 0.5063, 'sens': 0.4147, 'spec': 0.5126}
>>> k = 4
>>> results = check_chasedb1_vessel_aggregated(imageset='all',
@@ -176,7 +176,7 @@ In the following example we illustrate the evaluation of scores aggregated over
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_diaretdb0_class
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb0_class
>>> scores = {'acc': 0.4271, 'sens': 0.406, 'spec': 0.4765}
>>> results = check_diaretdb0_class(subset='test',
batch='all',
@@ -199,7 +199,7 @@ In the first example, we illustrate the consistency testing of image labeling, t
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_diaretdb1_class
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb1_class
>>> scores = {'acc': 0.3115, 'sens': 1.0, 'spec': 0.0455, 'f1p': 0.4474}
>>> results = check_diaretdb1_class(subset='test',
class_name=['hardexudates', 'softexudates'],
@@ -215,7 +215,7 @@ In the next example, we test if the reported scores could be yielded from the se
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_diaretdb1_segmentation_image
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb1_segmentation_image
>>> scores = {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}
>>> results = check_diaretdb1_segmentation_image(image_identifier='005',
class_name=['hardexudates', 'softexudates'],
@@ -231,7 +231,7 @@ In the last example, we illustrate the consistency testing of scores aggregated
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_diaretdb1_segmentation_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb1_segmentation_aggregated
>>> scores = {'acc': 0.7143, 'sens': 0.3775, 'spec': 0.7244}
>>> results = check_diaretdb1_segmentation_aggregated(subset='test',
class_name='hardexudates',
@@ -258,7 +258,7 @@ In the first example, we illustrate the consistency testing of image level segme
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_drishti_gs_segmentation_image
+ >>> from mlscorecheck.check.bundles.retina import check_drishti_gs_segmentation_image
>>> scores = {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}
>>> results = check_drishti_gs_segmentation_image(image_identifier='053',
confidence=0.75,
@@ -274,7 +274,7 @@ In the next example, we illustrate the consistency testing of scores aggregated
.. code-block:: Python
- >>> from mlscorecheck.bundles.retina import check_drishti_gs_segmentation_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_drishti_gs_segmentation_aggregated
>>> scores = {'acc': 0.4767, 'sens': 0.4845, 'spec': 0.4765, 'f1p': 0.0512}
>>> results = check_drishti_gs_segmentation_aggregated(subset='test',
confidence=0.75,
diff --git a/docs/02c_ehg.rst b/docs/02c_ehg.rst
index bf74594..5dd4d00 100644
--- a/docs/02c_ehg.rst
+++ b/docs/02c_ehg.rst
@@ -9,7 +9,7 @@ For illustration, given a set of scores reported in a real paper, the test below
.. code-block:: Python
- >>> from mlscorecheck.bundles.ehg import check_tpehg
+ >>> from mlscorecheck.check.bundles.ehg import check_tpehg
>>> # the 5-fold cross-validation scores reported in the paper
>>> scores = {'acc': 0.9447, 'sens': 0.9139, 'spec': 0.9733}
>>> eps = 0.0001
diff --git a/docs/02e_skinlesion.rst b/docs/02e_skinlesion.rst
index c850b62..317f868 100644
--- a/docs/02e_skinlesion.rst
+++ b/docs/02e_skinlesion.rst
@@ -12,7 +12,7 @@ In the following example, we illustrate the consistency testing of performance s
.. code-block:: Python
- >>> from mlscorecheck.bundles.skinlesion import check_isic2016
+ >>> from mlscorecheck.check.bundles.skinlesion import check_isic2016
>>> scores = {'acc': 0.7916, 'sens': 0.2933, 'spec': 0.9145}
>>> results = check_isic2016(scores=scores, eps=1e-4)
>>> results['inconsistency']
@@ -29,7 +29,7 @@ The consistency tests support these types of evaluations. In the following examp
.. code-block:: Python
- >>> from mlscorecheck.bundles.skinlesion import check_isic2017
+ >>> from mlscorecheck.check.bundles.skinlesion import check_isic2017
>>> scores = {'acc': 0.6183, 'sens': 0.4957, 'ppv': 0.2544, 'f1p': 0.3362}
>>> results = check_isic2017(target='M',
against=['SK', 'N'],
diff --git a/docs/03_api.rst b/docs/03_api.rst
index 5519586..0a59c13 100644
--- a/docs/03_api.rst
+++ b/docs/03_api.rst
@@ -9,17 +9,34 @@ Consistency testing (``check``)
The test functions implemented in the ``mlscorecheck.check`` module.
-.. autofunction:: mlscorecheck.check.check_1_testset_no_kfold_scores
-.. autofunction:: mlscorecheck.check.check_1_dataset_som_scores
-.. autofunction:: mlscorecheck.check.check_1_dataset_known_folds_mos_scores
-.. autofunction:: mlscorecheck.check.check_1_dataset_unknown_folds_mos_scores
-.. autofunction:: mlscorecheck.check.check_n_testsets_mos_no_kfold_scores
-.. autofunction:: mlscorecheck.check.check_n_testsets_som_no_kfold_scores
-.. autofunction:: mlscorecheck.check.check_n_datasets_som_kfold_som_scores
-.. autofunction:: mlscorecheck.check.check_n_datasets_mos_kfold_som_scores
-.. autofunction:: mlscorecheck.check.check_n_datasets_mos_known_folds_mos_scores
-.. autofunction:: mlscorecheck.check.check_n_datasets_mos_unknown_folds_mos_scores
-
+Binary classification
+---------------------
+
+.. autofunction:: mlscorecheck.check.binary.check_1_testset_no_kfold
+.. autofunction:: mlscorecheck.check.binary.check_1_dataset_kfold_som
+.. autofunction:: mlscorecheck.check.binary.check_1_dataset_known_folds_mos
+.. autofunction:: mlscorecheck.check.binary.check_1_dataset_unknown_folds_mos
+.. autofunction:: mlscorecheck.check.binary.check_n_testsets_mos_no_kfold
+.. autofunction:: mlscorecheck.check.binary.check_n_testsets_som_no_kfold
+.. autofunction:: mlscorecheck.check.binary.check_n_datasets_som_kfold_som
+.. autofunction:: mlscorecheck.check.binary.check_n_datasets_mos_kfold_som
+.. autofunction:: mlscorecheck.check.binary.check_n_datasets_mos_known_folds_mos
+.. autofunction:: mlscorecheck.check.binary.check_n_datasets_mos_unknown_folds_mos
+
+Multiclass classification
+-------------------------
+
+.. autofunction:: mlscorecheck.check.multiclass.check_1_testset_no_kfold_macro
+.. autofunction:: mlscorecheck.check.multiclass.check_1_testset_no_kfold_micro
+.. autofunction:: mlscorecheck.check.multiclass.check_1_dataset_known_folds_mos_macro
+.. autofunction:: mlscorecheck.check.multiclass.check_1_dataset_known_folds_mos_micro
+.. autofunction:: mlscorecheck.check.multiclass.check_1_dataset_known_folds_som_macro
+.. autofunction:: mlscorecheck.check.multiclass.check_1_dataset_known_folds_som_micro
+
+Regression
+----------
+
+.. autofunction:: mlscorecheck.check.regression.check_1_testset_no_kfold
Test bundles (``bundles``)
==========================
@@ -34,61 +51,61 @@ The test functions dedicated to retina image processing problems.
DRIVE
~~~~~
-.. autofunction:: mlscorecheck.bundles.retina.check_drive_vessel_image
-.. autofunction:: mlscorecheck.bundles.retina.check_drive_vessel_image_assumption
-.. autofunction:: mlscorecheck.bundles.retina.check_drive_vessel_aggregated
-.. autofunction:: mlscorecheck.bundles.retina.check_drive_vessel_aggregated_mos_assumption
-.. autofunction:: mlscorecheck.bundles.retina.check_drive_vessel_aggregated_som_assumption
+.. autofunction:: mlscorecheck.check.bundles.retina.check_drive_vessel_image
+.. autofunction:: mlscorecheck.check.bundles.retina.check_drive_vessel_image_assumption
+.. autofunction:: mlscorecheck.check.bundles.retina.check_drive_vessel_aggregated
+.. autofunction:: mlscorecheck.check.bundles.retina.check_drive_vessel_aggregated_mos_assumption
+.. autofunction:: mlscorecheck.check.bundles.retina.check_drive_vessel_aggregated_som_assumption
STARE
~~~~~
-.. autofunction:: mlscorecheck.bundles.retina.check_stare_vessel_image
-.. autofunction:: mlscorecheck.bundles.retina.check_stare_vessel_aggregated
-.. autofunction:: mlscorecheck.bundles.retina.check_stare_vessel_aggregated_mos
-.. autofunction:: mlscorecheck.bundles.retina.check_stare_vessel_aggregated_som
+.. autofunction:: mlscorecheck.check.bundles.retina.check_stare_vessel_image
+.. autofunction:: mlscorecheck.check.bundles.retina.check_stare_vessel_aggregated
+.. autofunction:: mlscorecheck.check.bundles.retina.check_stare_vessel_aggregated_mos
+.. autofunction:: mlscorecheck.check.bundles.retina.check_stare_vessel_aggregated_som
HRF
~~~
-.. autofunction:: mlscorecheck.bundles.retina.check_hrf_vessel_image
-.. autofunction:: mlscorecheck.bundles.retina.check_hrf_vessel_image_assumption
-.. autofunction:: mlscorecheck.bundles.retina.check_hrf_vessel_aggregated
-.. autofunction:: mlscorecheck.bundles.retina.check_hrf_vessel_aggregated_mos_assumption
-.. autofunction:: mlscorecheck.bundles.retina.check_hrf_vessel_aggregated_som_assumption
+.. autofunction:: mlscorecheck.check.bundles.retina.check_hrf_vessel_image
+.. autofunction:: mlscorecheck.check.bundles.retina.check_hrf_vessel_image_assumption
+.. autofunction:: mlscorecheck.check.bundles.retina.check_hrf_vessel_aggregated
+.. autofunction:: mlscorecheck.check.bundles.retina.check_hrf_vessel_aggregated_mos_assumption
+.. autofunction:: mlscorecheck.check.bundles.retina.check_hrf_vessel_aggregated_som_assumption
CHASE_DB1
~~~~~~~~~
-.. autofunction:: mlscorecheck.bundles.retina.check_chasedb1_vessel_image
-.. autofunction:: mlscorecheck.bundles.retina.check_chasedb1_vessel_aggregated
-.. autofunction:: mlscorecheck.bundles.retina.check_chasedb1_vessel_aggregated_mos
-.. autofunction:: mlscorecheck.bundles.retina.check_chasedb1_vessel_aggregated_som
+.. autofunction:: mlscorecheck.check.bundles.retina.check_chasedb1_vessel_image
+.. autofunction:: mlscorecheck.check.bundles.retina.check_chasedb1_vessel_aggregated
+.. autofunction:: mlscorecheck.check.bundles.retina.check_chasedb1_vessel_aggregated_mos
+.. autofunction:: mlscorecheck.check.bundles.retina.check_chasedb1_vessel_aggregated_som
DIARETDB0
~~~~~~~~~
-.. autofunction:: mlscorecheck.bundles.retina.check_diaretdb0_class
+.. autofunction:: mlscorecheck.check.bundles.retina.check_diaretdb0_class
DIARETDB1
~~~~~~~~~
-.. autofunction:: mlscorecheck.bundles.retina.check_diaretdb1_class
-.. autofunction:: mlscorecheck.bundles.retina.check_diaretdb1_segmentation_image
-.. autofunction:: mlscorecheck.bundles.retina.check_diaretdb1_segmentation_aggregated
+.. autofunction:: mlscorecheck.check.bundles.retina.check_diaretdb1_class
+.. autofunction:: mlscorecheck.check.bundles.retina.check_diaretdb1_segmentation_image
+.. autofunction:: mlscorecheck.check.bundles.retina.check_diaretdb1_segmentation_aggregated
DRISHTI_GS
~~~~~~~~~~
-.. autofunction:: mlscorecheck.bundles.retina.check_drishti_gs_segmentation_image
-.. autofunction:: mlscorecheck.bundles.retina.check_drishti_gs_segmentation_aggregated
+.. autofunction:: mlscorecheck.check.bundles.retina.check_drishti_gs_segmentation_image
+.. autofunction:: mlscorecheck.check.bundles.retina.check_drishti_gs_segmentation_aggregated
Preterm delivery prediction by EHG signals
------------------------------------------
The test bundle dedicated to the testing of electrohsyterogram data.
-.. autofunction:: mlscorecheck.bundles.ehg.check_tpehg
+.. autofunction:: mlscorecheck.check.bundles.ehg.check_tpehg
Skin lesion classification
--------------------------
@@ -98,12 +115,12 @@ The test bundle dedicated to the testing of skin lesion classification.
ISIC2016
~~~~~~~~
-.. autofunction:: mlscorecheck.bundles.skinlesion.check_isic2016
+.. autofunction:: mlscorecheck.check.bundles.skinlesion.check_isic2016
ISIC2017
~~~~~~~~
-.. autofunction:: mlscorecheck.bundles.skinlesion.check_isic2017
+.. autofunction:: mlscorecheck.check.bundles.skinlesion.check_isic2017
Experiments (``experiments``)
=============================
diff --git a/docs/index.rst b/docs/index.rst
index 70e26f1..f9e2809 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -6,8 +6,8 @@
.. raw:: html
:file: ga4.html
-Welcome to mlscorecheck's documentation!
-==========================================
+mlscorecheck: testing the consistency of machine learning performance scores
+****************************************************************************
.. toctree::
:maxdepth: 2
@@ -20,7 +20,6 @@ Welcome to mlscorecheck's documentation!
:caption: Consistency testing
01a_requirements
- 01b_specifying_setup
01c_consistency_checking
.. toctree::
diff --git a/mlscorecheck/check/_check_1_dataset_unknown_folds_mos_acc_score.py b/legacy/_check_1_dataset_unknown_folds_mos_acc_score.py
similarity index 96%
rename from mlscorecheck/check/_check_1_dataset_unknown_folds_mos_acc_score.py
rename to legacy/_check_1_dataset_unknown_folds_mos_acc_score.py
index 4b0380a..306e045 100644
--- a/mlscorecheck/check/_check_1_dataset_unknown_folds_mos_acc_score.py
+++ b/legacy/_check_1_dataset_unknown_folds_mos_acc_score.py
@@ -5,8 +5,8 @@
import itertools
-from ..core import NUMERICAL_TOLERANCE
-from ..aggregated import Evaluation
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import Evaluation
__all__ = ['check_1_dataset_unknown_folds_mos_acc_score']
diff --git a/tests/check/test_1_dataset_unknown_folds_mos_acc_score.py b/legacy/test_1_dataset_unknown_folds_mos_acc_score.py
similarity index 97%
rename from tests/check/test_1_dataset_unknown_folds_mos_acc_score.py
rename to legacy/test_1_dataset_unknown_folds_mos_acc_score.py
index 2f987e6..c2364b9 100644
--- a/tests/check/test_1_dataset_unknown_folds_mos_acc_score.py
+++ b/legacy/test_1_dataset_unknown_folds_mos_acc_score.py
@@ -7,7 +7,7 @@
import numpy as np
-from mlscorecheck.check import check_1_dataset_unknown_folds_mos_acc_score
+from mlscorecheck.check.binary import check_1_dataset_unknown_folds_mos_acc_score
from mlscorecheck.aggregated import (generate_dataset,
Evaluation, Folding)
diff --git a/mlscorecheck/__init__.py b/mlscorecheck/__init__.py
index 8d9a00a..adffef6 100644
--- a/mlscorecheck/__init__.py
+++ b/mlscorecheck/__init__.py
@@ -8,5 +8,4 @@
from . import symbolic
from . import individual
from . import aggregated
-from . import bundles
from ._version import *
diff --git a/mlscorecheck/_version.py b/mlscorecheck/_version.py
index 3695874..0363fca 100644
--- a/mlscorecheck/_version.py
+++ b/mlscorecheck/_version.py
@@ -2,4 +2,4 @@
This module contains the version of the package.
"""
-__version__ = "1.0.0"
+__version__ = "1.0.1"
diff --git a/mlscorecheck/aggregated/__init__.py b/mlscorecheck/aggregated/__init__.py
index f931259..e235f02 100644
--- a/mlscorecheck/aggregated/__init__.py
+++ b/mlscorecheck/aggregated/__init__.py
@@ -4,6 +4,7 @@
from ._folding import *
from ._fold import *
+from ._fold_enumeration import *
from ._dataset import *
from ._evaluation import *
from ._experiment import *
diff --git a/mlscorecheck/aggregated/_check_aggregated_scores.py b/mlscorecheck/aggregated/_check_aggregated_scores.py
index 293c1f4..29b5dd5 100644
--- a/mlscorecheck/aggregated/_check_aggregated_scores.py
+++ b/mlscorecheck/aggregated/_check_aggregated_scores.py
@@ -5,27 +5,34 @@
import pulp as pl
-from ..core import (logger, NUMERICAL_TOLERANCE, check_uncertainty_and_tolerance,
- update_uncertainty)
+from ..core import (
+ logger,
+ NUMERICAL_TOLERANCE,
+ check_uncertainty_and_tolerance,
+ update_uncertainty,
+)
from ..individual import resolve_aliases_and_complements
from ._experiment import Experiment
from ._linear_programming import solve
from ._utils import compare_scores, aggregated_scores
-__all__ = ['check_aggregated_scores']
+__all__ = ["check_aggregated_scores"]
solvers = pl.listSolvers(onlyAvailable=True)
-PREFERRED_SOLVER = 'PULP_CBC_CMD' if 'PULP_CBC_CMD' in solvers else solvers[0]
-
-def check_aggregated_scores(*,
- experiment: dict,
- scores: dict,
- eps,
- solver_name: str = None,
- timeout: int = None,
- verbosity: int = 1,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
+PREFERRED_SOLVER = "PULP_CBC_CMD" if "PULP_CBC_CMD" in solvers else solvers[0]
+
+
+def check_aggregated_scores(
+ *,
+ experiment: dict,
+ scores: dict,
+ eps,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
"""
Check aggregated scores
@@ -59,22 +66,37 @@ def check_aggregated_scores(*,
scores = resolve_aliases_and_complements(scores)
if all(score not in aggregated_scores for score in scores):
- logger.info('there are no scores suitable for aggregated checks')
- return {'inconsistency': False,
- 'message': 'no scores suitable for aggregated consistency checks'}
-
- experiment = Experiment(**experiment) if isinstance(experiment, dict) else experiment
-
- if experiment.aggregation == 'som' \
- and any(evaluation.aggregation == 'mos' for evaluation in experiment.evaluations):
- raise ValueError('experiment level MoS aggregation with dataset level SoM '\
- 'aggregation is an unlikely situation, it is not supported in this high level '\
- 'interface.')
-
- solver_name = (PREFERRED_SOLVER if solver_name is None or solver_name not in solvers
- else solver_name)
-
- solver = pl.getSolver(solver_name, timeLimit=timeout, msg=verbosity)
+ logger.info("there are no scores suitable for aggregated checks")
+ return {
+ "inconsistency": False,
+ "message": "no scores suitable for aggregated consistency checks",
+ }
+
+ experiment = (
+ Experiment(**experiment) if isinstance(experiment, dict) else experiment
+ )
+
+ if experiment.aggregation == "som" and any(
+ evaluation.aggregation == "mos" for evaluation in experiment.evaluations
+ ):
+ raise ValueError(
+ "experiment level MoS aggregation with dataset level SoM "
+ "aggregation is an unlikely situation, it is not supported in this high level "
+ "interface."
+ )
+
+ solver_name = (
+ PREFERRED_SOLVER
+ if solver_name is None or solver_name not in solvers
+ else solver_name
+ )
+
+ solver = pl.getSolver(
+ solver_name,
+ timeLimit=timeout,
+ msg=verbosity,
+ options=["RandomS 1", "RandomC 1", "randomS 1", "randomC 1"],
+ )
result = solve(experiment, scores, eps, solver)
@@ -84,21 +106,27 @@ def check_aggregated_scores(*,
if result.status == 1:
# the problem is feasible
- comp_flag = compare_scores(scores, populated.scores, eps+numerical_tolerance)
- bounds_flag = configuration_details['bounds_flag']
- return {'inconsistency': False,
- 'lp_scores': populated.scores,
- 'lp_status': 'feasible',
- 'lp_configuration_scores_match': comp_flag,
- 'lp_configuration_bounds_match': bounds_flag,
- 'lp_configuration': configuration_details}
+ comp_flag = compare_scores(scores, populated.scores, eps + numerical_tolerance)
+ bounds_flag = configuration_details["bounds_flag"]
+ return {
+ "inconsistency": False,
+ "lp_scores": populated.scores,
+ "lp_status": "feasible",
+ "lp_configuration_scores_match": comp_flag,
+ "lp_configuration_bounds_match": bounds_flag,
+ "lp_configuration": configuration_details,
+ }
if result.status == 0:
# timed out
- return {'inconsistency': False,
- 'lp_status': 'timeout',
- 'lp_configuration': configuration_details}
+ return {
+ "inconsistency": False,
+ "lp_status": "timeout",
+ "lp_configuration": configuration_details,
+ }
# infeasible
- return {'inconsistency': True,
- 'lp_status': 'infeasible',
- 'lp_configuration': configuration_details}
+ return {
+ "inconsistency": True,
+ "lp_status": "infeasible",
+ "lp_configuration": configuration_details,
+ }
diff --git a/mlscorecheck/aggregated/_dataset.py b/mlscorecheck/aggregated/_dataset.py
index 83eb7e8..9977477 100644
--- a/mlscorecheck/aggregated/_dataset.py
+++ b/mlscorecheck/aggregated/_dataset.py
@@ -8,17 +8,21 @@
from ..experiments import dataset_statistics
-__all__ = ['Dataset']
+__all__ = ["Dataset"]
+
class Dataset:
"""
The abstract representation of a dataset
"""
- def __init__(self,
- p: int = None,
- n: int = None,
- dataset_name: str = None,
- identifier: str = None):
+
+ def __init__(
+ self,
+ p: int = None,
+ n: int = None,
+ dataset_name: str = None,
+ identifier: str = None,
+ ):
"""
Constructor of a dataset
@@ -31,11 +35,11 @@ def __init__(self,
None)
"""
if (p is None and n is not None) or (p is not None and n is None):
- raise ValueError('specify either p and n or neither of them')
- if (p is None and dataset_name is None):
- raise ValueError('specify either p and n or the name')
- if (p is not None and dataset_name is not None):
- raise ValueError('specify either p and n or the name')
+ raise ValueError("specify either p and n or neither of them")
+ if p is None and dataset_name is None:
+ raise ValueError("specify either p and n or the name")
+ if p is not None and dataset_name is not None:
+ raise ValueError("specify either p and n or the name")
self.p = p
self.n = n
@@ -44,9 +48,11 @@ def __init__(self,
self.resolve_pn()
if identifier is None:
- self.identifier = (f'{dataset_name}_{random_identifier(3)}'
- if dataset_name is not None
- else random_identifier(5))
+ self.identifier = (
+ f"{dataset_name}_{random_identifier(3)}"
+ if dataset_name is not None
+ else random_identifier(5)
+ )
else:
self.identifier = identifier
@@ -56,8 +62,8 @@ def resolve_pn(self):
"""
if self.p is None:
dataset = dataset_statistics[self.dataset_name]
- self.p = dataset['p']
- self.n = dataset['n']
+ self.p = dataset["p"]
+ self.n = dataset["n"]
def to_dict(self) -> dict:
"""
@@ -66,7 +72,9 @@ def to_dict(self) -> dict:
Returns:
dict: to_dict
"""
- return {'p': self.p if self.dataset_name is None else None,
- 'n': self.n if self.dataset_name is None else None,
- 'dataset_name': self.dataset_name,
- 'identifier': self.identifier}
+ return {
+ "p": self.p if self.dataset_name is None else None,
+ "n": self.n if self.dataset_name is None else None,
+ "dataset_name": self.dataset_name,
+ "identifier": self.identifier,
+ }
diff --git a/mlscorecheck/aggregated/_evaluation.py b/mlscorecheck/aggregated/_evaluation.py
index 64417a9..d62ef89 100644
--- a/mlscorecheck/aggregated/_evaluation.py
+++ b/mlscorecheck/aggregated/_evaluation.py
@@ -6,20 +6,24 @@
from ._dataset import Dataset
from ._folding import Folding
-from ..core import (init_random_state, dict_mean, round_scores,
- NUMERICAL_TOLERANCE)
-from ..individual import calculate_scores_for_lp
+from ..core import init_random_state, dict_mean, round_scores, NUMERICAL_TOLERANCE
+from ..scores import calculate_scores_for_lp
from ._utils import check_bounds, aggregated_scores
from ._linear_programming import add_bounds
+
class Evaluation:
"""
Abstract representation of an evaluation
"""
- def __init__(self, dataset: dict,
- folding: dict,
- aggregation: str,
- fold_score_bounds: dict = None):
+
+ def __init__(
+ self,
+ dataset: dict,
+ folding: dict,
+ aggregation: str,
+ fold_score_bounds: dict = None,
+ ):
"""
Constructor of the object
@@ -35,16 +39,20 @@ def __init__(self, dataset: dict,
self.fold_score_bounds = fold_score_bounds
self.aggregation = aggregation
- if aggregation == 'som' and fold_score_bounds is not None:
- raise ValueError('It is unlikely that fold score bounds are set for a SoM '\
- 'aggregation, therefore, it is not supported.')
+ if aggregation == "som" and fold_score_bounds is not None:
+ raise ValueError(
+ "It is unlikely that fold score bounds are set for a SoM "
+ "aggregation, therefore, it is not supported."
+ )
self.folds = self.folding.generate_folds(self.dataset, self.aggregation)
- self.figures = {'tp': None,
- 'tn': None,
- 'p': sum(fold.p for fold in self.folds),
- 'n': sum(fold.n for fold in self.folds)}
+ self.figures = {
+ "tp": None,
+ "tn": None,
+ "p": sum(fold.p for fold in self.folds),
+ "n": sum(fold.n for fold in self.folds),
+ }
self.scores = None
@@ -55,12 +63,14 @@ def to_dict(self) -> dict:
Returns:
dict: the dictionary representation
"""
- return {'dataset': self.dataset.to_dict(),
- 'folding': self.folding.to_dict(),
- 'fold_score_bounds': self.fold_score_bounds,
- 'aggregation': self.aggregation}
-
- def sample_figures(self, random_state = None, score_subset: list = None):
+ return {
+ "dataset": self.dataset.to_dict(),
+ "folding": self.folding.to_dict(),
+ "fold_score_bounds": self.fold_score_bounds,
+ "aggregation": self.aggregation,
+ }
+
+ def sample_figures(self, random_state=None, score_subset: list = None):
"""
Samples the figures in the evaluation
@@ -79,9 +89,9 @@ def sample_figures(self, random_state = None, score_subset: list = None):
return self
- def calculate_scores(self,
- rounding_decimals: int = None,
- score_subset: list = None) -> dict:
+ def calculate_scores(
+ self, rounding_decimals: int = None, score_subset: list = None
+ ) -> dict:
"""
Calculates the scores
@@ -99,23 +109,26 @@ def calculate_scores(self,
fold.calculate_scores(score_subset=score_subset)
if isinstance(self.folds[0].tp, pl.LpVariable):
- self.figures['tp'] = pl.lpSum(fold.tp for fold in self.folds)
- self.figures['tn'] = pl.lpSum(fold.tn for fold in self.folds)
+ self.figures["tp"] = pl.lpSum(fold.tp for fold in self.folds)
+ self.figures["tn"] = pl.lpSum(fold.tn for fold in self.folds)
else:
- self.figures['tp'] = sum(fold.tp for fold in self.folds)
- self.figures['tn'] = sum(fold.tn for fold in self.folds)
-
- if self.aggregation == 'som':
- self.scores = calculate_scores_for_lp(self.figures, score_subset=score_subset)
- elif self.aggregation == 'mos':
+ self.figures["tp"] = sum(fold.tp for fold in self.folds)
+ self.figures["tn"] = sum(fold.tn for fold in self.folds)
+
+ if self.aggregation == "som":
+ self.scores = calculate_scores_for_lp(
+ self.figures, score_subset=score_subset
+ )
+ elif self.aggregation == "mos":
self.scores = dict_mean([fold.scores for fold in self.folds])
- return self.scores if rounding_decimals is None else round_scores(self.scores,
- rounding_decimals)
+ return (
+ self.scores
+ if rounding_decimals is None
+ else round_scores(self.scores, rounding_decimals)
+ )
- def init_lp(self,
- lp_problem: pl.LpProblem,
- scores: dict = None) -> pl.LpProblem:
+ def init_lp(self, lp_problem: pl.LpProblem, scores: dict = None) -> pl.LpProblem:
"""
Initializes a linear programming problem
@@ -141,8 +154,7 @@ def init_lp(self,
return lp_problem
- def populate(self,
- lp_problem: pl.LpProblem):
+ def populate(self, lp_problem: pl.LpProblem):
"""
Populates the evaluation with the figures in the solved linear programming problem
@@ -157,8 +169,7 @@ def populate(self,
return self
- def check_bounds(self,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
+ def check_bounds(self, numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
Check the bounds in the problem
@@ -169,18 +180,20 @@ def check_bounds(self,
dict: a summary of the test, with the boolean flag under ``bounds_flag``
indicating the overall results
"""
- results = {'folds': []}
+ results = {"folds": []}
for fold in self.folds:
- tmp = {'fold': fold.to_dict() | {'tp': fold.tp, 'tn': fold.tn},
- 'scores': fold.scores,
- 'score_bounds': self.fold_score_bounds}
+ tmp = {
+ "fold": fold.to_dict() | {"tp": fold.tp, "tn": fold.tn},
+ "scores": fold.scores,
+ "score_bounds": self.fold_score_bounds,
+ }
if self.fold_score_bounds is not None:
- tmp['bounds_flag'] = check_bounds(fold.scores,
- self.fold_score_bounds,
- numerical_tolerance)
+ tmp["bounds_flag"] = check_bounds(
+ fold.scores, self.fold_score_bounds, numerical_tolerance
+ )
else:
- tmp['bounds_flag'] = True
- results['folds'].append(tmp)
- results['bounds_flag'] = all(fold['bounds_flag'] for fold in results['folds'])
+ tmp["bounds_flag"] = True
+ results["folds"].append(tmp)
+ results["bounds_flag"] = all(fold["bounds_flag"] for fold in results["folds"])
return results
diff --git a/mlscorecheck/aggregated/_experiment.py b/mlscorecheck/aggregated/_experiment.py
index 8c3e834..f789206 100644
--- a/mlscorecheck/aggregated/_experiment.py
+++ b/mlscorecheck/aggregated/_experiment.py
@@ -4,25 +4,25 @@
import pulp as pl
-from ..core import (init_random_state, dict_mean, round_scores,
- NUMERICAL_TOLERANCE)
-from ..individual import calculate_scores_for_lp
+from ..core import init_random_state, dict_mean, round_scores, NUMERICAL_TOLERANCE
+from ..scores import calculate_scores_for_lp
from ._evaluation import Evaluation
from ._utils import check_bounds, aggregated_scores
from ._linear_programming import add_bounds
-__all__ = ['Experiment']
+__all__ = ["Experiment"]
+
class Experiment:
"""
Abstract representation of an experiment
"""
- def __init__(self,
- evaluations: list,
- aggregation: str,
- dataset_score_bounds: dict = None):
+
+ def __init__(
+ self, evaluations: list, aggregation: str, dataset_score_bounds: dict = None
+ ):
"""
Constructor of the experiment
@@ -35,14 +35,18 @@ def __init__(self,
self.dataset_score_bounds = dataset_score_bounds
self.aggregation = aggregation
- if aggregation == 'som' and dataset_score_bounds is not None:
- raise ValueError('It is unlikely that fold score bounds are set for a SoM '\
- 'aggregation, therefore, it is not supported.')
+ if aggregation == "som" and dataset_score_bounds is not None:
+ raise ValueError(
+ "It is unlikely that fold score bounds are set for a SoM "
+ "aggregation, therefore, it is not supported."
+ )
- self.figures = {'tp': None,
- 'tn': None,
- 'p': sum(evaluation.figures['p'] for evaluation in self.evaluations),
- 'n': sum(evaluation.figures['n'] for evaluation in self.evaluations)}
+ self.figures = {
+ "tp": None,
+ "tn": None,
+ "p": sum(evaluation.figures["p"] for evaluation in self.evaluations),
+ "n": sum(evaluation.figures["n"] for evaluation in self.evaluations),
+ }
self.scores = None
@@ -53,11 +57,13 @@ def to_dict(self) -> dict:
Returns:
dict: the dictionary representation of the object
"""
- return {'evaluations': [evaluation.to_dict() for evaluation in self.evaluations],
- 'dataset_score_bounds': self.dataset_score_bounds,
- 'aggregation': self.aggregation}
+ return {
+ "evaluations": [evaluation.to_dict() for evaluation in self.evaluations],
+ "dataset_score_bounds": self.dataset_score_bounds,
+ "aggregation": self.aggregation,
+ }
- def sample_figures(self, random_state = None, score_subset: list = None):
+ def sample_figures(self, random_state=None, score_subset: list = None):
"""
Samples the ``tp`` and ``tn`` figures
@@ -76,9 +82,9 @@ def sample_figures(self, random_state = None, score_subset: list = None):
return self
- def calculate_scores(self,
- rounding_decimals: int = None,
- score_subset: list = None) -> dict:
+ def calculate_scores(
+ self, rounding_decimals: int = None, score_subset: list = None
+ ) -> dict:
"""
Calculates the scores
@@ -89,35 +95,47 @@ def calculate_scores(self,
Returns:
dict(str,float): the scores
"""
- score_subset = ['acc', 'sens', 'spec', 'bacc'] if score_subset is None else score_subset
- score_subset = [score for score in score_subset if score in ['acc', 'sens', 'spec', 'bacc']]
+ score_subset = (
+ ["acc", "sens", "spec", "bacc"] if score_subset is None else score_subset
+ )
+ score_subset = [
+ score for score in score_subset if score in ["acc", "sens", "spec", "bacc"]
+ ]
for evaluation in self.evaluations:
evaluation.calculate_scores(score_subset=score_subset)
if isinstance(self.evaluations[0].folds[0].tp, pl.LpVariable):
- self.figures['tp'] = pl.lpSum(evaluation.figures['tp']
- for evaluation in self.evaluations)
- self.figures['tn'] = pl.lpSum(evaluation.figures['tn']
- for evaluation in self.evaluations)
+ self.figures["tp"] = pl.lpSum(
+ evaluation.figures["tp"] for evaluation in self.evaluations
+ )
+ self.figures["tn"] = pl.lpSum(
+ evaluation.figures["tn"] for evaluation in self.evaluations
+ )
else:
- self.figures['tp'] = sum(evaluation.figures['tp']
- for evaluation in self.evaluations)
- self.figures['tn'] = sum(evaluation.figures['tn']
- for evaluation in self.evaluations)
-
- if self.aggregation == 'som':
- self.scores = calculate_scores_for_lp(self.figures,
- score_subset=score_subset)
- elif self.aggregation == 'mos':
- self.scores = dict_mean([evaluation.scores for evaluation in self.evaluations])
-
- return self.scores if rounding_decimals is None else round_scores(self.scores,
- rounding_decimals)
-
- def init_lp(self,
- lp_problem: pl.LpProblem,
- scores: dict = None) -> pl.LpProblem:
+ self.figures["tp"] = sum(
+ evaluation.figures["tp"] for evaluation in self.evaluations
+ )
+ self.figures["tn"] = sum(
+ evaluation.figures["tn"] for evaluation in self.evaluations
+ )
+
+ if self.aggregation == "som":
+ self.scores = calculate_scores_for_lp(
+ self.figures, score_subset=score_subset
+ )
+ elif self.aggregation == "mos":
+ self.scores = dict_mean(
+ [evaluation.scores for evaluation in self.evaluations]
+ )
+
+ return (
+ self.scores
+ if rounding_decimals is None
+ else round_scores(self.scores, rounding_decimals)
+ )
+
+ def init_lp(self, lp_problem: pl.LpProblem, scores: dict = None) -> pl.LpProblem:
"""
Initializes a linear programming problem
@@ -130,8 +148,7 @@ def init_lp(self,
"""
for evaluation in self.evaluations:
- evaluation.init_lp(lp_problem,
- scores=scores)
+ evaluation.init_lp(lp_problem, scores=scores)
score_subset = aggregated_scores
if scores is not None:
@@ -140,10 +157,12 @@ def init_lp(self,
self.calculate_scores(score_subset=score_subset)
for evaluation in self.evaluations:
- add_bounds(lp_problem,
- evaluation.scores,
- self.dataset_score_bounds,
- evaluation.dataset.identifier)
+ add_bounds(
+ lp_problem,
+ evaluation.scores,
+ self.dataset_score_bounds,
+ evaluation.dataset.identifier,
+ )
return lp_problem
@@ -174,21 +193,24 @@ def check_bounds(self, numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict
indicating the overall results
"""
- results = {'evaluations': []}
+ results = {"evaluations": []}
for evaluation in self.evaluations:
- tmp = {'folds': evaluation.check_bounds(numerical_tolerance),
- 'scores': evaluation.scores,
- 'score_bounds': self.dataset_score_bounds}
+ tmp = {
+ "folds": evaluation.check_bounds(numerical_tolerance),
+ "scores": evaluation.scores,
+ "score_bounds": self.dataset_score_bounds,
+ }
if self.dataset_score_bounds is not None:
- tmp['bounds_flag'] = check_bounds(evaluation.scores,
- self.dataset_score_bounds,
- numerical_tolerance)
- tmp['bounds_flag'] = tmp['bounds_flag'] and tmp['folds']['bounds_flag']
+ tmp["bounds_flag"] = check_bounds(
+ evaluation.scores, self.dataset_score_bounds, numerical_tolerance
+ )
+ tmp["bounds_flag"] = tmp["bounds_flag"] and tmp["folds"]["bounds_flag"]
else:
- tmp['bounds_flag'] = tmp['folds']
- results['evaluations'].append(tmp)
+ tmp["bounds_flag"] = tmp["folds"]
+ results["evaluations"].append(tmp)
- results['bounds_flag'] = all(evaluation['bounds_flag']
- for evaluation in results['evaluations'])
+ results["bounds_flag"] = all(
+ evaluation["bounds_flag"] for evaluation in results["evaluations"]
+ )
return results
diff --git a/mlscorecheck/aggregated/_fold.py b/mlscorecheck/aggregated/_fold.py
index 2bce384..65beaaf 100644
--- a/mlscorecheck/aggregated/_fold.py
+++ b/mlscorecheck/aggregated/_fold.py
@@ -9,21 +9,20 @@
import pulp as pl
-from ..core import (init_random_state, round_scores)
-from ..individual import calculate_scores_for_lp
+from ..core import init_random_state, round_scores
+from ..scores import calculate_scores_for_lp
from ._utils import random_identifier, aggregated_scores
-__all__ = ['Fold']
+__all__ = ["Fold"]
+
class Fold:
"""
Abstract representation of a fold
"""
- def __init__(self,
- p: int,
- n: int,
- identifier: str = None):
+
+ def __init__(self, p: int, n: int, identifier: str = None):
"""
Constructor of a fold
@@ -40,8 +39,10 @@ def __init__(self,
self.tn = None
self.scores = None
- self.variable_names = {'tp': f'tp_{self.identifier}'.replace('-', '_'),
- 'tn': f'tn_{self.identifier}'.replace('-', '_')}
+ self.variable_names = {
+ "tp": f"tp_{self.identifier}".replace("-", "_"),
+ "tn": f"tn_{self.identifier}".replace("-", "_"),
+ }
def to_dict(self) -> dict:
"""
@@ -50,11 +51,9 @@ def to_dict(self) -> dict:
Returns:
dict: the dictionary representation
"""
- return {'p': self.p,
- 'n': self.n,
- 'identifier': self.identifier}
+ return {"p": self.p, "n": self.n, "identifier": self.identifier}
- def sample_figures(self, random_state = None):
+ def sample_figures(self, random_state=None):
"""
Samples the ``tp`` and ``tn`` figures
@@ -66,14 +65,14 @@ def sample_figures(self, random_state = None):
"""
random_state = init_random_state(random_state)
- self.tp = random_state.randint(self.p+1)
- self.tn = random_state.randint(self.n+1)
+ self.tp = random_state.randint(self.p + 1)
+ self.tn = random_state.randint(self.n + 1)
return self
- def calculate_scores(self,
- rounding_decimals: int = None,
- score_subset: list = None) -> dict:
+ def calculate_scores(
+ self, rounding_decimals: int = None, score_subset: list = None
+ ) -> dict:
"""
Calculate the scores for the fold
@@ -86,14 +85,16 @@ def calculate_scores(self,
"""
score_subset = score_subset if score_subset is not None else aggregated_scores
- self.scores = calculate_scores_for_lp({'p': self.p,
- 'n': self.n,
- 'tp': self.tp,
- 'tn': self.tn},
- score_subset=score_subset)
+ self.scores = calculate_scores_for_lp(
+ {"p": self.p, "n": self.n, "tp": self.tp, "tn": self.tn},
+ score_subset=score_subset,
+ )
- return self.scores if rounding_decimals is None else round_scores(self.scores,
- rounding_decimals)
+ return (
+ self.scores
+ if rounding_decimals is None
+ else round_scores(self.scores, rounding_decimals)
+ )
def set_initial_values(self, scores):
"""
@@ -102,17 +103,17 @@ def set_initial_values(self, scores):
Args:
scores (dict): the dictionary of scores
"""
- if 'acc' in scores:
- tp_init = scores['acc'] * self.p
- tn_init = scores['acc'] * self.n
- if 'bacc' in scores:
- tp_init = scores['bacc'] * self.p
- tn_init = scores['bacc'] * self.n
-
- if 'sens' in scores:
- tp_init = scores['sens'] * self.p
- if 'spec' in scores:
- tn_init = scores['spec'] * self.n
+ if "acc" in scores:
+ tp_init = scores["acc"] * self.p
+ tn_init = scores["acc"] * self.n
+ if "bacc" in scores:
+ tp_init = scores["bacc"] * self.p
+ tn_init = scores["bacc"] * self.n
+
+ if "sens" in scores:
+ tp_init = scores["sens"] * self.p
+ if "spec" in scores:
+ tn_init = scores["spec"] * self.n
self.tp.setInitialValue(int(tp_init))
self.tn.setInitialValue(int(tn_init))
@@ -127,8 +128,8 @@ def init_lp(self, scores: dict = None):
Returns:
pl.LpProblem: the updated problem
"""
- self.tp = pl.LpVariable(self.variable_names['tp'], 0, self.p, pl.LpInteger)
- self.tn = pl.LpVariable(self.variable_names['tn'], 0, self.n, pl.LpInteger)
+ self.tp = pl.LpVariable(self.variable_names["tp"], 0, self.p, pl.LpInteger)
+ self.tn = pl.LpVariable(self.variable_names["tn"], 0, self.n, pl.LpInteger)
if scores is not None:
self.set_initial_values(scores)
@@ -150,9 +151,9 @@ def populate(self, lp_problem: pl.LpProblem) -> pl.LpProblem:
obj: the self object populated with the ``tp`` and ``tn`` scores
"""
for variable in lp_problem.variables():
- if variable.name == self.variable_names['tp']:
+ if variable.name == self.variable_names["tp"]:
self.tp = variable.varValue
- if variable.name == self.variable_names['tn']:
+ if variable.name == self.variable_names["tn"]:
self.tn = variable.varValue
return self
diff --git a/mlscorecheck/aggregated/_fold_enumeration.py b/mlscorecheck/aggregated/_fold_enumeration.py
new file mode 100644
index 0000000..9af33be
--- /dev/null
+++ b/mlscorecheck/aggregated/_fold_enumeration.py
@@ -0,0 +1,366 @@
+"""
+This module implements the functionalities for enumerating fold configurations
+"""
+
+import copy
+import itertools
+
+from ..core import logger
+from ..experiments import dataset_statistics
+from ._utils import random_identifier
+
+__all__ = [
+ "integer_partitioning_generator",
+ "all_integer_partitioning_generator",
+ "fold_partitioning_generator",
+ "determine_min_max_p",
+ "kfolds_generator",
+ "repeated_kfolds_generator",
+ "experiment_kfolds_generator",
+ "_check_specification_and_determine_p_n",
+]
+
+
+def integer_partitioning_generator(n: int, m: int): # pylint: disable=invalid-name
+ """
+ Integer partitioning generator
+
+ Integer partitioning algorithm implemented following the algorithm on page 343 in
+ https://doi.org/10.1007/978-3-642-14764-7
+
+ Args:
+ n (int): the integer to partition
+ m (int): the number of partitions
+
+ Yields:
+ list: the next configuration
+ """
+ x = [0] * (m + 1) # pylint: disable=invalid-name
+ s = [0] * (m + 1) # pylint: disable=invalid-name
+
+ for k in range(1, m): # pylint: disable=invalid-name
+ x[k] = 1
+
+ x[m] = n - m + 1
+
+ for k in range(1, m + 1): # pylint: disable=invalid-name
+ s[k] = x[k] + s[k - 1]
+
+ while True:
+ yield x[1:]
+
+ u = x[m] # pylint: disable=invalid-name
+ k = m # pylint: disable=invalid-name
+ while k > 0:
+ k -= 1 # pylint: disable=invalid-name
+ if x[k] + 2 <= u:
+ break
+
+ if k == 0:
+ return
+
+ f = x[k] + 1 # pylint: disable=invalid-name
+ s_ = s[k - 1] # pylint: disable=invalid-name
+ while k < m:
+ x[k] = f
+ s_ += f # pylint: disable=invalid-name
+ s[k] = s_
+ k += 1 # pylint: disable=invalid-name
+
+ x[m] = n - s[m - 1]
+
+
+def all_integer_partitioning_generator(
+ n, k, non_zero, max_count
+): # pylint: disable=invalid-name
+ """
+ Generate all integer partitioning of n to k parts (including 0 parts)
+
+ Args:
+ n (int): the integer to partition
+ k (int): the maximum number of parts
+ non_zero (bool): True if all parts must be greater than zero, False otherwise
+ max_count (int): the maximum value a part can take
+
+ Yields:
+ list: the list of parts
+ """
+ if n == 0:
+ yield [0] * k
+ else:
+ lower_bound = min(k, n) if non_zero else 1
+ upper_bound = min(k, n)
+ for m in range(lower_bound, upper_bound + 1): # pylint: disable=invalid-name
+ for positives in integer_partitioning_generator(n, m):
+ if all(pos <= max_count for pos in positives):
+ yield [0] * (k - m) + positives
+
+
+def not_enough_diverse_folds(p_values, n_values):
+ """
+ Checks if there are enough folds with positive and negative samples
+
+ Args:
+ p_values (list): the list of counts of positives
+ n_values (list): the list of counts of negatives
+
+ Returns:
+ bool: True, if the configuration is incorrect, False otherwise
+ """
+
+ return len(p_values) > 1 and (
+ sum(p_tmp > 0 for p_tmp in p_values) < 2
+ or sum(n_tmp > 0 for n_tmp in n_values) < 2
+ )
+
+
+def determine_min_max_p(
+ *, p, n, k_a, k_b, c_a, p_non_zero, n_non_zero
+): # pylint: disable=too-many-locals
+ """
+ Determines the minimum and maximum number of positives that can appear in folds
+ of type A
+
+ Args:
+ p (int): the total number of positives
+ n (int): the total number of negatives
+ k_a (int): the number of folds of type A
+ k_b (int): the number of folds of type B
+ c_a (int): the count of elements in folds of type A
+ p_non_zero (bool): whether all p should be non-zero
+ n_non_zero (bool): whether all n should be non-zero
+
+ Returns:
+ int, int: the minimum and maximum number of positives in all folds of
+ type A
+ """
+
+ total_a = k_a * c_a
+
+ min_p_a = max(p_non_zero * k_a, total_a - (n - n_non_zero * k_b))
+ max_p_a = min(p - p_non_zero * k_b, total_a - n_non_zero * k_a)
+
+ return min_p_a, max_p_a
+
+
+def fold_partitioning_generator(
+ *, p, n, k, p_non_zero=True, n_non_zero=True, p_min=-1
+): # pylint: disable=invalid-name,too-many-locals
+ """
+ Generates the fold partitioning
+
+ Args:
+ p (int): the number of positives
+ n (int): the number of negatives
+ k (int): the number of folds
+ p_zero (bool): whether any p can be zero
+ n_zero (bool): whether any n can be zero
+
+ Yields:
+ list, list: the list of positive and negative counts in folds
+ """
+ k_div = (p + n) // k
+ k_mod = (p + n) % k
+
+ k_b = k - k_mod
+ k_a = k_mod
+ c_a = k_div + 1
+ c_b = k_div
+
+ min_p_a, max_p_a = determine_min_max_p(
+ p=p,
+ n=n,
+ k_a=k_a,
+ k_b=k_b,
+ c_a=c_a,
+ p_non_zero=p_non_zero,
+ n_non_zero=n_non_zero,
+ )
+
+ for p_a in range(min_p_a, max_p_a + 1):
+ p_b = p - p_a
+
+ for ps_a in all_integer_partitioning_generator(
+ p_a, k_a, p_non_zero, c_a - n_non_zero
+ ):
+ if any(p_tmp < p_min for p_tmp in ps_a):
+ continue
+
+ ns_a = [c_a - tmp for tmp in ps_a]
+
+ for ps_b in all_integer_partitioning_generator(
+ p_b, k_b, p_non_zero, c_b - n_non_zero
+ ):
+ if any(p_tmp < p_min for p_tmp in ps_b):
+ continue
+
+ ns_b = [c_b - tmp for tmp in ps_b]
+
+ ps_all = ps_a + ps_b
+ ns_all = ns_a + ns_b
+
+ if not_enough_diverse_folds(ps_all, ns_all):
+ continue
+
+ yield ps_all, ns_all
+
+
+def _check_specification_and_determine_p_n(dataset: dict, folding: dict) -> (int, int):
+ """
+ Checking if the dataset specification is correct and determining the p and n values
+
+ Args:
+ dataset (dict): the dataset specification
+ folding (dict): the folding specification
+
+ Returns:
+ int, int: the number of positives and negatives
+
+ Raises:
+ ValueError: if the specification is not suitable
+ """
+ if folding.get("folds") is not None:
+ raise ValueError('do not specify the "folds" key for the generation of folds')
+ if (dataset.get("p") is not None and dataset.get("n") is None) or (
+ dataset.get("p") is None and dataset.get("n") is not None
+ ):
+ raise ValueError('either specify both "p" and "n" or None of them')
+ if dataset.get("dataset_name") is not None and dataset.get("p") is not None:
+ raise ValueError('either specify "dataset_name" or "p" and "n"')
+
+ p = (
+ dataset_statistics[dataset["dataset_name"]]["p"]
+ if dataset.get("dataset_name") is not None
+ else dataset["p"]
+ )
+ n = (
+ dataset_statistics[dataset["dataset_name"]]["n"]
+ if dataset.get("dataset_name") is not None
+ else dataset["n"]
+ )
+
+ return p, n
+
+
+def kfolds_generator(evaluation: dict, available_scores: list, repeat_idx=0):
+ """
+ Generates the fold configurations
+
+ Args:
+ evaluation (dict): the evaluation to generate the configurations for
+ available_scores (list): the list of available scores
+ repeat_idx (int): the index of the repeat
+
+ Returns:
+ Generator: the generator
+
+ Yields:
+ list(dict): the list of fold specifications
+ """
+ p, n = _check_specification_and_determine_p_n(
+ evaluation.get("dataset"), evaluation.get("folding")
+ )
+
+ p_zero = False
+ n_zero = False
+
+ if "sens" not in available_scores and "bacc" not in available_scores:
+ p_zero = True
+ logger.info(
+ "sens and bacc not among the reported scores, p=0 folds are also considered"
+ )
+ if "spec" not in available_scores and "bacc" not in available_scores:
+ n_zero = True
+ logger.info(
+ "spec and bacc not among the reported scores, n=0 folds are also considered"
+ )
+
+ if evaluation["dataset"].get("dataset_name") is not None:
+ evaluation["dataset"][
+ "identifier"
+ ] = f'{evaluation["dataset"]["dataset_name"]}_{random_identifier(3)}'
+ else:
+ evaluation["dataset"]["identifier"] = random_identifier(6)
+
+ for jdx, (p_vals, n_vals) in enumerate(
+ fold_partitioning_generator(
+ p=p,
+ n=n,
+ k=evaluation["folding"].get("n_folds", 1),
+ p_non_zero=not p_zero,
+ n_non_zero=not n_zero,
+ )
+ ):
+ yield [
+ {
+ "p": p_,
+ "n": n_,
+ "identifier": f"{evaluation['dataset']['identifier']}_f{idx}_k{jdx}_r{repeat_idx}",
+ }
+ for idx, (p_, n_) in enumerate(zip(p_vals, n_vals))
+ ]
+
+
+def repeated_kfolds_generator(evaluation: dict, available_scores: list):
+ """
+ Generates the evaluation variations
+
+ Args:
+ evaluation (dict): the evaluation to generate the configurations for
+ available_scores (list): the list of available scores
+
+ Returns:
+ Generator: the generator
+
+ Yields:
+ dict: one evaluation
+ """
+ n_repeats = evaluation["folding"].get("n_repeats", 1)
+ generators = [
+ kfolds_generator(evaluation, available_scores, idx) for idx in range(n_repeats)
+ ]
+
+ if n_repeats > 1:
+ for folds in itertools.product(*generators):
+ yield {
+ "dataset": copy.deepcopy(evaluation["dataset"]),
+ "folding": {
+ "folds": [fold for fold_list in folds for fold in fold_list]
+ },
+ "fold_score_bounds": copy.deepcopy(evaluation.get("fold_score_bounds")),
+ "aggregation": evaluation.get("aggregation"),
+ }
+ else:
+ for fold_list in generators[0]:
+ yield {
+ "dataset": copy.deepcopy(evaluation["dataset"]),
+ "folding": {"folds": fold_list},
+ "fold_score_bounds": copy.deepcopy(evaluation.get("fold_score_bounds")),
+ "aggregation": evaluation.get("aggregation"),
+ }
+
+
+def experiment_kfolds_generator(experiment: dict, available_scores: list):
+ """
+ Generates the experiment variations
+
+ Args:
+ experiment (dict): the experiment to generate the configurations for
+ available_scores (list): the list of available scores
+
+ Returns:
+ Generator: the generator
+
+ Yields:
+ dict: one experiment
+ """
+ generators = [
+ repeated_kfolds_generator(evaluation, available_scores)
+ for evaluation in experiment["evaluations"]
+ ]
+ for evaluations in itertools.product(*generators):
+ yield {
+ "evaluations": list(evaluations),
+ "dataset_score_bounds": experiment.get("dataset_score_bounds"),
+ "aggregation": experiment["aggregation"],
+ }
diff --git a/mlscorecheck/aggregated/_folding.py b/mlscorecheck/aggregated/_folding.py
index ab0418a..2323542 100644
--- a/mlscorecheck/aggregated/_folding.py
+++ b/mlscorecheck/aggregated/_folding.py
@@ -7,17 +7,21 @@
from ._dataset import Dataset
-__all__ = ['Folding']
+__all__ = ["Folding"]
+
class Folding:
"""
Abstract representation of a folding
"""
- def __init__(self,
- n_folds: int = None,
- n_repeats: int = None,
- folds: list = None,
- strategy: str = None):
+
+ def __init__(
+ self,
+ n_folds: int = None,
+ n_repeats: int = None,
+ folds: list = None,
+ strategy: str = None,
+ ):
"""
Constructor of the folding
@@ -28,11 +32,11 @@ def __init__(self,
strategy (str): the folding strategy ('stratified_sklearn')
"""
if (n_folds is not None) and (folds is not None):
- raise ValueError('specify either n_folds,n_repeats,strategy or folds')
+ raise ValueError("specify either n_folds,n_repeats,strategy or folds")
if (n_folds is None) and (n_repeats is None) and (folds is None):
- raise ValueError('specify either n_folds,strategy or folds')
+ raise ValueError("specify either n_folds,strategy or folds")
if ((folds is None) and (strategy is None)) and (n_folds > 1):
- raise ValueError('specify strategy if folds are not set explicitly')
+ raise ValueError("specify strategy if folds are not set explicitly")
self.n_folds = n_folds
self.n_repeats = n_repeats if n_repeats is not None else 1
@@ -46,10 +50,12 @@ def to_dict(self) -> dict:
Returns:
dict: the representation of the folding
"""
- return {'n_folds': self.n_folds,
- 'n_repeats': self.n_repeats,
- 'folds': self.folds,
- 'strategy': self.strategy}
+ return {
+ "n_folds": self.n_folds,
+ "n_repeats": self.n_repeats,
+ "folds": self.folds,
+ "strategy": self.strategy,
+ }
def generate_folds(self, dataset: Dataset, aggregation: str) -> list:
"""
@@ -69,34 +75,44 @@ def generate_folds(self, dataset: Dataset, aggregation: str) -> list:
p = 0
n = 0
for fold in self.folds:
- p += fold['p']
- n += fold['n']
+ p += fold["p"]
+ n += fold["n"]
- term_a = ((dataset.p != p) and (p % dataset.p != 0))
- term_b = ((dataset.n != n) and (n % dataset.n != 0))
- term_c = (dataset.p > 0 and dataset.n > 0 and (p // dataset.p != n // dataset.n))
+ term_a = (dataset.p != p) and (p % dataset.p != 0)
+ term_b = (dataset.n != n) and (n % dataset.n != 0)
+ term_c = (
+ dataset.p > 0 and dataset.n > 0 and (p // dataset.p != n // dataset.n)
+ )
if term_a or term_b or term_c:
- raise ValueError("The total p and n figures in the folds are not "\
- "multiples of the dataset's p and n figures "\
- f"{p}, {dataset.p}, {n}, {dataset.n}")
+ raise ValueError(
+ "The total p and n figures in the folds are not "
+ "multiples of the dataset's p and n figures "
+ f"{p}, {dataset.p}, {n}, {dataset.n}"
+ )
return [Fold(**fold) for fold in self.folds]
p, n = dataset.p, dataset.n
- if aggregation == 'som':
- return [Fold(p=p * self.n_repeats,
- n=n * self.n_repeats,
- identifier=dataset.identifier)]
-
- if aggregation == 'mos':
- folds = _create_folds(p=p,
- n=n,
- n_folds=self.n_folds,
- n_repeats=self.n_repeats,
- folding=self.strategy,
- identifier=dataset.identifier)
+ if aggregation == "som":
+ return [
+ Fold(
+ p=p * self.n_repeats,
+ n=n * self.n_repeats,
+ identifier=dataset.identifier,
+ )
+ ]
+
+ if aggregation == "mos":
+ folds = _create_folds(
+ p=p,
+ n=n,
+ n_folds=self.n_folds,
+ n_repeats=self.n_repeats,
+ folding=self.strategy,
+ identifier=dataset.identifier,
+ )
return [Fold(**fold) for fold in folds]
- raise ValueError(f'aggregation mode {aggregation} is not supported')
+ raise ValueError(f"aggregation mode {aggregation} is not supported")
diff --git a/mlscorecheck/aggregated/_folding_utils.py b/mlscorecheck/aggregated/_folding_utils.py
index d38f53b..8489f24 100644
--- a/mlscorecheck/aggregated/_folding_utils.py
+++ b/mlscorecheck/aggregated/_folding_utils.py
@@ -3,27 +3,24 @@
"""
import copy
-import itertools
-from ..core import logger
-from ..experiments import dataset_statistics
+import numpy as np
+
+from sklearn.model_selection import StratifiedKFold
+
from ._utils import random_identifier
-__all__ = ['stratified_configurations_sklearn',
- 'determine_fold_configurations',
- '_create_folds',
- 'integer_partitioning_generator',
- 'all_integer_partitioning_generator',
- 'fold_partitioning_generator',
- '_check_specification_and_determine_p_n',
- 'determine_min_max_p',
- 'kfolds_generator',
- 'repeated_kfolds_generator',
- 'experiment_kfolds_generator']
-
-def stratified_configurations_sklearn(p: int,
- n: int,
- n_splits: int) -> list:
+__all__ = [
+ "stratified_configurations_sklearn",
+ "determine_fold_configurations",
+ "_create_folds",
+ "multiclass_stratified_folds",
+ "transform_multiclass_fold_to_binary",
+ "create_folds_multiclass",
+]
+
+
+def stratified_configurations_sklearn(p: int, n: int, n_splits: int) -> list:
"""
The sklearn stratification strategy
@@ -54,11 +51,10 @@ def stratified_configurations_sklearn(p: int,
return results
-def determine_fold_configurations(p: int,
- n: int,
- n_folds: int,
- n_repeats: int,
- folding: str = 'stratified_sklearn') -> list:
+
+def determine_fold_configurations(
+ p: int, n: int, n_folds: int, n_repeats: int, folding: str = "stratified_sklearn"
+) -> list:
"""
Determine fold configurations according to a folding
@@ -75,21 +71,24 @@ def determine_fold_configurations(p: int,
Raises:
ValueError: if the folding is not supported
"""
- if folding != 'stratified_sklearn':
- raise ValueError(f'folding strategy {folding} is not supported yet')
+ if folding != "stratified_sklearn":
+ raise ValueError(f"folding strategy {folding} is not supported yet")
configurations = stratified_configurations_sklearn(p=p, n=n, n_splits=n_folds)
- configurations = [{'n': conf[0], 'p': conf[1]} for conf in configurations]
+ configurations = [{"n": conf[0], "p": conf[1]} for conf in configurations]
return [{**item} for item in configurations for _ in range(n_repeats)]
-def _create_folds(p: int,
- n: int,
- *,
- n_folds: int = None,
- n_repeats: int = None,
- folding: str = None,
- score_bounds: dict = None,
- identifier: str = None) -> list:
+
+def _create_folds(
+ p: int,
+ n: int,
+ *,
+ n_folds: int = None,
+ n_repeats: int = None,
+ folding: str = None,
+ score_bounds: dict = None,
+ identifier: str = None,
+) -> list:
"""
Given a dataset, adds folds to it
@@ -110,22 +109,21 @@ def _create_folds(p: int,
"""
if n_folds == 1:
- folds = [{'p': p,
- 'n': n,
- 'identifier': f'{identifier}_0_r{idx}'} for idx in range(n_repeats)]
+ folds = [
+ {"p": p, "n": n, "identifier": f"{identifier}_0_r{idx}"}
+ for idx in range(n_repeats)
+ ]
elif folding is None:
- folds = [{'p': p * n_repeats, 'n': n * n_repeats, 'identifier': f'{identifier}_0'}]
+ folds = [
+ {"p": p * n_repeats, "n": n * n_repeats, "identifier": f"{identifier}_0"}
+ ]
else:
- folds = determine_fold_configurations(p,
- n,
- n_folds,
- n_repeats,
- folding)
+ folds = determine_fold_configurations(p, n, n_folds, n_repeats, folding)
n_fold = 0
n_repeat = 0
for fold in folds:
- fold['identifier'] = f'{identifier}_{n_repeat}_{n_fold}'
+ fold["identifier"] = f"{identifier}_{n_repeat}_{n_fold}"
n_fold += 1
if n_fold % n_folds == 0:
n_fold = 0
@@ -133,304 +131,80 @@ def _create_folds(p: int,
for fold in folds:
if score_bounds is not None:
- fold['score_bounds'] = {**score_bounds}
+ fold["score_bounds"] = {**score_bounds}
return folds
-def integer_partitioning_generator(n: int, m: int): # pylint: disable=invalid-name
- """
- Integer partitioning generator
-
- Integer partitioning algorithm implemented following the algorithm on page 343 in
- https://doi.org/10.1007/978-3-642-14764-7
-
- Args:
- n (int): the integer to partition
- m (int): the number of partitions
-
- Yields:
- list: the next configuration
- """
- x = [0] * (m+1) # pylint: disable=invalid-name
- s = [0] * (m+1) # pylint: disable=invalid-name
-
- for k in range(1, m): # pylint: disable=invalid-name
- x[k] = 1
-
- x[m] = n - m + 1
-
- for k in range(1, m + 1): # pylint: disable=invalid-name
- s[k] = x[k] + s[k-1]
-
- while True:
- yield x[1:]
-
- u = x[m] # pylint: disable=invalid-name
- k = m # pylint: disable=invalid-name
- while k > 0:
- k -= 1 # pylint: disable=invalid-name
- if x[k] + 2 <= u:
- break
-
- if k == 0:
- return
-
- f = x[k] + 1 # pylint: disable=invalid-name
- s_ = s[k-1] # pylint: disable=invalid-name
- while k < m:
- x[k] = f
- s_ += f # pylint: disable=invalid-name
- s[k] = s_
- k += 1 # pylint: disable=invalid-name
- x[m] = n - s[m-1]
-
-def all_integer_partitioning_generator(n, k, non_zero, max_count): # pylint: disable=invalid-name
- """
- Generate all integer partitioning of n to k parts (including 0 parts)
-
- Args:
- n (int): the integer to partition
- k (int): the maximum number of parts
- non_zero (bool): True if all parts must be greater than zero, False otherwise
- max_count (int): the maximum value a part can take
-
- Yields:
- list: the list of parts
+def multiclass_stratified_folds(dataset: dict, n_folds: int) -> list:
"""
- if n == 0:
- yield [0] * k
- else:
- lower_bound = min(k, n) if non_zero else 1
- upper_bound = min(k, n)
- for m in range(lower_bound, upper_bound + 1):# pylint: disable=invalid-name
- for positives in integer_partitioning_generator(n, m):
- if all(pos <= max_count for pos in positives):
- yield [0] * (k - m) + positives
-
-def not_enough_diverse_folds(p_values, n_values):
- """
- Checks if there are enough folds with positive and negative samples
+ Generating the folds for an sklearn stratified multiclass setup
Args:
- p_values (list): the list of counts of positives
- n_values (list): the list of counts of negatives
-
- Returns:
- bool: True, if the configuration is incorrect, False otherwise
- """
-
- return (len(p_values) > 1 and (sum(p_tmp > 0 for p_tmp in p_values) < 2
- or sum(n_tmp > 0 for n_tmp in n_values) < 2))
-
-def determine_min_max_p(*, p, n, k_a, k_b, c_a, p_non_zero, n_non_zero): # pylint: disable=too-many-locals
- """
- Determines the minimum and maximum number of positives that can appear in folds
- of type A
-
- Args:
- p (int): the total number of positives
- n (int): the total number of negatives
- k_a (int): the number of folds of type A
- k_b (int): the number of folds of type B
- c_a (int): the count of elements in folds of type A
- p_non_zero (bool): whether all p should be non-zero
- n_non_zero (bool): whether all n should be non-zero
+ dataset (dict): the specification of the dataset
+ n_folds (int): the number of folds
Returns:
- int, int: the minimum and maximum number of positives in all folds of
- type A
+ list(dict): the list of fold specifications
"""
+ folds = []
+ labels = np.hstack([np.repeat(key, value) for key, value in dataset.items()])
+ for _, test in StratifiedKFold(n_splits=n_folds).split(
+ labels.reshape(-1, 1), labels, labels
+ ):
+ folds.append(dict(enumerate(np.bincount(labels[test]))))
- total_a = k_a * c_a
-
- min_p_a = max(p_non_zero*k_a, total_a - (n - n_non_zero*k_b))
- max_p_a = min(p - p_non_zero*k_b, total_a - n_non_zero*k_a)
+ return folds
- return min_p_a, max_p_a
-def fold_partitioning_generator(*, p, n, k, p_non_zero=True, n_non_zero=True, p_min=-1): #pylint: disable=invalid-name,too-many-locals
+def transform_multiclass_fold_to_binary(fold: dict) -> list:
"""
- Generates the fold partitioning
+ Transforms a multiclass fold specification to a list of binary folds
Args:
- p (int): the number of positives
- n (int): the number of negatives
- k (int): the number of folds
- p_zero (bool): whether any p can be zero
- n_zero (bool): whether any n can be zero
+ fold (dict): a multiclass fold specification
- Yields:
- list, list: the list of positive and negative counts in folds
+ Returns:
+ list(dict): the list of binary folds
"""
- k_div = (p + n) // k
- k_mod = (p + n) % k
-
- k_b = k - k_mod
- k_a = k_mod
- c_a = k_div + 1
- c_b = k_div
-
- min_p_a, max_p_a = determine_min_max_p(p=p,
- n=n,
- k_a=k_a,
- k_b=k_b,
- c_a=c_a,
- p_non_zero=p_non_zero,
- n_non_zero=n_non_zero)
-
- for p_a in range(min_p_a, max_p_a + 1):
- p_b = p - p_a
-
- for ps_a in all_integer_partitioning_generator(p_a, k_a, p_non_zero, c_a - n_non_zero):
-
- if any(p_tmp < p_min for p_tmp in ps_a):
- continue
-
- ns_a = [c_a - tmp for tmp in ps_a]
-
- for ps_b in all_integer_partitioning_generator(p_b, k_b, p_non_zero, c_b - n_non_zero):
+ n_total = sum(fold.values())
+ folds = [{"p": value, "n": n_total - value} for value in fold.values()]
+ identifier = fold.get("identifier", random_identifier(4))
- if any(p_tmp < p_min for p_tmp in ps_b):
- continue
+ for idx, fold_ in enumerate(folds):
+ fold_["identifier"] = f"{identifier}_{idx}"
- ns_b = [c_b - tmp for tmp in ps_b]
-
- ps_all = ps_a + ps_b
- ns_all = ns_a + ns_b
-
- if not_enough_diverse_folds(ps_all, ns_all):
- continue
+ return folds
- yield ps_all, ns_all
-def _check_specification_and_determine_p_n(dataset: dict, folding: dict) -> (int, int):
+def create_folds_multiclass(dataset: dict, folding: dict) -> list:
"""
- Checking if the dataset specification is correct and determining the p and n values
+ Create the folds for the multiclass setup
Args:
dataset (dict): the dataset specification
folding (dict): the folding specification
Returns:
- int, int: the number of positives and negatives
-
- Raises:
- ValueError: if the specification is not suitable
- """
- if folding.get('folds') is not None:
- raise ValueError('do not specify the "folds" key for the generation of folds')
- if (dataset.get('p') is not None and dataset.get('n') is None)\
- or (dataset.get('p') is None and dataset.get('n') is not None):
- raise ValueError('either specify both "p" and "n" or None of them')
- if dataset.get('dataset_name') is not None and dataset.get('p') is not None:
- raise ValueError('either specify "dataset_name" or "p" and "n"')
-
- p = (dataset_statistics[dataset['dataset_name']]["p"]
- if dataset.get('dataset_name') is not None else dataset["p"])
- n = (dataset_statistics[dataset['dataset_name']]["n"]
- if dataset.get('dataset_name') is not None else dataset["n"])
-
- return p, n
-
-def kfolds_generator(evaluation: dict,
- available_scores: list,
- repeat_idx=0):
- """
- Generates the fold configurations
-
- Args:
- evaluation (dict): the evaluation to generate the configurations for
- available_scores (list): the list of available scores
- repeat_idx (int): the index of the repeat
-
- Returns:
- Generator: the generator
-
- Yields:
list(dict): the list of fold specifications
"""
- p, n = _check_specification_and_determine_p_n(evaluation.get('dataset'),
- evaluation.get('folding'))
-
- p_zero = False
- n_zero = False
-
- if 'sens' not in available_scores and 'bacc' not in available_scores:
- p_zero = True
- logger.info('sens and bacc not among the reported scores, p=0 folds are also considered')
- if 'spec' not in available_scores and 'bacc' not in available_scores:
- n_zero = True
- logger.info('spec and bacc not among the reported scores, n=0 folds are also considered')
-
- if evaluation['dataset'].get('dataset_name') is not None:
- evaluation['dataset']['identifier'] = \
- f'{evaluation["dataset"]["dataset_name"]}_{random_identifier(3)}'
- else:
- evaluation['dataset']['identifier'] = random_identifier(6)
-
- for jdx, (p_vals, n_vals) in enumerate(fold_partitioning_generator(p=p,
- n=n,
- k=evaluation['folding'].get('n_folds', 1),
- p_non_zero=not p_zero,
- n_non_zero=not n_zero)):
- yield [{'p': p_,
- 'n': n_,
- 'identifier': f"{evaluation['dataset']['identifier']}_f{idx}_k{jdx}_r{repeat_idx}"}
- for idx, (p_, n_) in enumerate(zip(p_vals, n_vals))]
-
-def repeated_kfolds_generator(evaluation: dict,
- available_scores: list):
- """
- Generates the evaluation variations
-
- Args:
- evaluation (dict): the evaluation to generate the configurations for
- available_scores (list): the list of available scores
-
- Returns:
- Generator: the generator
-
- Yields:
- dict: one evaluation
- """
- n_repeats = evaluation['folding'].get('n_repeats', 1)
- generators = [kfolds_generator(evaluation, available_scores, idx) for idx in range(n_repeats)]
-
- if n_repeats > 1:
-
- for folds in itertools.product(*generators):
- yield {'dataset': copy.deepcopy(evaluation['dataset']),
- 'folding': {
- 'folds': [fold for fold_list in folds for fold in fold_list]},
- 'fold_score_bounds': copy.deepcopy(evaluation.get('fold_score_bounds')),
- 'aggregation': evaluation.get('aggregation')}
+ if folding.get("folds") is not None and (
+ folding.get("n_repeats") is not None
+ or folding.get("strategy") is not None
+ or folding.get("n_folds") is not None
+ ):
+ raise ValueError("either specify the folds or the folding strategy")
+
+ if "folds" in folding:
+ return folding["folds"]
+ if folding.get("strategy") == "stratified_sklearn":
+ folds = multiclass_stratified_folds(dataset, folding.get("n_folds", 1))
else:
- for fold_list in generators[0]:
- yield {'dataset': copy.deepcopy(evaluation['dataset']),
- 'folding': {
- 'folds': fold_list},
- 'fold_score_bounds': copy.deepcopy(evaluation.get('fold_score_bounds')),
- 'aggregation': evaluation.get('aggregation')}
-
-def experiment_kfolds_generator(experiment: dict,
- available_scores: list):
- """
- Generates the experiment variations
+ folds = [dataset]
- Args:
- experiment (dict): the experiment to generate the configurations for
- available_scores (list): the list of available scores
+ n_repeats = folding.get("n_repeats", 1)
- Returns:
- Generator: the generator
+ folds = folds * n_repeats
+ folds = [copy.deepcopy(fold) for fold in folds]
- Yields:
- dict: one experiment
- """
- generators = [repeated_kfolds_generator(evaluation, available_scores)
- for evaluation in experiment['evaluations']]
- for evaluations in itertools.product(*generators):
- yield {'evaluations': list(evaluations),
- 'dataset_score_bounds': experiment.get('dataset_score_bounds'),
- 'aggregation': experiment['aggregation']}
+ return folds
diff --git a/mlscorecheck/aggregated/_generate_problems.py b/mlscorecheck/aggregated/_generate_problems.py
index 9ff93df..1cc3833 100644
--- a/mlscorecheck/aggregated/_generate_problems.py
+++ b/mlscorecheck/aggregated/_generate_problems.py
@@ -5,27 +5,35 @@
import numpy as np
-from ..core import (init_random_state, dict_minmax, dict_mean, round_scores)
+from ..core import init_random_state, dict_minmax, dict_mean, round_scores
from ..experiments import dataset_statistics
-from ..individual import calculate_scores, calculate_scores_for_lp
+from ..scores import calculate_scores, calculate_scores_for_lp
+
+from ..individual import sample_multiclass_dataset, generate_multiclass_dataset
+from ._folding_utils import multiclass_stratified_folds, create_folds_multiclass
+from ..scores import calculate_multiclass_scores
+
from ._dataset import Dataset
from ._folding import Folding
from ._evaluation import Evaluation
from ._experiment import Experiment
-__all__ = ['generate_dataset',
- 'generate_folding',
- 'generate_evaluation',
- 'generate_experiment',
- 'get_fold_score_bounds',
- 'get_dataset_score_bounds',
- 'generate_scores_for_testsets']
-
-def generate_dataset(max_p: int = 500,
- max_n: int = 500,
- random_state = None,
- no_name: bool = False) -> dict:
+__all__ = [
+ "generate_dataset",
+ "generate_folding",
+ "generate_evaluation",
+ "generate_experiment",
+ "get_fold_score_bounds",
+ "get_dataset_score_bounds",
+ "generate_scores_for_testsets",
+ "generate_dataset_folding_multiclass",
+]
+
+
+def generate_dataset(
+ max_p: int = 500, max_n: int = 500, random_state=None, no_name: bool = False
+) -> dict:
"""
Generate a random dataset specification
@@ -42,19 +50,22 @@ def generate_dataset(max_p: int = 500,
random_state = init_random_state(random_state)
if random_state.randint(2) == 0 or no_name:
- p = random_state.randint(1, max_p+1)
- n = random_state.randint(1, max_n+1)
- return {'p': p,
- 'n': n}
-
- return {'dataset_name': random_state.choice(list(dataset_statistics.keys()))}
-
-def generate_folding(*, dataset: dict,
- max_folds: int = 10,
- max_repeats: int = 5,
- strategies: list = None,
- random_state = None,
- no_folds: bool = False) -> dict:
+ p = random_state.randint(1, max_p + 1)
+ n = random_state.randint(1, max_n + 1)
+ return {"p": p, "n": n}
+
+ return {"dataset_name": random_state.choice(list(dataset_statistics.keys()))}
+
+
+def generate_folding(
+ *,
+ dataset: dict,
+ max_folds: int = 10,
+ max_repeats: int = 5,
+ strategies: list = None,
+ random_state=None,
+ no_folds: bool = False,
+) -> dict:
"""
Generate a random folding specification for a dataset
@@ -71,40 +82,42 @@ def generate_folding(*, dataset: dict,
"""
random_state = init_random_state(random_state)
- strategies = ['stratified_sklearn'] if strategies is None else strategies
+ strategies = ["stratified_sklearn"] if strategies is None else strategies
dataset = Dataset(**dataset)
p, n = dataset.p, dataset.n
max_folds = min(p, n, max_folds)
- n_folds = random_state.randint(1, max_folds+1)
- n_repeats = random_state.randint(1, max_repeats+1)
+ n_folds = random_state.randint(1, max_folds + 1)
+ n_repeats = random_state.randint(1, max_repeats + 1)
strategy = random_state.choice(strategies)
if random_state.randint(2) == 0 or no_folds:
- return {'n_folds': n_folds,
- 'n_repeats': n_repeats,
- 'strategy': strategy}
-
- folding = Folding(n_folds=n_folds,
- n_repeats=n_repeats,
- strategy=strategy)
-
- return {'folds': [fold.to_dict() for fold in folding.generate_folds(dataset, 'mos')]}
-
-def generate_evaluation(*, max_p: int = 500, # pylint: disable=too-many-locals
- max_n: int = 500,
- max_folds: int = 10,
- max_repeats: int = 5,
- strategies: list = None,
- feasible_fold_score_bounds: bool = None,
- aggregation: str = None,
- random_state = None,
- return_scores: bool = False,
- rounding_decimals: int = None,
- no_name: bool = False,
- no_folds: bool = False,
- score_subset: list = None) -> dict:
+ return {"n_folds": n_folds, "n_repeats": n_repeats, "strategy": strategy}
+
+ folding = Folding(n_folds=n_folds, n_repeats=n_repeats, strategy=strategy)
+
+ return {
+ "folds": [fold.to_dict() for fold in folding.generate_folds(dataset, "mos")]
+ }
+
+
+def generate_evaluation( # pylint: disable=too-many-locals
+ *,
+ max_p: int = 500,
+ max_n: int = 500,
+ max_folds: int = 10,
+ max_repeats: int = 5,
+ strategies: list = None,
+ feasible_fold_score_bounds: bool = None,
+ aggregation: str = None,
+ random_state=None,
+ return_scores: bool = False,
+ rounding_decimals: int = None,
+ no_name: bool = False,
+ no_folds: bool = False,
+ score_subset: list = None,
+) -> dict:
"""
Generate a random evaluation specification
@@ -131,46 +144,55 @@ def generate_evaluation(*, max_p: int = 500, # pylint: disable=too-many-locals
"""
random_state = init_random_state(random_state)
- result = {'dataset': generate_dataset(max_p=max_p,
- max_n=max_n,
- random_state=random_state,
- no_name=no_name)}
- result['folding'] = generate_folding(dataset=result['dataset'],
- max_folds=max_folds,
- max_repeats=max_repeats,
- strategies=strategies,
- random_state=random_state,
- no_folds=no_folds)
-
- aggregation = aggregation if aggregation is not None else random_state.choice(['som', 'mos'])
-
- evaluation = Evaluation(dataset=result['dataset'],
- folding=result['folding'],
- aggregation=aggregation).sample_figures(random_state)
-
- if aggregation == 'som':
- scores = calculate_scores(problem=evaluation.figures | {'beta_positive': 2,
- 'beta_negative': 2},
- rounding_decimals=rounding_decimals)
- scores['beta_positive'] = 2
- scores['beta_negative'] = 2
+ result = {
+ "dataset": generate_dataset(
+ max_p=max_p, max_n=max_n, random_state=random_state, no_name=no_name
+ )
+ }
+ result["folding"] = generate_folding(
+ dataset=result["dataset"],
+ max_folds=max_folds,
+ max_repeats=max_repeats,
+ strategies=strategies,
+ random_state=random_state,
+ no_folds=no_folds,
+ )
+
+ aggregation = (
+ aggregation if aggregation is not None else random_state.choice(["som", "mos"])
+ )
+
+ evaluation = Evaluation(
+ dataset=result["dataset"], folding=result["folding"], aggregation=aggregation
+ ).sample_figures(random_state)
+
+ if aggregation == "som":
+ scores = calculate_scores(
+ problem=evaluation.figures | {"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=rounding_decimals,
+ )
+ scores["beta_positive"] = 2
+ scores["beta_negative"] = 2
else:
- scores = evaluation.calculate_scores(rounding_decimals,
- score_subset=score_subset)
+ scores = evaluation.calculate_scores(
+ rounding_decimals, score_subset=score_subset
+ )
if feasible_fold_score_bounds is None:
- result['fold_score_bounds'] = None
+ result["fold_score_bounds"] = None
else:
- result['fold_score_bounds'] = get_fold_score_bounds(evaluation,
- feasible_fold_score_bounds)
+ result["fold_score_bounds"] = get_fold_score_bounds(
+ evaluation, feasible_fold_score_bounds
+ )
- result['aggregation'] = aggregation
+ result["aggregation"] = aggregation
return (result, scores) if return_scores else result
-def get_fold_score_bounds(evaluation: Evaluation,
- feasible: bool = True,
- numerical_tolerance: float = 1*1e-1) -> dict:
+
+def get_fold_score_bounds(
+ evaluation: Evaluation, feasible: bool = True, numerical_tolerance: float = 1 * 1e-1
+) -> dict:
"""
Extract fold score bounds from an evaluation (sampled and scores computed)
@@ -185,8 +207,10 @@ def get_fold_score_bounds(evaluation: Evaluation,
score_bounds = dict_minmax([fold.scores for fold in evaluation.folds])
for key, value in score_bounds.items():
- score_bounds[key] = (max(0.0, value[0] - numerical_tolerance),
- min(1.0, value[1] + numerical_tolerance))
+ score_bounds[key] = (
+ max(0.0, value[0] - numerical_tolerance),
+ min(1.0, value[1] + numerical_tolerance),
+ )
if feasible:
return score_bounds
@@ -195,14 +219,18 @@ def get_fold_score_bounds(evaluation: Evaluation,
return score_bounds
-def generate_experiment(*, max_evaluations: int = 5,
- evaluation_params: dict = None,
- feasible_dataset_score_bounds: bool = None,
- aggregation: str = None,
- random_state = None,
- return_scores: bool = False,
- rounding_decimals: int = None,
- score_subset: list = None) -> dict:
+
+def generate_experiment(
+ *,
+ max_evaluations: int = 5,
+ evaluation_params: dict = None,
+ feasible_dataset_score_bounds: bool = None,
+ aggregation: str = None,
+ random_state=None,
+ return_scores: bool = False,
+ rounding_decimals: int = None,
+ score_subset: list = None,
+) -> dict:
"""
Generate a random experiment specification
@@ -227,47 +255,57 @@ def generate_experiment(*, max_evaluations: int = 5,
random_state = init_random_state(random_state)
- n_evaluations = random_state.randint(1, max_evaluations+1)
-
- aggregation = (aggregation if aggregation is not None
- else random_state.choice(['som', 'mos']))
-
- evaluations = [generate_evaluation(**evaluation_params,
- random_state=random_state)
- for _ in range(n_evaluations)]
-
- experiment = Experiment(evaluations=evaluations,
- aggregation=aggregation).sample_figures(random_state)
- if aggregation == 'som':
- scores = calculate_scores(problem=experiment.figures | {'beta_positive': 2,
- 'beta_negative': 2},
- rounding_decimals=rounding_decimals)
- scores['beta_positive'] = 2
- scores['beta_negative'] = 2
+ n_evaluations = random_state.randint(1, max_evaluations + 1)
+
+ aggregation = (
+ aggregation if aggregation is not None else random_state.choice(["som", "mos"])
+ )
+
+ evaluations = [
+ generate_evaluation(**evaluation_params, random_state=random_state)
+ for _ in range(n_evaluations)
+ ]
+
+ experiment = Experiment(
+ evaluations=evaluations, aggregation=aggregation
+ ).sample_figures(random_state)
+ if aggregation == "som":
+ scores = calculate_scores(
+ problem=experiment.figures | {"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=rounding_decimals,
+ )
+ scores["beta_positive"] = 2
+ scores["beta_negative"] = 2
else:
- scores = experiment.calculate_scores(rounding_decimals,
- score_subset=score_subset)
+ scores = experiment.calculate_scores(
+ rounding_decimals, score_subset=score_subset
+ )
- if evaluation_params.get('feasible_fold_score_bounds') is not None:
+ if evaluation_params.get("feasible_fold_score_bounds") is not None:
for idx, evaluation in enumerate(experiment.evaluations):
- evaluations[idx]['fold_score_bounds'] = \
- get_fold_score_bounds(evaluation,
- evaluation_params['feasible_fold_score_bounds'])
+ evaluations[idx]["fold_score_bounds"] = get_fold_score_bounds(
+ evaluation, evaluation_params["feasible_fold_score_bounds"]
+ )
if feasible_dataset_score_bounds is None:
score_bounds = None
else:
- score_bounds = get_dataset_score_bounds(experiment, feasible_dataset_score_bounds)
+ score_bounds = get_dataset_score_bounds(
+ experiment, feasible_dataset_score_bounds
+ )
- experiment = {'evaluations': evaluations,
- 'aggregation': aggregation,
- 'dataset_score_bounds': score_bounds}
+ experiment = {
+ "evaluations": evaluations,
+ "aggregation": aggregation,
+ "dataset_score_bounds": score_bounds,
+ }
return (experiment, scores) if return_scores else experiment
-def get_dataset_score_bounds(experiment: Experiment,
- feasible: bool = True,
- numerical_tolerance: float = 2*1e-2) -> dict:
+
+def get_dataset_score_bounds(
+ experiment: Experiment, feasible: bool = True, numerical_tolerance: float = 2 * 1e-2
+) -> dict:
"""
Extract fold score bounds from an experiment (sampled and scores computed)
@@ -279,10 +317,14 @@ def get_dataset_score_bounds(experiment: Experiment,
Returns:
dict(str,tuple(float,float)): the score bounds
"""
- score_bounds = dict_minmax([evaluation.scores for evaluation in experiment.evaluations])
+ score_bounds = dict_minmax(
+ [evaluation.scores for evaluation in experiment.evaluations]
+ )
for key, value in score_bounds.items():
- score_bounds[key] = (max(0.0, value[0] - numerical_tolerance),
- min(1.0, value[1] + numerical_tolerance))
+ score_bounds[key] = (
+ max(0.0, value[0] - numerical_tolerance),
+ min(1.0, value[1] + numerical_tolerance),
+ )
if feasible:
return score_bounds
@@ -291,11 +333,10 @@ def get_dataset_score_bounds(experiment: Experiment,
return score_bounds
-def generate_scores_for_testsets(testsets,
- rounding_decimals=None,
- subset=None,
- random_state=None,
- aggregation='mos'):
+
+def generate_scores_for_testsets(
+ testsets, rounding_decimals=None, subset=None, random_state=None, aggregation="mos"
+):
"""
Sample scores for testsets
@@ -312,22 +353,120 @@ def generate_scores_for_testsets(testsets,
if not isinstance(random_state, np.random.RandomState):
random_state = np.random.RandomState(random_state)
- subset = subset if subset is not None else ['acc', 'sens', 'spec']
+ subset = subset if subset is not None else ["acc", "sens", "spec"]
- testsets = [{'p': testset['p'], 'n': testset['n']} for testset in testsets]
+ testsets = [{"p": testset["p"], "n": testset["n"]} for testset in testsets]
for testset in testsets:
- testset['tp'] = random_state.randint(testset['p'] + 1)
- testset['tn'] = random_state.randint(testset['n'] + 1)
+ testset["tp"] = random_state.randint(testset["p"] + 1)
+ testset["tn"] = random_state.randint(testset["n"] + 1)
- if aggregation == 'mos':
+ if aggregation == "mos":
scores = [calculate_scores_for_lp(testset) for testset in testsets]
scores = round_scores(dict_mean(scores), rounding_decimals=rounding_decimals)
return {key: value for key, value in scores.items() if key in subset}
mean_figures = dict_mean(testsets)
- scores = calculate_scores(mean_figures | {'beta_positive': 2, 'beta_negative': 2},
- rounding_decimals=rounding_decimals,
- subset=subset)
+ scores = calculate_scores(
+ mean_figures | {"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=rounding_decimals,
+ subset=subset,
+ )
+
+ return scores | {"beta_positive": 2, "beta_negative": 2}
+
+
+def generate_dataset_folding_multiclass(
+ *,
+ random_state=None,
+ max_n_classes=5,
+ min_n_classes=3,
+ max_class_size=200,
+ min_class_size=10,
+ max_n_folds=5,
+ max_n_repeats=3,
+ average=None,
+ aggregation=None,
+ rounding_decimals=None,
+ subset=None,
+):
+ """
+ Generates a multiclass dataset and folding with scores
+
+ Args:
+ random_state (None|int|np.random.RandomState): the random state/seed to use
+ max_n_classes (int): the maximum number of classes
+ min_n_classes (int): the minimum number of classes
+ max_class_size (int): the maximum class size
+ min_class_size (int): the minimum class size
+ max_n_folds (int): the maximum number of folds
+ max_n_repeats (int): the maximum number of repeats
+ average (str): the type of averaging to use
+ aggregation (str): the type of aggregation to use
+ rounding_decimals (None|int): the number of decimals to round to
+ subset (None|list): the subset of scores
- return scores | {'beta_positive': 2, 'beta_negative': 2}
+ Returns:
+ tuple(dict,dict,dict): the dataset, folding and scores
+ """
+ if not isinstance(random_state, np.random.RandomState):
+ random_state = np.random.RandomState(random_state)
+
+ if not aggregation in {"mos", "som"}:
+ raise ValueError(f"Unknown aggregation: {aggregation}")
+
+ dataset = generate_multiclass_dataset(
+ random_state=random_state,
+ max_n_classes=max_n_classes,
+ min_n_classes=min_n_classes,
+ max_class_size=max_class_size,
+ min_class_size=min_class_size,
+ )
+
+ folding = {
+ "n_folds": min(
+ [min(list(dataset.values())), random_state.randint(2, max_n_folds)]
+ )
+ }
+
+ if random_state.randint(2) == 0:
+ folding = folding | {
+ "n_repeats": random_state.randint(1, max_n_repeats),
+ "strategy": "stratified_sklearn",
+ }
+ elif random_state.randint(2) == 0:
+ folding = {
+ "folds": multiclass_stratified_folds(
+ dataset=dataset, n_folds=folding["n_folds"]
+ )
+ }
+ else:
+ folding = {"n_folds": 1}
+
+ samples = [
+ sample_multiclass_dataset(dataset=fold, random_state=random_state)
+ for fold in create_folds_multiclass(dataset, folding)
+ ]
+
+ if aggregation == "mos":
+ scores = [
+ calculate_multiclass_scores(
+ sample,
+ average=average,
+ additional_symbols={"beta_positive": 2, "beta_negative": 2},
+ subset=subset,
+ )
+ for sample in samples
+ ]
+ scores = round_scores(dict_mean(scores), rounding_decimals=rounding_decimals)
+ return dataset, folding, scores
+
+ # if aggregation == 'som':
+ scores = calculate_multiclass_scores(
+ np.sum(np.array(samples), axis=0),
+ average=average,
+ additional_symbols={"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=rounding_decimals,
+ subset=subset,
+ )
+ return dataset, folding, scores | {"beta_positive": 2, "beta_negative": 2}
diff --git a/mlscorecheck/aggregated/_linear_programming.py b/mlscorecheck/aggregated/_linear_programming.py
index 228cefc..6978504 100644
--- a/mlscorecheck/aggregated/_linear_programming.py
+++ b/mlscorecheck/aggregated/_linear_programming.py
@@ -9,14 +9,12 @@
from ._utils import random_identifier
-__all__ = ['add_bounds',
- 'solve',
- 'create_lp_target']
-
-def add_bounds(lp_problem: pl.LpProblem,
- variables: dict,
- bounds: dict,
- label: str) -> pl.LpProblem:
+__all__ = ["add_bounds", "solve", "create_lp_target"]
+
+
+def add_bounds(
+ lp_problem: pl.LpProblem, variables: dict, bounds: dict, label: str
+) -> pl.LpProblem:
"""
Adding bounds to a linear program
@@ -36,26 +34,30 @@ def add_bounds(lp_problem: pl.LpProblem,
for variable in bounds:
if variable in variables:
if bounds[variable][0] is not None and not np.isnan(bounds[variable][0]):
- logger.info('%s: adding lower bound %s for %s',
- label, str(bounds[variable][0]), variable)
+ logger.info(
+ "%s: adding lower bound %s for %s",
+ label,
+ str(bounds[variable][0]),
+ variable,
+ )
lp_problem += bounds[variable][0] <= variables[variable]
else:
- logger.info('%s: No lower bound for variable %s', label, variable)
+ logger.info("%s: No lower bound for variable %s", label, variable)
if bounds[variable][1] is not None and not np.isnan(bounds[variable][1]):
- logger.info('%s: adding upper bound %s for %s', label,
- bounds[variable][1],
- variable)
+ logger.info(
+ "%s: adding upper bound %s for %s",
+ label,
+ bounds[variable][1],
+ variable,
+ )
lp_problem += variables[variable] <= bounds[variable][1]
else:
- logger.info('%s: No upper bound for variable %s', label, variable)
+ logger.info("%s: No upper bound for variable %s", label, variable)
return lp_problem
-def create_lp_target(obj,
- scores: dict,
- eps,
- lp_problem: pl.LpProblem) -> pl.LpProblem:
+def create_lp_target(obj, scores: dict, eps, lp_problem: pl.LpProblem) -> pl.LpProblem:
"""
Add the target and the score conditions to the linear programming problem
@@ -69,7 +71,7 @@ def create_lp_target(obj,
pl.LpProblem: the updated linear programming problem
"""
for key in scores:
- if key in ['acc', 'sens', 'spec', 'bacc'] and key in obj.scores:
+ if key in ["acc", "sens", "spec", "bacc"] and key in obj.scores:
lp_problem += obj.scores[key] >= scores[key] - eps[key]
lp_problem += obj.scores[key] <= scores[key] + eps[key]
@@ -78,10 +80,8 @@ def create_lp_target(obj,
return lp_problem
-def solve(obj,
- scores: dict,
- eps,
- solver = None) -> pl.LpProblem:
+
+def solve(obj, scores: dict, eps, solver=None) -> pl.LpProblem:
"""
Solving a problem.
@@ -95,9 +95,9 @@ def solve(obj,
pl.LpProblem: the solved linear programming problem
"""
if not isinstance(eps, dict):
- eps = {key: eps for key in ['acc', 'sens', 'spec', 'bacc']}
+ eps = {key: eps for key in ["acc", "sens", "spec", "bacc"]}
- lp_problem = pl.LpProblem(f'feasibility_{random_identifier(8)}')
+ lp_problem = pl.LpProblem(f"feasibility_{random_identifier(8)}")
lp_problem = obj.init_lp(lp_problem, scores)
diff --git a/mlscorecheck/aggregated/_utils.py b/mlscorecheck/aggregated/_utils.py
index 1b1660e..a736a84 100644
--- a/mlscorecheck/aggregated/_utils.py
+++ b/mlscorecheck/aggregated/_utils.py
@@ -7,12 +7,10 @@
import numpy as np
-__all__ = ['random_identifier',
- 'check_bounds',
- 'compare_scores',
- 'aggregated_scores']
+__all__ = ["random_identifier", "check_bounds", "compare_scores", "aggregated_scores"]
+
+aggregated_scores = ["acc", "sens", "spec", "bacc"]
-aggregated_scores = ['acc', 'sens', 'spec', 'bacc']
def random_identifier(length: int):
"""
@@ -25,11 +23,10 @@ def random_identifier(length: int):
str: the identifier
"""
letters = string.ascii_lowercase
- return ''.join(random.choice(letters) for _ in range(length))
+ return "".join(random.choice(letters) for _ in range(length))
+
-def check_bounds(scores: dict,
- bounds: dict = None,
- tolerance: float = 1e-5) -> bool:
+def check_bounds(scores: dict, bounds: dict = None, tolerance: float = 1e-5) -> bool:
"""
Checks the bounds for the scores
@@ -50,17 +47,16 @@ def check_bounds(scores: dict,
for key in bounds:
if key in scores:
if bounds[key][0] is not None and not np.isnan(bounds[key][0]):
- flag = flag and (bounds[key][0]-tolerance <= scores[key])
+ flag = flag and (bounds[key][0] - tolerance <= scores[key])
if bounds[key][1] is not None and not np.isnan(bounds[key][1]):
- flag = flag and (scores[key] <= bounds[key][1]+tolerance)
+ flag = flag and (scores[key] <= bounds[key][1] + tolerance)
return flag
-def compare_scores(scores0: dict,
- scores1: dict,
- eps,
- subset: list = None,
- tolerance: float = 1e-5):
+
+def compare_scores(
+ scores0: dict, scores1: dict, eps, subset: list = None, tolerance: float = 1e-5
+):
"""
Compares two sets of scores
@@ -78,5 +74,8 @@ def compare_scores(scores0: dict,
if not isinstance(eps, dict):
eps = {key: eps for key in scores0}
- return all(abs(scores0[key] - scores1[key]) <= eps[key] + tolerance
- for key in scores1 if key in scores0)
+ return all(
+ abs(scores0[key] - scores1[key]) <= eps[key] + tolerance
+ for key in scores1
+ if key in scores0
+ )
diff --git a/mlscorecheck/bundles/skinlesion/_isic2016.py b/mlscorecheck/bundles/skinlesion/_isic2016.py
deleted file mode 100644
index cf623d7..0000000
--- a/mlscorecheck/bundles/skinlesion/_isic2016.py
+++ /dev/null
@@ -1,46 +0,0 @@
-"""
-This module implements the tests for the ISIC2016 dataset
-"""
-
-from ...experiments import get_experiment
-from ...check import check_1_testset_no_kfold_scores
-from ...core import NUMERICAL_TOLERANCE
-
-__all__ = ['check_isic2016']
-
-def check_isic2016(*,
- scores: dict,
- eps: float,
- numerical_tolerance: float = NUMERICAL_TOLERANCE):
- """
- Tests if the scores are consistent with the test set of the ISIC2016
- melanoma classification dataset
-
- Args:
- scores (dict): the scores to check
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
- Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible
- with all scores. Under the key ``prefiltering_details`` one finds the results of the
- prefiltering by using the solutions for the score pairs.
-
- Examples:
- >>> from mlscorecheck.bundles.skinlesion import check_isic2016
- >>> scores = {'acc': 0.7916, 'sens': 0.2933, 'spec': 0.9145}
- >>> results = check_isic2016(scores=scores, eps=1e-4)
- >>> results['inconsistency']
- # False
- """
- data = get_experiment('skinlesion.isic2016')
- return check_1_testset_no_kfold_scores(scores=scores,
- testset=data,
- eps=eps,
- numerical_tolerance=numerical_tolerance)
diff --git a/mlscorecheck/check/__init__.py b/mlscorecheck/check/__init__.py
index 422b85a..e2c1444 100644
--- a/mlscorecheck/check/__init__.py
+++ b/mlscorecheck/check/__init__.py
@@ -1,18 +1,8 @@
"""
-This module brings together all the check functionalities
+This module brings together all test functionalities.
"""
-from ._check_1_testset_no_kfold_scores import *
-
-from ._check_1_dataset_kfold_som_scores import *
-from ._check_1_dataset_known_folds_mos_scores import *
-from ._check_1_dataset_unknown_folds_mos_acc_score import *
-from ._check_1_dataset_unknown_folds_mos_scores import *
-
-from ._check_n_testsets_mos_no_kfold_scores import *
-from ._check_n_testsets_som_no_kfold_scores import *
-
-from ._check_n_datasets_som_kfold_som_scores import *
-from ._check_n_datasets_mos_kfold_som_scores import *
-from ._check_n_datasets_mos_known_folds_mos_scores import *
-from ._check_n_datasets_mos_unknown_folds_mos_scores import *
+from . import binary
+from . import bundles
+from . import multiclass
+from . import regression
diff --git a/mlscorecheck/check/_check_1_dataset_kfold_som_scores.py b/mlscorecheck/check/_check_1_dataset_kfold_som_scores.py
deleted file mode 100644
index b0581c6..0000000
--- a/mlscorecheck/check/_check_1_dataset_kfold_som_scores.py
+++ /dev/null
@@ -1,94 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated by the score-of-means aggregation
-in a kfold scenario on one single dataset.
-"""
-
-from ..core import NUMERICAL_TOLERANCE
-from ..individual import check_scores_tptn_pairs
-from ..aggregated import Experiment
-
-__all__ = ['check_1_dataset_som_scores']
-
-def check_1_dataset_som_scores(dataset: dict,
- folding: dict,
- scores: dict,
- eps,
- *,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
- """
- Checking the consistency of scores calculated by applying k-fold
- cross validation to one single dataset and aggregating the figures
- over the folds in the score of means fashion. All pairs of
- the supported individual scores are checked against all other as in
- the 1_dataset_no_kfold case, however, additionally, if score_bounds
- are specified in the folds, the aggregated check is also executed
- on the supported acc, bacc, sens and spec scores.
-
- Args:
- dataset (dict): the dataset specification
- folding (dict): the folding specification
- scores (dict(str,float)): the scores to check ('acc', 'sens', 'spec',
- 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
- 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
- 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
- f-beta positive or f-beta negative, also set
- 'beta_positive' and 'beta_negative'.
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
-
- Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> dataset = {'dataset_name': 'common_datasets.monk-2'}
- >>> folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}
- >>> scores = {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667,
- 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}
- >>> result = check_1_dataset_som_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # False
-
- >>> dataset = {'p': 10, 'n': 20}
- >>> folding = {'n_folds': 5, 'n_repeats': 1}
- >>> scores = {'acc': 0.428, 'npv': 0.392, 'bacc': 0.442, 'f1p': 0.391}
- >>> result = check_1_dataset_som_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # True
-
- """
- if folding.get('folds') is None and folding.get('strategy') is None:
- # any folding strategy results the same
- folding = {**folding} | {'strategy': 'stratified_sklearn'}
-
- # creating the experiment consisting of one single dataset, the
- # outer level aggregation can be arbitrary
- experiment = Experiment(evaluations=[{'dataset': dataset,
- 'folding': folding,
- 'aggregation': 'som'}],
- aggregation='som')
-
- # executing the individual tests
- return check_scores_tptn_pairs(scores=scores,
- eps=eps,
- p=experiment.figures['p'],
- n=experiment.figures['n'],
- numerical_tolerance=numerical_tolerance)
diff --git a/mlscorecheck/check/_check_1_dataset_known_folds_mos_scores.py b/mlscorecheck/check/_check_1_dataset_known_folds_mos_scores.py
deleted file mode 100644
index fcd868e..0000000
--- a/mlscorecheck/check/_check_1_dataset_known_folds_mos_scores.py
+++ /dev/null
@@ -1,114 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated by the mean-of-scores aggregation
-in a kfold scenario on one single dataset.
-"""
-
-from ..core import NUMERICAL_TOLERANCE
-from ..aggregated import check_aggregated_scores, Experiment, Evaluation
-
-__all__ = ['check_1_dataset_known_folds_mos_scores']
-
-def check_1_dataset_known_folds_mos_scores(dataset: dict,
- folding: dict,
- scores: dict,
- eps,
- fold_score_bounds: dict = None,
- *,
- solver_name: str = None,
- timeout: int = None,
- verbosity: int = 1,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
- """
- Checking the consistency of scores calculated by applying k-fold
- cross validation to one single dataset and aggregating the figures
- over the folds in the mean of scores fashion. Note that this
- test can only check the consistency of the 'acc', 'sens', 'spec'
- and 'bacc' scores. Note that without bounds, if there is a large
- number of folds, it is likely that there will be a configuration
- matching the scores provided. In order to increase the strength of
- the test, one can add score_bounds to the individual folds if
- for example, besides the average score, the minimum and the maximum
- scores over the folds are also provided.
-
- Args:
- dataset (dict): the specification of the dataset
- folding (dict): the specification of the folding
- scores (dict(str,float)): the scores to check
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- fold_score_bounds (dict): bounds on the scores in the folds
- solver_name (None|str): the solver to use
- timeout (None|int): the timeout for the linear programming solver in seconds
- verbosity (int): the verbosity level of the pulp linear programming solver
- 0: silent, non-zero: verbose.
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
-
- Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> dataset = {'p': 126, 'n': 131}
- >>> folding = {'folds': [{'p': 52, 'n': 94}, {'p': 74, 'n': 37}]}
- >>> scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
- >>> result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # False
-
- >>> dataset = {'p': 398, 'n': 569}
- >>> folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
- >>> scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6}
- >>> result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-2)
- >>> result['inconsistency']
- # True
-
- >>> dataset = {'dataset_name': 'common_datasets.glass_0_1_6_vs_2'}
- >>> folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
- >>> scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6, 'bacc': 0.1, 'f1': 0.95}
- >>> result = check_1_dataset_known_folds_mos_scores(dataset=dataset,
- folding=folding,
- fold_score_bounds={'acc': (0.8, 1.0)},
- scores=scores,
- eps=1e-2,
- numerical_tolerance=1e-6)
- >>> result['inconsistency']
- # True
- """
-
- evaluation = Evaluation(dataset=dataset,
- folding=folding,
- fold_score_bounds=fold_score_bounds,
- aggregation='mos')
-
- experiment = Experiment(evaluations=[evaluation.to_dict()],
- aggregation='mos')
-
- return check_aggregated_scores(experiment=experiment.to_dict(),
- scores=scores,
- eps=eps,
- solver_name=solver_name,
- timeout=timeout,
- verbosity=verbosity,
- numerical_tolerance=numerical_tolerance)
diff --git a/mlscorecheck/check/_check_1_dataset_unknown_folds_mos_scores.py b/mlscorecheck/check/_check_1_dataset_unknown_folds_mos_scores.py
deleted file mode 100644
index 656fee8..0000000
--- a/mlscorecheck/check/_check_1_dataset_unknown_folds_mos_scores.py
+++ /dev/null
@@ -1,137 +0,0 @@
-"""
-This module implements consistency testing for scores calculated in a k-fold cross-validation
-scenario with unknown fold structures.
-"""
-
-from ..core import NUMERICAL_TOLERANCE
-from ..aggregated import (Dataset,
- repeated_kfolds_generator,
- kfolds_generator)
-from ._check_1_dataset_known_folds_mos_scores import check_1_dataset_known_folds_mos_scores
-
-__all__ = ['check_1_dataset_unknown_folds_mos_scores',
- 'estimate_n_evaluations']
-
-def estimate_n_evaluations(dataset: dict,
- folding: dict,
- available_scores: list = None) -> int:
- """
- Estimates the number of estimations with different fold combinations.
-
- Args:
- dataset (dict): the dataset specification
- folding (dict): the folding specification
- available_scores (list): the list of available scores
-
- Returns:
- int: the estimated number of different fold configurations.
- """
- dataset = Dataset(**dataset)
- n_repeats = folding.get('n_repeats', 1)
-
- available_scores = [] if available_scores is None else available_scores
-
- count = sum(1 for _ in kfolds_generator({'dataset': dataset.to_dict(),
- 'folding': folding},
- available_scores))
-
- return count**n_repeats
-
-def check_1_dataset_unknown_folds_mos_scores(
- dataset: dict,
- folding: dict,
- scores: dict,
- eps,
- fold_score_bounds: dict = None,
- *,
- solver_name: str = None,
- timeout: int = None,
- verbosity: int = 1,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
- """
- Checking the consistency of scores calculated in a k-fold cross validation on a single
- dataset, in a mean-of-scores fashion, without knowing the folding strategy.
- The function generates all possible foldings of k-valid folds and evaluates the
- consistency on each of them. The scores are inconsistent if all the k-fold configurations
- lead to inconsistencies identified.
-
- Note that depending on the size of the dataset (especially the number of minority instances)
- and the folding configuration, this test might lead to an untractable number of problems to
- be solved. Use the function ``estimate_n_evaluations`` to get a rough upper bound estimate
- on the number of fold combinations.
-
- Args:
- dataset (dict): the dataset specification
- folding (dict): the folding specification
- scores (dict(str,float)): the scores to check
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- solver_name (None|str): the solver to use
- timeout (None|int): the timeout for the linear programming solver in seconds
- verbosity (int): the verbosity level of the pulp linear programming solver
- 0: silent, non-zero: verbose.
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
-
- Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The details of the mean-of-scores checks and all fold configurations
- can be found under the ``details`` key.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> dataset = {'p': 126, 'n': 131}
- >>> folding = {'n_folds': 2, 'n_repeats': 1}
- >>> scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
- >>> result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # False
-
- >>> dataset = {'p': 19, 'n': 97}
- >>> folding = {'n_folds': 3, 'n_repeats': 1}
- >>> scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6}
- >>> result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,
- folding=folding,
- scores=scores,
- eps=1e-4)
- >>> result['inconsistency']
- # True
- """
- evaluation = {'dataset': dataset,
- 'folding': folding,
- 'fold_score_bounds': fold_score_bounds,
- 'aggregation': 'mos'}
-
- results = {'details': []}
-
- idx = 0
- for evaluation_0 in repeated_kfolds_generator(evaluation,
- list(scores.keys())):
- tmp = {'folds': evaluation_0['folding']['folds'],
- 'details': check_1_dataset_known_folds_mos_scores(
- scores=scores,
- eps=eps,
- dataset=evaluation_0['dataset'],
- folding=evaluation_0['folding'],
- fold_score_bounds=evaluation_0.get('fold_score_bounds'),
- solver_name=solver_name,
- timeout=timeout,
- verbosity=verbosity,
- numerical_tolerance=numerical_tolerance),
- 'configuration_id': idx}
- results['details'].append(tmp)
- if not tmp['details']['inconsistency']:
- break
- idx += 1
-
- results['inconsistency'] = all(tmp['details']['inconsistency'] for tmp in results['details'])
-
- return results
diff --git a/mlscorecheck/check/_check_1_testset_no_kfold_scores.py b/mlscorecheck/check/_check_1_testset_no_kfold_scores.py
deleted file mode 100644
index 46c73f0..0000000
--- a/mlscorecheck/check/_check_1_testset_no_kfold_scores.py
+++ /dev/null
@@ -1,98 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated from raw figures
-"""
-
-import warnings
-
-from ..core import logger, NUMERICAL_TOLERANCE
-from ..individual import check_scores_tptn_pairs
-from ..experiments import dataset_statistics
-
-__all__ = ['check_1_testset_no_kfold_scores']
-
-def check_1_testset_no_kfold_scores(testset: dict,
- scores: dict,
- eps,
- *,
- numerical_tolerance: float = NUMERICAL_TOLERANCE,
- prefilter_by_pairs: bool = True) -> dict:
- """
- Use this check if the scores are calculated on one single test set
- with no kfolding or aggregation over multiple datasets.
-
- Args:
- testset (dict): the specification of a testset with p, n or its name
- scores (dict(str,float)): the scores to check ('acc', 'sens', 'spec',
- 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
- 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
- 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
- f-beta positive or f-beta negative, also set
- 'beta_positive' and 'beta_negative'.
- eps (float|dict(str,float)): the numerical uncertainty (potentially for each score)
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
- prefilter_by_pairs (bool): whether to do a prefiltering based on the score-pair tp-tn
- solutions (faster)
-
- Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> from mlscorecheck.check import check_1_testset_no_kfold_scores
- >>> testset = {'p': 530, 'n': 902}
- >>> scores = {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
- >>> result = check_1_testset_no_kfold_scores(testset=testset,
- scores=scores,
- eps=1e-2)
- >>> result['inconsistency']
- # False
-
- >>> testset = {'p': 530, 'n': 902}
- >>> scores = {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
- >>> result = check_1_testset_no_kfold_scores(testset=testset,
- scores=scores,
- eps=1e-2)
- >>> result['inconsistency']
- # True
-
- """
- logger.info('Use this function if the scores originate from the '\
- 'tp and tn statistics calculated on one test set with '\
- 'no aggregation of any kind.')
-
- if ('p' not in testset or 'n' not in testset) and ('name' not in testset):
- raise ValueError('either "p" and "n" or "name" should be specified')
-
- if ('n_repeats' in testset) or ('n_folds' in testset) \
- or ('folds' in testset) or ('aggregation' in testset):
- warnings.warn('Additional fields beyond ("p", "n") or "name" present ' \
- 'in the specification, you might want to use another check '\
- 'function specialized to datasets')
-
- p = testset.get('p')
- n = testset.get('n')
- if 'name' in testset:
- p = dataset_statistics[testset['name']]['p']
- n = dataset_statistics[testset['name']]['n']
-
- logger.info('calling the score check with scores %s, uncertainty %s, p %d and n %d',
- str(scores), str(eps), p, n)
-
- return check_scores_tptn_pairs(scores=scores,
- eps=eps,
- p=p,
- n=n,
- numerical_tolerance=numerical_tolerance,
- prefilter_by_pairs=prefilter_by_pairs)
diff --git a/mlscorecheck/check/_check_n_datasets_mos_kfold_som_scores.py b/mlscorecheck/check/_check_n_datasets_mos_kfold_som_scores.py
deleted file mode 100644
index 135d146..0000000
--- a/mlscorecheck/check/_check_n_datasets_mos_kfold_som_scores.py
+++ /dev/null
@@ -1,114 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated by the score of means aggregation
-in a kfold scenarios and mean of scores aggregation on multiple datasets.
-"""
-
-import copy
-
-from ..aggregated import check_aggregated_scores, Experiment
-from ..core import NUMERICAL_TOLERANCE
-
-__all__ = ['check_n_datasets_mos_kfold_som_scores']
-
-def check_n_datasets_mos_kfold_som_scores(evaluations: list,
- scores: dict,
- eps,
- dataset_score_bounds: dict = None,
- *,
- solver_name: str = None,
- timeout: int = None,
- verbosity: int = 1,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
- """
- Checking the consistency of scores calculated by applying k-fold
- cross validation to multiple datasets and aggregating the figures
- over the folds in the score of means fashion and over the datasets
- in the mean of scores fashion. This aggregated check can be applied
- only if some of the acc, sens, spec and bacc scores are provided.
-
- Args:
- evaluations (list(dict)): the list of evaluation specifications
- scores (dict(str,float)): the scores to check
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- solver_name (None|str): the solver to use
- timeout (None|int): the timeout for the linear programming solver in seconds
- verbosity (int): the verbosity of the linear programming solver,
- 0: silent, 1: verbose.
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
-
- Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> from mlscorecheck.check import check_n_datasets_mos_kfold_som_scores
- >>> evaluation0 = {'dataset': {'p': 39, 'n': 822},
- 'folding': {'n_folds': 5, 'n_repeats': 3,
- 'strategy': 'stratified_sklearn'}}
- >>> evaluation1 = {'dataset': {'dataset_name': 'common_datasets.winequality-white-3_vs_7'},
- 'folding': {'n_folds': 5, 'n_repeats': 3,
- 'strategy': 'stratified_sklearn'}}
- >>> evaluations = [evaluation0, evaluation1]
- >>> scores = {'acc': 0.312, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
- >>> result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,
- dataset_score_bounds={'acc': (0.0, 0.5)},
- eps=1e-4,
- scores=scores)
- >>> result['inconsistency']
- # False
-
- >>> evaluation0 = {'dataset': {'p': 39, 'n': 822},
- 'folding': {'n_folds': 5, 'n_repeats': 3,
- 'strategy': 'stratified_sklearn'}}
- >>> evaluation1 = {'dataset': {'dataset_name': 'common_datasets.winequality-white-3_vs_7'},
- 'folding': {'n_folds': 5, 'n_repeats': 3,
- 'strategy': 'stratified_sklearn'}}
- >>> evaluations = [evaluation0, evaluation1]
- >>> scores = {'acc': 0.412, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
- >>> result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,
- dataset_score_bounds={'acc': (0.5, 1.0)},
- eps=1e-4,
- scores=scores)
- >>> result['inconsistency']
- # True
- """
-
- if any(evaluation.get('aggregation', 'som') != 'som' for evaluation in evaluations):
- raise ValueError('the aggregation specified in each dataset must be "rom" or nothing.')
-
- if any(evaluation.get('fold_score_bounds') is not None for evaluation in evaluations):
- raise ValueError('do not specify fold_score_bounds for a SoM evaluation')
-
- evaluations = copy.deepcopy(evaluations)
-
- for evaluation in evaluations:
- evaluation['aggregation'] = 'som'
-
- experiment = Experiment(evaluations=evaluations,
- dataset_score_bounds=dataset_score_bounds,
- aggregation='mos')
-
- return check_aggregated_scores(experiment=experiment.to_dict(),
- scores=scores,
- eps=eps,
- solver_name=solver_name,
- timeout=timeout,
- verbosity=verbosity,
- numerical_tolerance=numerical_tolerance)
diff --git a/mlscorecheck/check/_check_n_datasets_mos_known_folds_mos_scores.py b/mlscorecheck/check/_check_n_datasets_mos_known_folds_mos_scores.py
deleted file mode 100644
index c0e4072..0000000
--- a/mlscorecheck/check/_check_n_datasets_mos_known_folds_mos_scores.py
+++ /dev/null
@@ -1,104 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated by the mean of scores aggregation
-in a kfold scenarios and mean of scores aggregation on multiple datastes.
-"""
-
-import copy
-
-from ..aggregated import check_aggregated_scores, Experiment
-from ..core import NUMERICAL_TOLERANCE
-
-__all__ = ['check_n_datasets_mos_known_folds_mos_scores']
-
-def check_n_datasets_mos_known_folds_mos_scores(evaluations: list,
- scores: dict,
- eps,
- dataset_score_bounds: dict = None,
- *,
- solver_name: str = None,
- timeout: int = None,
- verbosity: int = 1,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
- """
- Checking the consistency of scores calculated by applying k-fold
- cross validation to multiple datasets and aggregating the figures
- over the folds in the mean of scores fashion and over the datasets
- in the mean of scores fashion.
-
- Args:
- evaluations (list(dict)): the list of evaluation specifications
- scores (dict(str,float)): the scores to check
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- solver_name (None|str): the solver to use
- timeout (None|int): the timeout for the linear programming solver in seconds
- verbosity (int): the verbosity of the pulp linear programming solver,
- 0: silent, non-zero: verbose
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
-
- Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> evaluation0 = {'dataset': {'p': 118, 'n': 95},
- 'folding': {'folds': [{'p': 22, 'n': 23}, {'p': 96, 'n': 72}]}}
- >>> evaluation1 = {'dataset': {'p': 781, 'n': 423},
- 'folding': {'folds': [{'p': 300, 'n': 200}, {'p': 481, 'n': 223}]}}
- >>> evaluations = [evaluation0, evaluation1]
- >>> scores = {'acc': 0.61, 'sens': 0.709, 'spec': 0.461, 'bacc': 0.585}
- >>> result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # False
-
- >>> evaluation0 = {'dataset': {'p': 118, 'n': 95},
- 'folding': {'folds': [{'p': 22, 'n': 23}, {'p': 96, 'n': 72}]}}
- >>> evaluation1 = {'dataset': {'p': 781, 'n': 423},
- 'folding': {'folds': [{'p': 300, 'n': 200}, {'p': 481, 'n': 223}]}}
- >>> evaluations = [evaluation0, evaluation1]
- >>> scores = {'acc': 0.71, 'sens': 0.709, 'spec': 0.461}
- >>> result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # True
- """
- if any(evaluation.get('aggregation', 'mos') != 'mos' for evaluation in evaluations):
- raise ValueError('the aggregation specified in each dataset must be "mor" or nothing.')
- if any(evaluation.get('fold_score_bounds') is not None for evaluation in evaluations):
- raise ValueError('do not specify fold_score_bounds through this interface')
-
- evaluations = copy.deepcopy(evaluations)
-
- for evaluation in evaluations:
- evaluation['aggregation'] = 'mos'
-
- experiment = Experiment(evaluations=evaluations,
- dataset_score_bounds=dataset_score_bounds,
- aggregation='mos')
-
- return check_aggregated_scores(experiment=experiment.to_dict(),
- scores=scores,
- eps=eps,
- solver_name=solver_name,
- timeout=timeout,
- verbosity=verbosity,
- numerical_tolerance=numerical_tolerance)
diff --git a/mlscorecheck/check/_check_n_datasets_mos_unknown_folds_mos_scores.py b/mlscorecheck/check/_check_n_datasets_mos_unknown_folds_mos_scores.py
deleted file mode 100644
index e26b9ab..0000000
--- a/mlscorecheck/check/_check_n_datasets_mos_unknown_folds_mos_scores.py
+++ /dev/null
@@ -1,143 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated by the mean of scores aggregation
-in a kfold scenarios and mean of scores aggregation on multiple datasets.
-"""
-
-import copy
-
-import numpy as np
-
-from ._check_n_datasets_mos_known_folds_mos_scores \
- import check_n_datasets_mos_known_folds_mos_scores
-from ._check_1_dataset_unknown_folds_mos_scores import estimate_n_evaluations
-from ..core import NUMERICAL_TOLERANCE
-from ..aggregated import (experiment_kfolds_generator)
-
-__all__ = ['check_n_datasets_mos_unknown_folds_mos_scores',
- 'estimate_n_experiments']
-
-def estimate_n_experiments(evaluations: list,
- available_scores: list = None) -> int:
- """
- Estimates the number of estimations with different fold combinations.
-
- Args:
- evaluations (list): a list of evaluation specifications
-
- Returns:
- int: the estimated number of different fold configurations.
- """
- available_scores = [] if available_scores is None else available_scores
-
- counts = [estimate_n_evaluations(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- available_scores=available_scores)
- for evaluation in evaluations]
- return np.prod(counts)
-
-def check_n_datasets_mos_unknown_folds_mos_scores(evaluations: list,
- scores: dict,
- eps,
- dataset_score_bounds: dict = None,
- *,
- solver_name: str = None,
- timeout: int = None,
- verbosity: int = 1,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
- """
- Checking the consistency of scores calculated by applying k-fold
- cross validation to multiple datasets and aggregating the figures
- over the folds in the mean of scores fashion and over the datasets
- in the mean of scores fashion.
-
- Note that depending on the number of the minority instances and on the
- folding structure, this test might lead to enormous execution times.
- Use the function ``estimate_n_experiments`` to get a rough upper bound estimate
- on the number of experiments with different fold combinations.
-
- Args:
- evaluations (list(dict)): the list of evaluation specifications
- scores (dict(str,float)): the scores to check
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- solver_name (None|str): the solver to use
- timeout (None|int): the timeout for the linear programming solver in seconds
- verbosity (int): the verbosity of the pulp linear programming solver,
- 0: silent, non-zero: verbose
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
-
- Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The ``details`` entry contains all possible folding
- combinations and the corresponding detailed results.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> from mlscorecheck.check import check_n_datasets_mos_unknown_folds_mos_scores
- >>> evaluation0 = {'dataset': {'p': 13, 'n': 73},
- 'folding': {'n_folds': 4, 'n_repeats': 1}}
- >>> evaluation1 = {'dataset': {'p': 7, 'n': 26},
- 'folding': {'n_folds': 3, 'n_repeats': 1}}
- >>> evaluations = [evaluation0, evaluation1]
- >>> scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.343}
- >>> result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # False
-
- >>> evaluation0 = {'dataset': {'p': 13, 'n': 73},
- 'folding': {'n_folds': 4, 'n_repeats': 1}}
- >>> evaluation1 = {'dataset': {'p': 7, 'n': 26},
- 'folding': {'n_folds': 3, 'n_repeats': 1}}
- >>> evaluations = [evaluation0, evaluation1]
- >>> scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.9}
- >>> result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # True
- """
- if any(evaluation.get('aggregation', 'mos') != 'mos' for evaluation in evaluations):
- raise ValueError('the aggregation specified in each dataset must be "mor" or nothing.')
- if any(evaluation.get('fold_score_bounds') is not None for evaluation in evaluations):
- raise ValueError('do not specify fold score bounds through this interface')
-
- evaluations = copy.deepcopy(evaluations)
-
- for evaluation in evaluations:
- evaluation['aggregation'] = 'mos'
-
- experiment = {'evaluations': evaluations,
- 'dataset_score_bounds': dataset_score_bounds,
- 'aggregation': 'mos'}
-
- results = {'details': [],
- 'inconsistency': True}
-
- for experiment in experiment_kfolds_generator(experiment,
- list(scores.keys())):
- result = check_n_datasets_mos_known_folds_mos_scores(
- evaluations=experiment['evaluations'],
- dataset_score_bounds=experiment.get('dataset_score_bounds'),
- scores=scores,
- eps=eps,
- timeout=timeout,
- solver_name=solver_name,
- verbosity=verbosity,
- numerical_tolerance=numerical_tolerance)
-
- results['details'].append(result)
- results['inconsistency'] = results['inconsistency'] and result['inconsistency']
-
- if not result['inconsistency']:
- break
-
- return results
diff --git a/mlscorecheck/check/_check_n_testsets_mos_no_kfold_scores.py b/mlscorecheck/check/_check_n_testsets_mos_no_kfold_scores.py
deleted file mode 100644
index 9a2bd11..0000000
--- a/mlscorecheck/check/_check_n_testsets_mos_no_kfold_scores.py
+++ /dev/null
@@ -1,96 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated by the mean of scores aggregation over multiple testsets.
-"""
-
-from ..aggregated import check_aggregated_scores, Experiment, Dataset
-from ..core import NUMERICAL_TOLERANCE
-
-__all__ = ['check_n_testsets_mos_no_kfold_scores']
-
-def check_n_testsets_mos_no_kfold_scores(testsets: list,
- scores: dict,
- eps,
- testset_score_bounds: dict = None,
- *,
- solver_name: str = None,
- timeout: int = None,
- verbosity: int = 1,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
- """
- Checking the consistency of scores calculated by the mean of scores
- aggregation over multiple testsets. This aggregated check can be applied
- only if some of the acc, sens, spec and bacc scores are provided.
-
- Args:
- testsets (list(dict)): the list of testset specifications
- scores (dict(str,float)): the scores to check
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- testset_score_bounds (dict(str,tuple(float,float))): the potential bounds on the scores
- of the testsets
- solver_name (None|str): the solver to use
- timeout (None|int): the timeout for the linear programming solver in seconds
- verbosity (int): the verbosity of the linear programming solver,
- 0: silent, 1: verbose.
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
-
- Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> from mlscorecheck.check import check_n_testsets_mos_no_kfold_scores
- >>> testsets = [{'p': 349, 'n': 50},
- {'p': 478, 'n': 323},
- {'p': 324, 'n': 83},
- {'p': 123, 'n': 145}]
- >>> scores = {'acc': 0.6441, 'sens': 0.6706, 'spec': 0.3796, 'bacc': 0.5251}
- >>> result = check_n_testsets_mos_no_kfold_scores(testsets=testsets,
- eps=1e-4,
- scores=scores)
- >>> result['inconsistency']
- # False
-
- >>> scores['sens'] = 0.6756
- >>> result = check_n_datasets_mos_no_kfold_scores(testsets=testsets,
- eps=1e-4,
- scores=scores)
- >>> result['inconsistency']
- # True
- """
-
- datasets = [Dataset(**dataset) for dataset in testsets]
-
- evaluations = [{'dataset': dataset.to_dict(),
- 'folding': {'folds': [{'p': dataset.p, 'n': dataset.n,
- 'identifier': f'{dataset.identifier}_{idx}'}]},
- 'aggregation': 'mos'}
- for idx, dataset in enumerate(datasets)]
-
- experiment = Experiment(evaluations=evaluations,
- dataset_score_bounds=testset_score_bounds,
- aggregation='mos')
-
- return check_aggregated_scores(experiment=experiment.to_dict(),
- scores=scores,
- eps=eps,
- solver_name=solver_name,
- timeout=timeout,
- verbosity=verbosity,
- numerical_tolerance=numerical_tolerance)
diff --git a/mlscorecheck/check/_check_n_testsets_som_no_kfold_scores.py b/mlscorecheck/check/_check_n_testsets_som_no_kfold_scores.py
deleted file mode 100644
index f5a69d5..0000000
--- a/mlscorecheck/check/_check_n_testsets_som_no_kfold_scores.py
+++ /dev/null
@@ -1,90 +0,0 @@
-"""
-This module implements the top level check function for
-scores calculated by the score-of-means aggregation
-over multiple testsets.
-"""
-
-from ..core import NUMERICAL_TOLERANCE
-from ..individual import check_scores_tptn_pairs
-from ..aggregated import Experiment, Dataset
-
-__all__ = ['check_n_testsets_som_no_kfold_scores']
-
-def check_n_testsets_som_no_kfold_scores(testsets: list,
- scores: dict,
- eps,
- *,
- numerical_tolerance: float = NUMERICAL_TOLERANCE,
- prefilter_by_pairs=True):
- """
- Checking the consistency of scores calculated by aggregating the figures
- over testsets in the score of means fashion. If
- score bounds are specified and some of the 'acc', 'sens', 'spec' and
- 'bacc' scores are supplied, the linear programming based check is
- executed to see if the bound conditions can be satisfied.
-
- Args:
- datasets (list(dict)): the specification of the evaluations
- scores (dict(str,float)): the scores to check ('acc', 'sens', 'spec',
- 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
- 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
- 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
- f-beta positive or f-beta negative, also set
- 'beta_positive' and 'beta_negative'.
- eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
- numerical_tolerance (float): in practice, beyond the numerical uncertainty of
- the scores, some further tolerance is applied. This is
- orders of magnitude smaller than the uncertainty of the
- scores. It does ensure that the specificity of the test
- is 1, it might slightly decrease the sensitivity.
- prefilter_by_pairs (bool): whether to prefilter the solution space by pair
- solutions when possible to speed up the process
-
- Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
-
- Raises:
- ValueError: if the problem is not specified properly
-
- Examples:
- >>> testsets = [{'p': 405, 'n': 223}, {'p': 3, 'n': 422}, {'p': 109, 'n': 404}]
- >>> scores = {'acc': 0.4719, 'npv': 0.6253, 'f1p': 0.3091}
- >>> result = check_n_datasets_som_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # False
-
- >>> scores['npv'] = 0.6263
- >>> result = check_n_datasets_som_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=1e-3)
- >>> result['inconsistency']
- # True
- """
-
- datasets = [Dataset(**dataset) for dataset in testsets]
-
- evaluations = [{'dataset': dataset.to_dict(),
- 'folding': {'folds': [{'p': dataset.p, 'n': dataset.n,
- 'identifier': f'{dataset.identifier}_{idx}'}]},
- 'aggregation': 'mos'}
- for idx, dataset in enumerate(datasets)]
-
- # creating the experiment consisting of one single dataset, the
- # outer level aggregation can be arbitrary
- experiment = Experiment(evaluations=evaluations,
- aggregation='som')
-
- # executing the individual tests
- return check_scores_tptn_pairs(scores=scores,
- eps=eps,
- p=experiment.figures['p'],
- n=experiment.figures['n'],
- numerical_tolerance=numerical_tolerance,
- prefilter_by_pairs=prefilter_by_pairs)
diff --git a/mlscorecheck/check/binary/__init__.py b/mlscorecheck/check/binary/__init__.py
new file mode 100644
index 0000000..fb9ce9b
--- /dev/null
+++ b/mlscorecheck/check/binary/__init__.py
@@ -0,0 +1,17 @@
+"""
+This module brings together all the check functionalities
+"""
+
+from ._check_1_testset_no_kfold import *
+
+from ._check_1_dataset_kfold_som import *
+from ._check_1_dataset_known_folds_mos import *
+from ._check_1_dataset_unknown_folds_mos import *
+
+from ._check_n_testsets_mos_no_kfold import *
+from ._check_n_testsets_som_no_kfold import *
+
+from ._check_n_datasets_som_kfold_som import *
+from ._check_n_datasets_mos_kfold_som import *
+from ._check_n_datasets_mos_known_folds_mos import *
+from ._check_n_datasets_mos_unknown_folds_mos import *
diff --git a/mlscorecheck/check/binary/_check_1_dataset_kfold_som.py b/mlscorecheck/check/binary/_check_1_dataset_kfold_som.py
new file mode 100644
index 0000000..21d8d64
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_1_dataset_kfold_som.py
@@ -0,0 +1,112 @@
+"""
+This module implements the top level check function for
+scores calculated by the score-of-means aggregation
+in a kfold scenario on one single dataset.
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...individual import check_scores_tptn_pairs
+from ...aggregated import Experiment
+
+__all__ = ["check_1_dataset_kfold_som"]
+
+
+def check_1_dataset_kfold_som(
+ dataset: dict,
+ folding: dict,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ prefilter_by_pairs: bool = True
+) -> dict:
+ """
+ This function checks the consistency of scores calculated by applying k-fold cross validation
+ to a single dataset and aggregating the figures over the folds in the score of means fashion.
+ The test is performed by exhaustively testing all possible confusion matrices.
+
+ Args:
+ dataset (dict): The dataset specification.
+ folding (dict): The folding specification.
+ scores (dict(str,float)): The scores to check ('acc', 'sens', 'spec', 'bacc', 'npv', 'ppv',
+ 'f1', 'fm', 'f1n', 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn',
+ 'mcc', 'bm', 'pt', 'dor', 'ji', 'kappa'). When using f-beta
+ positive or f-beta negative, also set 'beta_positive' and
+ 'beta_negative'.
+ eps (float|dict(str,float)): The numerical uncertainty(ies) of the scores.
+ numerical_tolerance (float, optional): In practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This
+ is orders of magnitude smaller than the uncertainty
+ of the scores. It does ensure that the specificity
+ of the test is 1, it might slightly decrease the
+ sensitivity. Defaults to NUMERICAL_TOLERANCE.
+ prefilter_by_pairs (bool): whether to do a prefiltering based on the score-pair tp-tn
+ solutions (faster)
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the dataset.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_1_dataset_kfold_som
+ >>> dataset = {'dataset_name': 'common_datasets.monk-2'}
+ >>> folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}
+ >>> scores = {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667,
+ 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}
+ >>> result = check_1_dataset_kfold_som(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # False
+
+ >>> dataset = {'p': 10, 'n': 20}
+ >>> folding = {'n_folds': 5, 'n_repeats': 1}
+ >>> scores = {'acc': 0.428, 'npv': 0.392, 'bacc': 0.442, 'f1p': 0.391}
+ >>> result = check_1_dataset_kfold_som(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # True
+
+ """
+ if folding.get("folds") is None and folding.get("strategy") is None:
+ # any folding strategy results the same
+ folding = {**folding} | {"strategy": "stratified_sklearn"}
+
+ # creating the experiment consisting of one single dataset, the
+ # outer level aggregation can be arbitrary
+ experiment = Experiment(
+ evaluations=[{"dataset": dataset, "folding": folding, "aggregation": "som"}],
+ aggregation="som",
+ )
+
+ # executing the individual tests
+ return check_scores_tptn_pairs(
+ scores=scores,
+ eps=eps,
+ p=experiment.figures["p"],
+ n=experiment.figures["n"],
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
diff --git a/mlscorecheck/check/binary/_check_1_dataset_known_folds_mos.py b/mlscorecheck/check/binary/_check_1_dataset_known_folds_mos.py
new file mode 100644
index 0000000..07b78b3
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_1_dataset_known_folds_mos.py
@@ -0,0 +1,125 @@
+"""
+This module implements the top level check function for
+scores calculated by the mean-of-scores aggregation
+in a k-fold scenario on one single dataset.
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import check_aggregated_scores, Experiment, Evaluation
+
+__all__ = ["check_1_dataset_known_folds_mos"]
+
+
+def check_1_dataset_known_folds_mos(
+ dataset: dict,
+ folding: dict,
+ scores: dict,
+ eps,
+ fold_score_bounds: dict = None,
+ *,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ This function checks the consistency of scores calculated by applying k-fold cross validation
+ to a single dataset and aggregating the figures over the folds in the mean of scores fashion.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add ``fold_score_bounds`` when, for example, the minimum
+ and the maximum scores over the folds are also provided.
+
+ Args:
+ dataset (dict): The dataset specification.
+ folding (dict): The folding specification.
+ scores (dict(str,float)): The scores to check.
+ eps (float|dict(str,float)): The numerical uncertainty(ies) of the scores.
+ fold_score_bounds (None|dict(str,tuple(float,float))): Bounds on the scores in the folds.
+ solver_name (None|str): The solver to use.
+ timeout (None|int): The timeout for the linear programming solver in seconds.
+ verbosity (int): The verbosity level of the pulp linear programming solver.
+ 0: silent, non-zero: verbose.
+ numerical_tolerance (float): In practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_1_dataset_known_folds_mos
+ >>> dataset = {'p': 126, 'n': 131}
+ >>> folding = {'folds': [{'p': 52, 'n': 94}, {'p': 74, 'n': 37}]}
+ >>> scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
+ >>> result = check_1_dataset_known_folds_mos(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # False
+
+ >>> dataset = {'p': 398, 'n': 569}
+ >>> folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
+ >>> scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6}
+ >>> result = check_1_dataset_known_folds_mos(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-2)
+ >>> result['inconsistency']
+ # True
+
+ >>> dataset = {'dataset_name': 'common_datasets.glass_0_1_6_vs_2'}
+ >>> folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
+ >>> scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6, 'bacc': 0.1, 'f1': 0.95}
+ >>> result = check_1_dataset_known_folds_mos(dataset=dataset,
+ folding=folding,
+ fold_score_bounds={'acc': (0.8, 1.0)},
+ scores=scores,
+ eps=1e-2,
+ numerical_tolerance=1e-6)
+ >>> result['inconsistency']
+ # True
+ """
+
+ evaluation = Evaluation(
+ dataset=dataset,
+ folding=folding,
+ fold_score_bounds=fold_score_bounds,
+ aggregation="mos",
+ )
+
+ experiment = Experiment(evaluations=[evaluation.to_dict()], aggregation="mos")
+
+ return check_aggregated_scores(
+ experiment=experiment.to_dict(),
+ scores=scores,
+ eps=eps,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/binary/_check_1_dataset_unknown_folds_mos.py b/mlscorecheck/check/binary/_check_1_dataset_unknown_folds_mos.py
new file mode 100644
index 0000000..04bd032
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_1_dataset_unknown_folds_mos.py
@@ -0,0 +1,164 @@
+"""
+This module implements consistency testing for scores calculated in a k-fold cross-validation
+scenario with unknown fold configuration.
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import Dataset, repeated_kfolds_generator, kfolds_generator
+from ._check_1_dataset_known_folds_mos import check_1_dataset_known_folds_mos
+
+__all__ = ["check_1_dataset_unknown_folds_mos", "estimate_n_evaluations"]
+
+
+def estimate_n_evaluations(
+ dataset: dict, folding: dict, available_scores: list = None
+) -> int:
+ """
+ Estimates the number of estimations with different fold combinations.
+
+ Args:
+ dataset (dict): the dataset specification
+ folding (dict): the folding specification
+ available_scores (list): the list of available scores
+
+ Returns:
+ int: the estimated number of different fold configurations.
+ """
+ dataset = Dataset(**dataset)
+ n_repeats = folding.get("n_repeats", 1)
+
+ available_scores = [] if available_scores is None else available_scores
+
+ count = sum(
+ 1
+ for _ in kfolds_generator(
+ {"dataset": dataset.to_dict(), "folding": folding}, available_scores
+ )
+ )
+
+ return count**n_repeats
+
+
+def check_1_dataset_unknown_folds_mos(
+ dataset: dict,
+ folding: dict,
+ scores: dict,
+ eps,
+ fold_score_bounds: dict = None,
+ *,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ Checking the consistency of scores calculated in a k-fold cross validation on a single
+ dataset, in a mean-of-scores fashion, without knowing the fold configuration.
+ The function generates all possible fold configurations and tests the
+ consistency of each. The scores are inconsistent if all the k-fold configurations
+ lead to inconsistencies identified.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add fold_score_bounds when, for example, the minimum and
+ the maximum scores over the folds are also provided.
+
+ Note that depending on the size of the dataset (especially the number of minority instances)
+ and the folding configuration, this test might lead to an untractable number of problems to
+ be solved. Use the function ``estimate_n_evaluations`` to get an upper bound estimate
+ on the number of fold combinations.
+
+ The evaluation of possible fold configurations stops when a feasible configuration is found.
+
+ Args:
+ dataset (dict): the dataset specification
+ folding (dict): the folding specification
+ scores (dict(str,float)): the scores to check
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ fold_score_bounds (None|dict(str,dict(str,str))): bounds on the scores in the folds
+ solver_name (None|str): the solver to use
+ timeout (None|int): the timeout for the linear programming solver in seconds
+ verbosity (int): the verbosity level of the pulp linear programming solver
+ 0: silent, non-zero: verbose.
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list of dictionaries containing the details of the consistency tests. Each
+ entry contains the specification of the folds being tested and the
+ outcome of the ``check_1_dataset_known_folds_mos`` function.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_1_dataset_unknown_folds_mos
+ >>> dataset = {'p': 126, 'n': 131}
+ >>> folding = {'n_folds': 2, 'n_repeats': 1}
+ >>> scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}
+ >>> result = check_1_dataset_unknown_folds_mos(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # False
+
+ >>> dataset = {'p': 19, 'n': 97}
+ >>> folding = {'n_folds': 3, 'n_repeats': 1}
+ >>> scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6}
+ >>> result = check_1_dataset_unknown_folds_mos(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+ """
+ evaluation = {
+ "dataset": dataset,
+ "folding": folding,
+ "fold_score_bounds": fold_score_bounds,
+ "aggregation": "mos",
+ }
+
+ results = {"details": []}
+
+ idx = 0
+ for evaluation_0 in repeated_kfolds_generator(evaluation, list(scores.keys())):
+ tmp = {
+ "folds": evaluation_0["folding"]["folds"],
+ "details": check_1_dataset_known_folds_mos(
+ scores=scores,
+ eps=eps,
+ dataset=evaluation_0["dataset"],
+ folding=evaluation_0["folding"],
+ fold_score_bounds=evaluation_0.get("fold_score_bounds"),
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ ),
+ "configuration_id": idx,
+ }
+ results["details"].append(tmp)
+ if not tmp["details"]["inconsistency"]:
+ break
+ idx += 1
+
+ results["inconsistency"] = all(
+ tmp["details"]["inconsistency"] for tmp in results["details"]
+ )
+
+ return results
diff --git a/mlscorecheck/check/binary/_check_1_testset_no_kfold.py b/mlscorecheck/check/binary/_check_1_testset_no_kfold.py
new file mode 100644
index 0000000..c411d6b
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_1_testset_no_kfold.py
@@ -0,0 +1,129 @@
+"""
+This module implements the top level check function for
+scores calculated from one single confusion matrix.
+"""
+
+import warnings
+
+from ...core import logger, NUMERICAL_TOLERANCE
+from ...individual import check_scores_tptn_pairs
+from ...experiments import dataset_statistics
+
+__all__ = ["check_1_testset_no_kfold"]
+
+
+def check_1_testset_no_kfold(
+ testset: dict,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ prefilter_by_pairs: bool = True
+) -> dict:
+ """
+ Use this check if the scores are calculated on one single test set
+ with no kfolding. The test is performed by exhaustively testing all
+ possible confusion matrices.
+
+ Args:
+ testset (dict): the specification of a testset with p, n or its name
+ scores (dict(str,float)): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
+ eps (float|dict(str,float)): the numerical uncertainty (potentially for each score)
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+ prefilter_by_pairs (bool): whether to do a prefiltering based on the score-pair tp-tn
+ solutions (faster)
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the dataset.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_1_testset_no_kfold
+ >>> testset = {'p': 530, 'n': 902}
+ >>> scores = {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
+ >>> result = check_1_testset_no_kfold(testset=testset,
+ scores=scores,
+ eps=1e-2)
+ >>> result['inconsistency']
+ # False
+
+ >>> testset = {'p': 530, 'n': 902}
+ >>> scores = {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}
+ >>> result = check_1_testset_no_kfold(testset=testset,
+ scores=scores,
+ eps=1e-2)
+ >>> result['inconsistency']
+ # True
+
+ """
+ logger.info(
+ "Use this function if the scores originate from the "
+ "tp and tn statistics calculated on one test set with "
+ "no aggregation of any kind."
+ )
+
+ if ("p" not in testset or "n" not in testset) and ("name" not in testset):
+ raise ValueError('either "p" and "n" or "name" should be specified')
+
+ if (
+ ("n_repeats" in testset)
+ or ("n_folds" in testset)
+ or ("folds" in testset)
+ or ("aggregation" in testset)
+ ):
+ warnings.warn(
+ 'Additional fields beyond ("p", "n") or "name" present '
+ "in the specification, you might want to use another check "
+ "function specialized to datasets"
+ )
+
+ p = testset.get("p")
+ n = testset.get("n")
+ if "name" in testset:
+ p = dataset_statistics[testset["name"]]["p"]
+ n = dataset_statistics[testset["name"]]["n"]
+
+ logger.info(
+ "calling the score check with scores %s, uncertainty %s, p %d and n %d",
+ str(scores),
+ str(eps),
+ p,
+ n,
+ )
+
+ return check_scores_tptn_pairs(
+ scores=scores,
+ eps=eps,
+ p=p,
+ n=n,
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
diff --git a/mlscorecheck/check/binary/_check_n_datasets_mos_kfold_som.py b/mlscorecheck/check/binary/_check_n_datasets_mos_kfold_som.py
new file mode 100644
index 0000000..fe46e10
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_n_datasets_mos_kfold_som.py
@@ -0,0 +1,137 @@
+"""
+This module implements the top level check function for
+scores calculated by the score of means aggregation
+in k-fold scenarios and mean of scores aggregation on multiple datasets.
+"""
+
+import copy
+
+from ...aggregated import check_aggregated_scores, Experiment
+from ...core import NUMERICAL_TOLERANCE
+
+__all__ = ["check_n_datasets_mos_kfold_som"]
+
+
+def check_n_datasets_mos_kfold_som(
+ evaluations: list,
+ scores: dict,
+ eps,
+ dataset_score_bounds: dict = None,
+ *,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ This function checks the consistency of scores calculated on multiple datasets with k-fold
+ cross-validation, applying score of means aggregation over the folds and mean of scores
+ aggregation over the datasets.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add ``dataset_score_bounds`` when, for example, the minimum
+ and the maximum scores over the datasets are also provided.
+
+ Args:
+ evaluations (list(dict)): the list of evaluation specifications
+ scores (dict(str,float)): the scores to check
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ dataset_score_bounds (None|dict(str,tuple(float,float))): the potential bounds on the
+ scores in the datasets
+ solver_name (None|str): the solver to use
+ timeout (None|int): the timeout for the linear programming solver in seconds
+ verbosity (int): the verbosity of the linear programming solver,
+ 0: silent, 1: verbose.
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_n_datasets_mos_kfold_som
+ >>> evaluation0 = {'dataset': {'p': 39, 'n': 822},
+ 'folding': {'n_folds': 5, 'n_repeats': 3,
+ 'strategy': 'stratified_sklearn'}}
+ >>> evaluation1 = {'dataset': {'dataset_name': 'common_datasets.winequality-white-3_vs_7'},
+ 'folding': {'n_folds': 5, 'n_repeats': 3,
+ 'strategy': 'stratified_sklearn'}}
+ >>> evaluations = [evaluation0, evaluation1]
+ >>> scores = {'acc': 0.312, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
+ >>> result = check_n_datasets_mos_kfold_som(evaluations=evaluations,
+ dataset_score_bounds={'acc': (0.0, 0.5)},
+ eps=1e-4,
+ scores=scores)
+ >>> result['inconsistency']
+ # False
+
+ >>> evaluation0 = {'dataset': {'p': 39, 'n': 822},
+ 'folding': {'n_folds': 5, 'n_repeats': 3,
+ 'strategy': 'stratified_sklearn'}}
+ >>> evaluation1 = {'dataset': {'dataset_name': 'common_datasets.winequality-white-3_vs_7'},
+ 'folding': {'n_folds': 5, 'n_repeats': 3,
+ 'strategy': 'stratified_sklearn'}}
+ >>> evaluations = [evaluation0, evaluation1]
+ >>> scores = {'acc': 0.412, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}
+ >>> result = check_n_datasets_mos_kfold_som(evaluations=evaluations,
+ dataset_score_bounds={'acc': (0.5, 1.0)},
+ eps=1e-4,
+ scores=scores)
+ >>> result['inconsistency']
+ # True
+ """
+
+ if any(evaluation.get("aggregation", "som") != "som" for evaluation in evaluations):
+ raise ValueError(
+ 'the aggregation specified in each dataset must be "rom" or nothing.'
+ )
+
+ if any(
+ evaluation.get("fold_score_bounds") is not None for evaluation in evaluations
+ ):
+ raise ValueError("do not specify fold_score_bounds for a SoM evaluation")
+
+ evaluations = copy.deepcopy(evaluations)
+
+ for evaluation in evaluations:
+ evaluation["aggregation"] = "som"
+
+ experiment = Experiment(
+ evaluations=evaluations,
+ dataset_score_bounds=dataset_score_bounds,
+ aggregation="mos",
+ )
+
+ return check_aggregated_scores(
+ experiment=experiment.to_dict(),
+ scores=scores,
+ eps=eps,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/binary/_check_n_datasets_mos_known_folds_mos.py b/mlscorecheck/check/binary/_check_n_datasets_mos_known_folds_mos.py
new file mode 100644
index 0000000..2d20457
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_n_datasets_mos_known_folds_mos.py
@@ -0,0 +1,129 @@
+"""
+This module implements the top level check function for
+scores calculated by the mean of scores aggregation
+in kfold scenarios and mean of scores aggregation on multiple datasets.
+"""
+
+import copy
+
+from ...aggregated import check_aggregated_scores, Experiment
+from ...core import NUMERICAL_TOLERANCE
+
+__all__ = ["check_n_datasets_mos_known_folds_mos"]
+
+
+def check_n_datasets_mos_known_folds_mos(
+ evaluations: list,
+ scores: dict,
+ eps,
+ dataset_score_bounds: dict = None,
+ *,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ This function checks the consistency of scores calculated by applying k-fold cross validation
+ to N datasets and aggregating the figures over the folds and datasets in the mean of scores
+ fashion.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add ``dataset_score_bounds`` when, for example, the
+ minimum and the maximum scores over the datasets are also provided.
+
+ Args:
+ evaluations (list): The list of evaluation specifications.
+ scores (dict(str,float)): The scores to check.
+ eps (float|dict(str,float)): The numerical uncertainty(ies) of the scores.
+ dataset_score_bounds (None|dict(str,tuple(float,float))): Bounds on the scores for the
+ datasets.
+ solver_name (None|str): The solver to use.
+ timeout (None|int): The timeout for the linear programming solver in seconds.
+ verbosity (int): The verbosity level of the pulp linear programming solver.
+ 0: silent, non-zero: verbose.
+ numerical_tolerance (float): In practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary check_n_datasets_mos_known_folds_mos
+ >>> evaluation0 = {'dataset': {'p': 118, 'n': 95},
+ 'folding': {'folds': [{'p': 22, 'n': 23}, {'p': 96, 'n': 72}]}}
+ >>> evaluation1 = {'dataset': {'p': 781, 'n': 423},
+ 'folding': {'folds': [{'p': 300, 'n': 200}, {'p': 481, 'n': 223}]}}
+ >>> evaluations = [evaluation0, evaluation1]
+ >>> scores = {'acc': 0.61, 'sens': 0.709, 'spec': 0.461, 'bacc': 0.585}
+ >>> result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # False
+
+ >>> evaluation0 = {'dataset': {'p': 118, 'n': 95},
+ 'folding': {'folds': [{'p': 22, 'n': 23}, {'p': 96, 'n': 72}]}}
+ >>> evaluation1 = {'dataset': {'p': 781, 'n': 423},
+ 'folding': {'folds': [{'p': 300, 'n': 200}, {'p': 481, 'n': 223}]}}
+ >>> evaluations = [evaluation0, evaluation1]
+ >>> scores = {'acc': 0.71, 'sens': 0.709, 'spec': 0.461}
+ >>> result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # True
+ """
+ if any(evaluation.get("aggregation", "mos") != "mos" for evaluation in evaluations):
+ raise ValueError(
+ 'the aggregation specified in each dataset must be "mor" or nothing.'
+ )
+ if any(
+ evaluation.get("fold_score_bounds") is not None for evaluation in evaluations
+ ):
+ raise ValueError("do not specify fold_score_bounds through this interface")
+
+ evaluations = copy.deepcopy(evaluations)
+
+ for evaluation in evaluations:
+ evaluation["aggregation"] = "mos"
+
+ experiment = Experiment(
+ evaluations=evaluations,
+ dataset_score_bounds=dataset_score_bounds,
+ aggregation="mos",
+ )
+
+ return check_aggregated_scores(
+ experiment=experiment.to_dict(),
+ scores=scores,
+ eps=eps,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/binary/_check_n_datasets_mos_unknown_folds_mos.py b/mlscorecheck/check/binary/_check_n_datasets_mos_unknown_folds_mos.py
new file mode 100644
index 0000000..58b1352
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_n_datasets_mos_unknown_folds_mos.py
@@ -0,0 +1,173 @@
+"""
+This module implements the top level check function for
+scores calculated by the mean of scores aggregation
+in a kfold scenarios and mean of scores aggregation on multiple datasets,
+but without knowing the fold configurations.
+"""
+
+import copy
+
+import numpy as np
+
+from ._check_n_datasets_mos_known_folds_mos import check_n_datasets_mos_known_folds_mos
+from ._check_1_dataset_unknown_folds_mos import estimate_n_evaluations
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import experiment_kfolds_generator
+
+__all__ = ["check_n_datasets_mos_unknown_folds_mos", "estimate_n_experiments"]
+
+
+def estimate_n_experiments(evaluations: list, available_scores: list = None) -> int:
+ """
+ Estimates the number of estimations with different fold combinations.
+
+ Args:
+ evaluations (list): a list of evaluation specifications
+
+ Returns:
+ int: the estimated number of different fold configurations.
+ """
+ available_scores = [] if available_scores is None else available_scores
+
+ counts = [
+ estimate_n_evaluations(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ available_scores=available_scores,
+ )
+ for evaluation in evaluations
+ ]
+ return np.prod(counts)
+
+
+def check_n_datasets_mos_unknown_folds_mos(
+ evaluations: list,
+ scores: dict,
+ eps,
+ dataset_score_bounds: dict = None,
+ *,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ Checking the consistency of scores calculated in k-fold cross validation on multiple
+ datasets, in mean-of-scores fashion, without knowing the fold configurations.
+ The function generates all possible fold configurations and tests the
+ consistency of each. The scores are inconsistent if all the k-fold configurations
+ lead to inconsistencies identified.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add dataset_score_bounds when, for example, the minimum and
+ the maximum scores over the datasets are also provided.
+
+ Note that depending on the size of the dataset (especially the number of minority instances)
+ and the folding configuration, this test might lead to an untractable number of problems to
+ be solved. Use the function ``estimate_n_experiments`` to get an upper bound estimate
+ on the number of fold combinations.
+
+ The evaluation of possible fold configurations stops when a feasible configuration is found.
+
+ Args:
+ evaluations (list(dict)): the list of evaluation specifications
+ scores (dict(str,float)): the scores to check
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ dataset_score_bounds (None|dict(str,dict(float,float))): bounds on the scores in the
+ datasets
+ solver_name (None|str): the solver to use
+ timeout (None|int): the timeout for the linear programming solver in seconds
+ verbosity (int): the verbosity of the pulp linear programming solver,
+ 0: silent, non-zero: verbose
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list of dictionaries containing the details of the consistency tests. Each
+ entry contains the specification of the folds being tested and the
+ outcome of the ``check_n_datasets_known_folds_mos`` function.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_n_datasets_mos_unknown_folds_mos
+ >>> evaluation0 = {'dataset': {'p': 13, 'n': 73},
+ 'folding': {'n_folds': 4, 'n_repeats': 1}}
+ >>> evaluation1 = {'dataset': {'p': 7, 'n': 26},
+ 'folding': {'n_folds': 3, 'n_repeats': 1}}
+ >>> evaluations = [evaluation0, evaluation1]
+ >>> scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.343}
+ >>> result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # False
+
+ >>> evaluation0 = {'dataset': {'p': 13, 'n': 73},
+ 'folding': {'n_folds': 4, 'n_repeats': 1}}
+ >>> evaluation1 = {'dataset': {'p': 7, 'n': 26},
+ 'folding': {'n_folds': 3, 'n_repeats': 1}}
+ >>> evaluations = [evaluation0, evaluation1]
+ >>> scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.9}
+ >>> result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # True
+ """
+ if any(evaluation.get("aggregation", "mos") != "mos" for evaluation in evaluations):
+ raise ValueError(
+ 'the aggregation specified in each dataset must be "mor" or nothing.'
+ )
+ if any(
+ evaluation.get("fold_score_bounds") is not None for evaluation in evaluations
+ ):
+ raise ValueError("do not specify fold score bounds through this interface")
+
+ evaluations = copy.deepcopy(evaluations)
+
+ for evaluation in evaluations:
+ evaluation["aggregation"] = "mos"
+
+ experiment = {
+ "evaluations": evaluations,
+ "dataset_score_bounds": dataset_score_bounds,
+ "aggregation": "mos",
+ }
+
+ results = {"details": [], "inconsistency": True}
+
+ for experiment in experiment_kfolds_generator(experiment, list(scores.keys())):
+ result = check_n_datasets_mos_known_folds_mos(
+ evaluations=experiment["evaluations"],
+ dataset_score_bounds=experiment.get("dataset_score_bounds"),
+ scores=scores,
+ eps=eps,
+ timeout=timeout,
+ solver_name=solver_name,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
+
+ results["details"].append(result)
+ results["inconsistency"] = results["inconsistency"] and result["inconsistency"]
+
+ if not result["inconsistency"]:
+ break
+
+ return results
diff --git a/mlscorecheck/check/_check_n_datasets_som_kfold_som_scores.py b/mlscorecheck/check/binary/_check_n_datasets_som_kfold_som.py
similarity index 53%
rename from mlscorecheck/check/_check_n_datasets_som_kfold_som_scores.py
rename to mlscorecheck/check/binary/_check_n_datasets_som_kfold_som.py
index 8e9f164..4502629 100644
--- a/mlscorecheck/check/_check_n_datasets_som_kfold_som_scores.py
+++ b/mlscorecheck/check/binary/_check_n_datasets_som_kfold_som.py
@@ -6,24 +6,26 @@
import copy
-from ..core import NUMERICAL_TOLERANCE
-from ..individual import check_scores_tptn_pairs
-from ..aggregated import Experiment
+from ...core import NUMERICAL_TOLERANCE
+from ...individual import check_scores_tptn_pairs
+from ...aggregated import Experiment
-__all__ = ['check_n_datasets_som_kfold_som_scores']
+__all__ = ["check_n_datasets_som_kfold_som"]
-def check_n_datasets_som_kfold_som_scores(evaluations: list,
- scores: dict,
- eps,
- *,
- numerical_tolerance: float = NUMERICAL_TOLERANCE):
+
+def check_n_datasets_som_kfold_som(
+ evaluations: list,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ prefilter_by_pairs: bool = True
+):
"""
Checking the consistency of scores calculated by applying k-fold
cross validation to multiple datasets and aggregating the figures
- over the folds and datasets in the score of means fashion. If
- score bounds are specified and some of the 'acc', 'sens', 'spec' and
- 'bacc' scores are supplied, the linear programming based check is
- executed to see if the bound conditions can be satisfied.
+ over the folds and datasets in the score of means fashion. The test is
+ performed by exhaustively testing all possible confusion matrices.
Args:
evaluations (list(dict)): the specification of the evaluations
@@ -39,19 +41,34 @@ def check_n_datasets_som_kfold_som_scores(evaluations: list,
orders of magnitude smaller than the uncertainty of the
scores. It does ensure that the specificity of the test
is 1, it might slightly decrease the sensitivity.
+ prefilter_by_pairs (bool): whether to do a prefiltering based on the score-pair tp-tn
+ solutions (faster)
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the dataset.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
Raises:
ValueError: if the problem is not specified properly
Examples:
+ >>> from mlscorecheck.check.binary import check_n_datasets_som_kfold_som
>>> evaluation0 = {'dataset': {'p': 389, 'n': 630},
'folding': {'n_folds': 5, 'n_repeats': 2,
'strategy': 'stratified_sklearn'}}
@@ -60,9 +77,9 @@ def check_n_datasets_som_kfold_som_scores(evaluations: list,
'strategy': 'stratified_sklearn'}}
>>> evaluations = [evaluation0, evaluation1]
>>> scores = {'acc': 0.631, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}
- >>> result = check_n_datasets_som_kfold_som_scores(scores=scores,
- evaluations=evaluations,
- eps=1e-3)
+ >>> result = check_n_datasets_som_kfold_som(scores=scores,
+ evaluations=evaluations,
+ eps=1e-3)
>>> result['inconsistency']
# False
@@ -74,29 +91,32 @@ def check_n_datasets_som_kfold_som_scores(evaluations: list,
'strategy': 'stratified_sklearn'}}
>>> evaluations = [evaluation0, evaluation1]
>>> scores = {'acc': 0.731, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}
- >>> result = check_n_datasets_som_kfold_som_scores(scores=scores,
- evaluations=evaluations,
- eps=1e-3)
+ >>> result = check_n_datasets_som_kfold_som(scores=scores,
+ evaluations=evaluations,
+ eps=1e-3)
>>> result['inconsistency']
# True
"""
- if any(evaluation.get('aggregation', 'som') != 'som' for evaluation in evaluations):
- raise ValueError('the aggregation specifications cannot be anything else '\
- 'but "rom"')
+ if any(evaluation.get("aggregation", "som") != "som" for evaluation in evaluations):
+ raise ValueError(
+ "the aggregation specifications cannot be anything else but 'rom'"
+ )
evaluations = copy.deepcopy(evaluations)
for evaluation in evaluations:
- evaluation['aggregation'] = 'som'
+ evaluation["aggregation"] = "som"
# creating the experiment consisting of one single dataset, the
# outer level aggregation can be arbitrary
- experiment = Experiment(evaluations=evaluations,
- aggregation='som')
+ experiment = Experiment(evaluations=evaluations, aggregation="som")
# executing the individual tests
- return check_scores_tptn_pairs(scores=scores,
- eps=eps,
- p=experiment.figures['p'],
- n=experiment.figures['n'],
- numerical_tolerance=numerical_tolerance)
+ return check_scores_tptn_pairs(
+ scores=scores,
+ eps=eps,
+ p=experiment.figures["p"],
+ n=experiment.figures["n"],
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
diff --git a/mlscorecheck/check/binary/_check_n_testsets_mos_no_kfold.py b/mlscorecheck/check/binary/_check_n_testsets_mos_no_kfold.py
new file mode 100644
index 0000000..1c23180
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_n_testsets_mos_no_kfold.py
@@ -0,0 +1,126 @@
+"""
+This module implements the top level check function for
+scores calculated by the mean of scores aggregation over multiple testsets
+with no k-fold cross-validation.
+"""
+
+from ...aggregated import check_aggregated_scores, Experiment, Dataset
+from ...core import NUMERICAL_TOLERANCE
+
+__all__ = ["check_n_testsets_mos_no_kfold"]
+
+
+def check_n_testsets_mos_no_kfold(
+ testsets: list,
+ scores: dict,
+ eps,
+ testset_score_bounds: dict = None,
+ *,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+) -> dict:
+ """
+ This function checks the consistency of scores calculated on multiple testsets with no k-fold
+ and aggregating the figures over the testsets in the mean of scores fashion.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add ``testset_score_bounds`` when, for example, the minimum
+ and the maximum scores over the testsets are also provided.
+
+ Args:
+ testsets (list(dict)): the list of testset specifications
+ scores (dict(str,float)): the scores to check
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ testset_score_bounds (None|dict(str,tuple(float,float))): the potential bounds on the
+ scores in the testsets
+ solver_name (None|str): the solver to use
+ timeout (None|int): the timeout for the linear programming solver in seconds
+ verbosity (int): the verbosity of the linear programming solver,
+ 0: silent, 1: verbose.
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_n_testsets_mos_no_kfold
+ >>> testsets = [{'p': 349, 'n': 50},
+ {'p': 478, 'n': 323},
+ {'p': 324, 'n': 83},
+ {'p': 123, 'n': 145}]
+ >>> scores = {'acc': 0.6441, 'sens': 0.6706, 'spec': 0.3796, 'bacc': 0.5251}
+ >>> result = check_n_testsets_mos_no_kfold(testsets=testsets,
+ eps=1e-4,
+ scores=scores)
+ >>> result['inconsistency']
+ # False
+
+ >>> scores['sens'] = 0.6756
+ >>> result = check_n_datasets_mos_no_kfold(testsets=testsets,
+ eps=1e-4,
+ scores=scores)
+ >>> result['inconsistency']
+ # True
+ """
+
+ datasets = [Dataset(**dataset) for dataset in testsets]
+
+ evaluations = [
+ {
+ "dataset": dataset.to_dict(),
+ "folding": {
+ "folds": [
+ {
+ "p": dataset.p,
+ "n": dataset.n,
+ "identifier": f"{dataset.identifier}_{idx}",
+ }
+ ]
+ },
+ "aggregation": "mos",
+ }
+ for idx, dataset in enumerate(datasets)
+ ]
+
+ experiment = Experiment(
+ evaluations=evaluations,
+ dataset_score_bounds=testset_score_bounds,
+ aggregation="mos",
+ )
+
+ return check_aggregated_scores(
+ experiment=experiment.to_dict(),
+ scores=scores,
+ eps=eps,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/binary/_check_n_testsets_som_no_kfold.py b/mlscorecheck/check/binary/_check_n_testsets_som_no_kfold.py
new file mode 100644
index 0000000..c02f74c
--- /dev/null
+++ b/mlscorecheck/check/binary/_check_n_testsets_som_no_kfold.py
@@ -0,0 +1,117 @@
+"""
+This module implements the top level check function for
+scores calculated by the score-of-means aggregation
+over multiple testsets (with no kfold).
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...individual import check_scores_tptn_pairs
+from ...aggregated import Experiment, Dataset
+
+__all__ = ["check_n_testsets_som_no_kfold"]
+
+
+def check_n_testsets_som_no_kfold(
+ testsets: list,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ prefilter_by_pairs: bool = True,
+):
+ """
+ Checking the consistency of scores calculated by aggregating the figures
+ over testsets in the score of means fashion, without k-folding.
+
+ The test is performed by exhaustively testing all possible confusion matrices.
+
+ Args:
+ datasets (list(dict)): the specification of the evaluations
+ scores (dict(str,float)): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+ prefilter_by_pairs (bool): whether to prefilter the solution space by pair
+ solutions when possible to speed up the process
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.binary import check_n_datasets_som_no_kfold
+ >>> testsets = [{'p': 405, 'n': 223}, {'p': 3, 'n': 422}, {'p': 109, 'n': 404}]
+ >>> scores = {'acc': 0.4719, 'npv': 0.6253, 'f1p': 0.3091}
+ >>> result = check_n_datasets_som_no_kfold(testsets=testsets,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # False
+
+ >>> scores['npv'] = 0.6263
+ >>> result = check_n_datasets_som_no_kfold(testsets=testsets,
+ scores=scores,
+ eps=1e-3)
+ >>> result['inconsistency']
+ # True
+ """
+
+ datasets = [Dataset(**dataset) for dataset in testsets]
+
+ evaluations = [
+ {
+ "dataset": dataset.to_dict(),
+ "folding": {
+ "folds": [
+ {
+ "p": dataset.p,
+ "n": dataset.n,
+ "identifier": f"{dataset.identifier}_{idx}",
+ }
+ ]
+ },
+ "aggregation": "mos",
+ }
+ for idx, dataset in enumerate(datasets)
+ ]
+
+ # creating the experiment consisting of one single dataset, the
+ # outer level aggregation can be arbitrary
+ experiment = Experiment(evaluations=evaluations, aggregation="som")
+
+ # executing the individual tests
+ return check_scores_tptn_pairs(
+ scores=scores,
+ eps=eps,
+ p=experiment.figures["p"],
+ n=experiment.figures["n"],
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
diff --git a/mlscorecheck/bundles/__init__.py b/mlscorecheck/check/bundles/__init__.py
similarity index 100%
rename from mlscorecheck/bundles/__init__.py
rename to mlscorecheck/check/bundles/__init__.py
diff --git a/mlscorecheck/bundles/ehg/__init__.py b/mlscorecheck/check/bundles/ehg/__init__.py
similarity index 100%
rename from mlscorecheck/bundles/ehg/__init__.py
rename to mlscorecheck/check/bundles/ehg/__init__.py
diff --git a/mlscorecheck/bundles/ehg/_tpehg.py b/mlscorecheck/check/bundles/ehg/_tpehg.py
similarity index 71%
rename from mlscorecheck/bundles/ehg/_tpehg.py
rename to mlscorecheck/check/bundles/ehg/_tpehg.py
index 8bb0c38..e3008fe 100644
--- a/mlscorecheck/bundles/ehg/_tpehg.py
+++ b/mlscorecheck/check/bundles/ehg/_tpehg.py
@@ -2,9 +2,9 @@
Test functions for the EHG problem
"""
-from ...core import NUMERICAL_TOLERANCE
-from ...experiments import get_experiment
-from ...check import check_1_dataset_unknown_folds_mos_scores
+from ....core import NUMERICAL_TOLERANCE
+from ....experiments import get_experiment
+from ...binary import check_1_dataset_unknown_folds_mos
__all__ = ['check_tpehg']
@@ -22,7 +22,8 @@ def check_tpehg(scores: dict,
Checks the cross-validated TPEHG scores
Args:
- scores (dict(str,float)): the dictionary of scores
+ scores (dict(str,float)): the dictionary of scores (supports only 'acc', 'sens', 'spec',
+ 'bacc')
eps (float|dict(str,float)): the numerical uncertainties
n_folds (int): the number of folds
n_repeats (int): the number of repetitions
@@ -39,10 +40,20 @@ def check_tpehg(scores: dict,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: the result of the consistency testing
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list of dictionaries containing the details of the consistency tests. Each
+ entry contains the specification of the folds being tested and the
+ outcome of the ``check_1_dataset_known_folds_mos`` function.
Examples:
- >>> from mlscorecheck.bundles.ehg import check_tpehg
+ >>> from mlscorecheck.check.bundles.ehg import check_tpehg
>>> # the 5-fold cross-validation scores reported in the paper
>>> scores = {'acc': 0.9447, 'sens': 0.9139, 'spec': 0.9733}
>>> eps = 0.0001
@@ -56,7 +67,7 @@ def check_tpehg(scores: dict,
evaluation = {'dataset': get_experiment('ehg.tpehg'),
'folding': {'n_folds': n_folds, 'n_repeats': n_repeats}}
- return check_1_dataset_unknown_folds_mos_scores(scores=scores,
+ return check_1_dataset_unknown_folds_mos(scores=scores,
eps=eps,
dataset=evaluation['dataset'],
folding=evaluation['folding'],
diff --git a/mlscorecheck/bundles/retina/__init__.py b/mlscorecheck/check/bundles/retina/__init__.py
similarity index 100%
rename from mlscorecheck/bundles/retina/__init__.py
rename to mlscorecheck/check/bundles/retina/__init__.py
diff --git a/mlscorecheck/bundles/retina/_chasedb1.py b/mlscorecheck/check/bundles/retina/_chasedb1.py
similarity index 68%
rename from mlscorecheck/bundles/retina/_chasedb1.py
rename to mlscorecheck/check/bundles/retina/_chasedb1.py
index 92e6163..fb0a0e3 100644
--- a/mlscorecheck/bundles/retina/_chasedb1.py
+++ b/mlscorecheck/check/bundles/retina/_chasedb1.py
@@ -2,11 +2,11 @@
This module implements the tests for the CHASE_DB1 retina vessel segmentation dataset
"""
-from ...experiments import get_experiment
-from ...check import check_n_testsets_som_no_kfold_scores
-from ...check import check_n_testsets_mos_no_kfold_scores
-from ...check import check_1_testset_no_kfold_scores
-from ...core import NUMERICAL_TOLERANCE
+from ....experiments import get_experiment
+from ...binary import check_n_testsets_som_no_kfold
+from ...binary import check_n_testsets_mos_no_kfold
+from ...binary import check_1_testset_no_kfold
+from ....core import NUMERICAL_TOLERANCE
__all__ = ['check_chasedb1_vessel_image',
'check_chasedb1_vessel_aggregated',
@@ -43,13 +43,14 @@ def check_chasedb1_vessel_aggregated_mos(imageset,
numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
Checking the consistency of scores with calculated for some images of
- the CHASEDB1 dataset with the mean-of-scores aggregation.
+ the CHASEDB1 dataset with the mean of scores aggregation.
Args:
imageset (str|list): 'all' if all images are used, or a list of identifiers of
images (e.g. ['11R', '07L'])
annotator (str): the annotation to be used ('manual1'/'manual2')
- scores (dict): the scores to check
+ scores (dict): the scores to check (supports only 'acc', 'sens',
+ 'spec', 'bacc')
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
score_bounds (dict(str,tuple(float,float))): the potential bounds on the scores
of the images
@@ -64,26 +65,28 @@ def check_chasedb1_vessel_aggregated_mos(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
"""
data = get_experiment('retina.chase_db1')
testsets = _filter_chasedb1(data, imageset, annotator)
- return check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
testset_score_bounds=score_bounds,
@@ -106,7 +109,12 @@ def check_chasedb1_vessel_aggregated_som(imageset,
imageset (str|list): 'all' if all images are used, or a list of identifiers of
images (e.g. ['11R', '07L'])
annotator (str): the annotation to be used ('manual1'/'manual2')
- scores (dict): the scores to check
+ scores (dict): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -115,21 +123,30 @@ def check_chasedb1_vessel_aggregated_som(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
data = get_experiment('retina.chase_db1')
testsets = _filter_chasedb1(data, imageset, annotator)
- return check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
@@ -146,7 +163,8 @@ def check_chasedb1_vessel_aggregated(imageset,
verbosity: int = 1,
numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
- Testing the scores calculated for the CHASEDB1 dataset
+ Testing the scores calculated for the CHASEDB1 dataset with both assumptions
+ on the mode of aggregation.
Args:
imageset (str|list): 'all' if all images are used, or a list of identifiers of
@@ -167,12 +185,15 @@ def check_chasedb1_vessel_aggregated(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_chasedb1_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_chasedb1_vessel_aggregated
>>> scores = {'acc': 0.5063, 'sens': 0.4147, 'spec': 0.5126}
>>> k = 4
>>> results = check_chasedb1_vessel_aggregated(imageset='all',
@@ -219,7 +240,12 @@ def check_chasedb1_vessel_image(image_identifier: str,
Args:
image_identifier (str): the identifier of the image (like "11R")
annotator (str): the annotation to use ('manual1'/'manual2')
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -228,12 +254,27 @@ def check_chasedb1_vessel_image(image_identifier: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
Examples:
- >>> from mlscorecheck.bundles.retina import check_chasedb1_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_chasedb1_vessel_image
>>> img_identifier = '11R'
>>> scores = {'acc': 0.4457, 'sens': 0.0051, 'spec': 0.4706}
>>> results = check_chasedb1_vessel_image(image_identifier=img_identifier,
@@ -247,7 +288,7 @@ def check_chasedb1_vessel_image(image_identifier: str,
image = [image for image in images[annotator]['images']
if image['identifier'] == image_identifier][0]
- return check_1_testset_no_kfold_scores(testset=image,
+ return check_1_testset_no_kfold(testset=image,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
diff --git a/mlscorecheck/bundles/retina/_diaretdb0.py b/mlscorecheck/check/bundles/retina/_diaretdb0.py
similarity index 79%
rename from mlscorecheck/bundles/retina/_diaretdb0.py
rename to mlscorecheck/check/bundles/retina/_diaretdb0.py
index 870d49a..7124fe3 100644
--- a/mlscorecheck/bundles/retina/_diaretdb0.py
+++ b/mlscorecheck/check/bundles/retina/_diaretdb0.py
@@ -2,10 +2,10 @@
Tests for the DIARETDB0 dataset
"""
-from mlscorecheck.core import NUMERICAL_TOLERANCE
-from mlscorecheck.experiments import get_experiment
-from mlscorecheck.check import (check_n_testsets_mos_no_kfold_scores,
- check_n_testsets_som_no_kfold_scores)
+from ....core import NUMERICAL_TOLERANCE
+from ....experiments import get_experiment
+from ...binary import (check_n_testsets_mos_no_kfold,
+ check_n_testsets_som_no_kfold)
__all__ = ['_prepare_configuration_diaretdb0',
'check_diaretdb0_class',
@@ -74,7 +74,12 @@ def check_diaretdb0_class_som(subset: str,
class_name (str|list): the name of the class being evaluated ('neovascularisation'|
'hardexudates'|'softexudates'|'hemorrhages'|'redsmalldots'), a list if
a list of classes is treated as positive
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -83,21 +88,34 @@ def check_diaretdb0_class_som(subset: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
testsets = _prepare_configuration_diaretdb0(subset,
batch,
class_name)
- return check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
- numerical_tolerance=numerical_tolerance)
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=True)
def check_diaretdb0_class_mos(subset: str,
@@ -130,7 +148,8 @@ def check_diaretdb0_class_mos(subset: str,
class_name (str|list): the name of the class being evaluated ('neovascularisation'|
'hardexudates'|'softexudates'|'hemorrhages'|'redsmalldots'), a list if
a list of classes is treated as positive
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested (supports only 'acc', 'sens', 'spec',
+ 'bacc')
eps (float): the numerical uncertainty
score_bounds (dict(str,tuple(float,float))): the potential bounds on the scores
of the images
@@ -145,23 +164,28 @@ def check_diaretdb0_class_mos(subset: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
"""
testsets = _prepare_configuration_diaretdb0(subset,
batch,
class_name)
- return check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
@@ -216,12 +240,15 @@ def check_diaretdb0_class(subset: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_diaretdb0_class
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb0_class
>>> scores = {'acc': 0.4271, 'sens': 0.406, 'spec': 0.4765}
>>> results = check_diaretdb0_class(subset='test',
batch='all',
diff --git a/mlscorecheck/bundles/retina/_diaretdb1.py b/mlscorecheck/check/bundles/retina/_diaretdb1.py
similarity index 81%
rename from mlscorecheck/bundles/retina/_diaretdb1.py
rename to mlscorecheck/check/bundles/retina/_diaretdb1.py
index dfc88ac..c1ef1d1 100644
--- a/mlscorecheck/bundles/retina/_diaretdb1.py
+++ b/mlscorecheck/check/bundles/retina/_diaretdb1.py
@@ -2,11 +2,11 @@
This module implements tests for the DIARETDB1 dataset
"""
-from ...experiments import get_experiment
-from ...check import check_1_testset_no_kfold_scores
-from ...check import check_n_testsets_mos_no_kfold_scores
-from ...check import check_n_testsets_som_no_kfold_scores
-from ...core import NUMERICAL_TOLERANCE, logger
+from ....experiments import get_experiment
+from ...binary import check_1_testset_no_kfold
+from ...binary import check_n_testsets_mos_no_kfold
+from ...binary import check_n_testsets_som_no_kfold
+from ....core import NUMERICAL_TOLERANCE, logger
__all__ = ['_prepare_configuration_diaretdb1',
'check_diaretdb1_class',
@@ -125,7 +125,12 @@ def check_diaretdb1_class(*,
subset (str): the subset to be used ('train'/'test'), typically 'test'
class_name (str|list): the name or list of names of classes used as "positive"
confidence (float): the confidence threshold, typically 0.75
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -134,15 +139,27 @@ def check_diaretdb1_class(*,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
Examples:
- >>> from mlscorecheck.bundles.retina import check_diaretdb1_class
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb1_class
>>> scores = {'acc': 0.3115, 'sens': 1.0, 'spec': 0.0455, 'f1p': 0.4474}
>>> results = check_diaretdb1_class(subset='test',
class_name=['hardexudates', 'softexudates'],
@@ -158,7 +175,7 @@ def check_diaretdb1_class(*,
assumption='all',
confidence=confidence)
- return check_1_testset_no_kfold_scores(testset=testset,
+ return check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance)
@@ -181,7 +198,12 @@ def check_diaretdb1_segmentation_image_assumption(*,
class_name (str|list): the name or list of names of classes used as "positive"
assumption (str): the assumption on the region of evaluation ('fov'/'all')
confidence (float): the confidence threshold, typically 0.75
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -190,12 +212,24 @@ def check_diaretdb1_segmentation_image_assumption(*,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
testset_test = _prepare_configuration_diaretdb1(subset='test',
class_name=class_name,
@@ -210,7 +244,7 @@ def check_diaretdb1_segmentation_image_assumption(*,
testset = [tset for tset in testset_test + testset_train
if tset['identifier'] == image_identifier][0]
- return check_1_testset_no_kfold_scores(testset=testset,
+ return check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance)
@@ -232,7 +266,12 @@ def check_diaretdb1_segmentation_image(*,
image_identifier (str): the identifier of the image to be tested (e.g. '001')
class_name (str|list): the name or list of names of classes used as "positive"
confidence (float): the confidence threshold, typically 0.75
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -241,12 +280,15 @@ def check_diaretdb1_segmentation_image(*,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_diaretdb1_segmentation_image
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb1_segmentation_image
>>> scores = {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}
>>> results = check_diaretdb1_segmentation_image(image_identifier='005',
class_name=['hardexudates', 'softexudates'],
@@ -322,9 +364,12 @@ def check_diaretdb1_segmentation_aggregated_assumption(*,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
"""
testsets = _prepare_configuration_diaretdb1(subset=subset,
@@ -335,14 +380,14 @@ def check_diaretdb1_segmentation_aggregated_assumption(*,
only_valid=only_valid)
results = {}
- results['details_som'] = check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ results['details_som'] = check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance)
results['inconsistency'] = {'inconsistency_som': results['details_som']['inconsistency']}
if not (any(testset['p'] == 0 for testset in testsets) and 'sens' in scores):
- results['details_mos'] = check_n_testsets_mos_no_kfold_scores(
+ results['details_mos'] = check_n_testsets_mos_no_kfold(
testsets=testsets,
scores=scores,
eps=eps,
@@ -373,7 +418,8 @@ def check_diaretdb1_segmentation_aggregated(*,
"""
Tests the scores describing the segmentation of multiple images of DIARETDB1 in an aggregated
way. This test function supports binary the testing of binary subproblems (for example, the
- pixels of the 'hardexudates' class being segmented in an image).
+ pixels of the 'hardexudates' class being segmented in an image). The test evaluates
+ both assumption on the region of evaluation.
Args:
subset (str|list): the subset of images to be used ('train'/'test') or the list of
@@ -400,12 +446,15 @@ def check_diaretdb1_segmentation_aggregated(*,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_diaretdb1_segmentation_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_diaretdb1_segmentation_aggregated
>>> scores = {'acc': 0.7143, 'sens': 0.3775, 'spec': 0.7244}
>>> results = check_diaretdb1_segmentation_aggregated(subset='test',
class_name='hardexudates',
diff --git a/mlscorecheck/bundles/retina/_drishti_gs.py b/mlscorecheck/check/bundles/retina/_drishti_gs.py
similarity index 72%
rename from mlscorecheck/bundles/retina/_drishti_gs.py
rename to mlscorecheck/check/bundles/retina/_drishti_gs.py
index b66ae07..a15d166 100644
--- a/mlscorecheck/bundles/retina/_drishti_gs.py
+++ b/mlscorecheck/check/bundles/retina/_drishti_gs.py
@@ -2,11 +2,11 @@
This module implements tests for the DRISHTI_GS dataset
"""
-from ...experiments import get_experiment
-from ...core import NUMERICAL_TOLERANCE
-from ...check import check_1_testset_no_kfold_scores
-from ...check import check_n_testsets_mos_no_kfold_scores
-from ...check import check_n_testsets_som_no_kfold_scores
+from ....experiments import get_experiment
+from ....core import NUMERICAL_TOLERANCE
+from ...binary import check_1_testset_no_kfold
+from ...binary import check_n_testsets_mos_no_kfold
+from ...binary import check_n_testsets_som_no_kfold
__all__ = ['_prepare_testsets_drishti_gs',
'check_drishti_gs_segmentation_image',
@@ -70,7 +70,12 @@ def check_drishti_gs_segmentation_image(image_identifier: str,
confidence (float): the confidence level (in [0,1]), used for thresholding
the soft segmentation ground truth image at threshold*255
target (str): the target anatomical part ('OD'/'OC')
- scores (dict): the scores to check
+ scores (dict): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -79,15 +84,27 @@ def check_drishti_gs_segmentation_image(image_identifier: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible
- with all scores. Under the key ``prefiltering_details`` one finds the results of the
- prefiltering by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
Examples:
- >>> from mlscorecheck.bundles.retina import check_drishti_gs_segmentation_image
+ >>> from mlscorecheck.check.bundles.retina import check_drishti_gs_segmentation_image
>>> scores = {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}
>>> results = check_drishti_gs_segmentation_image(image_identifier='053',
confidence=0.75,
@@ -101,7 +118,7 @@ def check_drishti_gs_segmentation_image(image_identifier: str,
target=target,
confidence=confidence)[0]
- return check_1_testset_no_kfold_scores(testset=testset,
+ return check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance)
@@ -126,7 +143,8 @@ def check_drishti_gs_segmentation_aggregated_mos(subset,
confidence (float): the confidence level (in [0,1]), used for thresholding
the soft segmentation ground truth image at threshold*255
target (str): the target anatomical part ('OD'/'OC')
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested (supports only 'acc', 'sens', 'spec',
+ 'bacc')
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
score_bounds (dict(str,tuple(float,float))): the potential bounds on the scores
of the images
@@ -141,23 +159,28 @@ def check_drishti_gs_segmentation_aggregated_mos(subset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
"""
testsets = _prepare_testsets_drishti_gs(subset=subset,
target=target,
confidence=confidence)
- return check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
testset_score_bounds=score_bounds,
@@ -182,7 +205,12 @@ def check_drishti_gs_segmentation_aggregated_som(subset: str,
confidence (float): the confidence level (in [0,1]), used for thresholding
the soft segmentation ground truth image at threshold*255
target (str): the target anatomical part ('OD'/'OC')
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -191,21 +219,34 @@ def check_drishti_gs_segmentation_aggregated_som(subset: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
testsets = _prepare_testsets_drishti_gs(subset=subset,
target=target,
confidence=confidence)
- return check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
- numerical_tolerance=numerical_tolerance)
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=True)
def check_drishti_gs_segmentation_aggregated(subset: str,
confidence: float,
@@ -242,12 +283,15 @@ def check_drishti_gs_segmentation_aggregated(subset: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_drishti_gs_segmentation_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_drishti_gs_segmentation_aggregated
>>> scores = {'acc': 0.4767, 'sens': 0.4845, 'spec': 0.4765, 'f1p': 0.0512}
>>> results = check_drishti_gs_segmentation_aggregated(subset='test',
confidence=0.75,
diff --git a/mlscorecheck/bundles/retina/_drive.py b/mlscorecheck/check/bundles/retina/_drive.py
similarity index 66%
rename from mlscorecheck/bundles/retina/_drive.py
rename to mlscorecheck/check/bundles/retina/_drive.py
index c0745ff..8a6cb9e 100644
--- a/mlscorecheck/bundles/retina/_drive.py
+++ b/mlscorecheck/check/bundles/retina/_drive.py
@@ -3,11 +3,11 @@
segmentation drive dataset
"""
-from ...core import NUMERICAL_TOLERANCE
-from ...experiments import get_experiment
-from ...check import (check_1_testset_no_kfold_scores,
- check_n_testsets_mos_no_kfold_scores,
- check_n_testsets_som_no_kfold_scores)
+from ....core import NUMERICAL_TOLERANCE
+from ....experiments import get_experiment
+from ...binary import (check_1_testset_no_kfold,
+ check_n_testsets_mos_no_kfold,
+ check_n_testsets_som_no_kfold)
__all__ = ['check_drive_vessel_aggregated_mos_assumption',
'check_drive_vessel_aggregated_som_assumption',
@@ -53,15 +53,17 @@ def check_drive_vessel_aggregated_mos_assumption(imageset,
verbosity: int = 1,
numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
- Checking the consistency of scores with calculated for some images of
- the DRIVE dataset with the mean-of-scores aggregation.
+ Checking the consistency of scores calculated for some images of
+ the DRIVE dataset with the mean of scores aggregation and a particular
+ assumption on the region of evaluation.
Args:
imageset (str|list): 'train'/'test' for all images in the train or test set, or a list of
identifiers of images (e.g. ['21', '22'])
assumption (str): the assumption on the region of evaluation to test ('fov'/'all')
annotator (int): the annotation to be used (1/2) (typically annotator 1 is used in papers)
- scores (dict): the scores to check
+ scores (dict): the scores to check the scores to check (supports only 'acc', 'sens',
+ 'spec', 'bacc')
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
score_bounds (dict(str,tuple(float,float))): the potential bounds on the scores
of the images
@@ -76,26 +78,28 @@ def check_drive_vessel_aggregated_mos_assumption(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
"""
data = get_experiment('retina.drive')
testsets = _filter_drive(data, imageset, annotator, assumption)
- return check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
testset_score_bounds=score_bounds,
@@ -114,14 +118,20 @@ def check_drive_vessel_aggregated_som_assumption(imageset,
numerical_tolerance=NUMERICAL_TOLERANCE):
"""
Tests the consistency of scores calculated on the DRIVE dataset using
- the score-of-means aggregation.
+ the score of means aggregation and a particular assumption on the region
+ of evaluation.
Args:
imageset (str|list): 'train'/'test' for all images in the train or test set, or a list of
identifiers of images (e.g. ['21', '22'])
assumption (str): the assumption on the region of evaluation to test ('fov'/'all')
annotator (int): the annotation to be used (1/2) (typically annotator 1 is used in papers)
- scores (dict): the scores to check
+ scores (dict): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -130,21 +140,30 @@ def check_drive_vessel_aggregated_som_assumption(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
data = get_experiment('retina.drive')
testsets = _filter_drive(data, imageset, annotator, assumption)
- return check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
@@ -159,23 +178,45 @@ def check_drive_vessel_image_assumption(image_identifier: str,
*,
numerical_tolerance: float = NUMERICAL_TOLERANCE):
"""
- Testing the scores calculated for one image of the DRIVE dataset
+ Testing the scores calculated for one image of the DRIVE dataset with a particular
+ assumption on the region of evaluation.
Args:
image_identifier (str): the identifier of the image (like "21")
assumption (str): the assumption on the region of evaluation to test ('fov'/'all')
annotator (int): the annotation to use (1, 2) (typically annotator 1 is used in papers)
- scores (dict(str,float)): the scores to be tested
- eps (float): the numerical uncertainty
- numerical_tolerance (float): the additional numerical tolerance
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible
- with all scores. Under the key ``prefiltering_details`` one finds the results of the
- prefiltering by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
images = get_experiment('retina.drive')
testset = [image for image in images[(annotator, assumption)]['train']['images']
@@ -184,7 +225,7 @@ def check_drive_vessel_image_assumption(image_identifier: str,
if image['identifier'] == image_identifier]
testset = testset[0]
- return check_1_testset_no_kfold_scores(testset=testset,
+ return check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
@@ -202,7 +243,11 @@ def check_drive_vessel_aggregated(imageset,
verbosity: int = 1,
numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
- Testing the scores calculated for the DRIVE dataset with all assumptions
+ Testing the scores calculated for the DRIVE dataset with both assumptions
+ regarding the region of evaluation (using the FoV or all pixels of the images).
+
+ The strength of the test can be improved by specifying the ``score_bounds`` (minimum
+ and maximum scores) for the images when available.
Args:
imageset (str|list): 'train'/'test' for all images in the train or test set, or a list of
@@ -223,12 +268,15 @@ def check_drive_vessel_aggregated(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_drive_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_drive_vessel_aggregated
>>> scores = {'acc': 0.9494, 'sens': 0.7450, 'spec': 0.9793}
>>> k = 4
>>> results = check_drive_vessel_aggregated(scores=scores,
@@ -278,12 +326,17 @@ def check_drive_vessel_image(image_identifier: str,
numerical_tolerance: float = NUMERICAL_TOLERANCE):
"""
Testing the scores calculated for one image of the DRIVE dataset with
- both assumptions on the region of evaluation ('fov'/'all')
+ both assumptions on the region of evaluation ('fov'/'all').
Args:
image_identifier (str): the identifier of the image (like "21")
annotator (int): the annotation to use (1, 2) (typically annotator 1 is used in papers)
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -292,12 +345,15 @@ def check_drive_vessel_image(image_identifier: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: A summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``'details*'``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_drive_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_drive_vessel_image
>>> scores = {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}
>>> identifier = '01'
>>> k = 4
diff --git a/mlscorecheck/bundles/retina/_hrf.py b/mlscorecheck/check/bundles/retina/_hrf.py
similarity index 66%
rename from mlscorecheck/bundles/retina/_hrf.py
rename to mlscorecheck/check/bundles/retina/_hrf.py
index 5855d14..b8297a2 100644
--- a/mlscorecheck/bundles/retina/_hrf.py
+++ b/mlscorecheck/check/bundles/retina/_hrf.py
@@ -3,11 +3,11 @@
segmentation HRF dataset
"""
-from ...core import NUMERICAL_TOLERANCE
-from ...experiments import get_experiment
-from ...check import (check_1_testset_no_kfold_scores,
- check_n_testsets_mos_no_kfold_scores,
- check_n_testsets_som_no_kfold_scores)
+from ....core import NUMERICAL_TOLERANCE
+from ....experiments import get_experiment
+from ...binary import (check_1_testset_no_kfold,
+ check_n_testsets_mos_no_kfold,
+ check_n_testsets_som_no_kfold)
__all__ = ['check_hrf_vessel_aggregated_mos_assumption',
'check_hrf_vessel_aggregated_som_assumption',
@@ -51,12 +51,13 @@ def check_hrf_vessel_aggregated_mos_assumption(imageset,
numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
Checking the consistency of scores with calculated for some images of
- the HRF dataset with the mean-of-scores aggregation.
+ the HRF dataset with the mean of scores aggregation and an assumption on
+ the region of evaluation.
Args:
imageset (str|list): 'all' or the list of identifiers of images (e.g. ['13_h', '01_g'])
assumption (str): the assumption on the region of evaluation to test ('fov'/'all')
- scores (dict): the scores to check
+ scores (dict): the scores to check (supports only 'acc', 'sens', 'spec', 'bacc')
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
score_bounds (dict(str,tuple(float,float))): the potential bounds on the scores
of the images
@@ -71,26 +72,28 @@ def check_hrf_vessel_aggregated_mos_assumption(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
"""
data = get_experiment('retina.hrf')
testsets = _filter_hrf(data, imageset, assumption)
- return check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
testset_score_bounds=score_bounds,
@@ -107,12 +110,17 @@ def check_hrf_vessel_aggregated_som_assumption(imageset,
numerical_tolerance=NUMERICAL_TOLERANCE):
"""
Tests the consistency of scores calculated on the HRF dataset using
- the score-of-means aggregation.
+ the score-of-means aggregation and an assumption on the region of evaluation.
Args:
imageset (str|list): 'all' or the list of identifiers of images (e.g. ['13_h', '01_g'])
assumption (str): the assumption on the region of evaluation to test ('fov'/'all')
- scores (dict): the scores to check
+ scores (dict): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -121,21 +129,30 @@ def check_hrf_vessel_aggregated_som_assumption(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
data = get_experiment('retina.hrf')
testsets = _filter_hrf(data, imageset, assumption)
- return check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
@@ -149,22 +166,40 @@ def check_hrf_vessel_image_assumption(image_identifier: str,
*,
numerical_tolerance: float = NUMERICAL_TOLERANCE):
"""
- Testing the scores calculated for one image of the HRF dataset
+ Testing the scores calculated for one image of the HRF dataset using an
+ assumption on the region of evaluation.
Args:
image_identifier (str): the identifier of the image (like "01_g")
assumption (str): the assumption on the region of evaluation to test ('fov'/'all')
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): the additional numerical tolerance
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible
- with all scores. Under the key ``prefiltering_details`` one finds the results of the
- prefiltering by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
images = get_experiment('retina.hrf')
testset = [image for image in images[assumption]['images']
@@ -172,7 +207,7 @@ def check_hrf_vessel_image_assumption(image_identifier: str,
testset = testset[0]
- return check_1_testset_no_kfold_scores(testset=testset,
+ return check_1_testset_no_kfold(testset=testset,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
@@ -189,7 +224,8 @@ def check_hrf_vessel_aggregated(imageset,
verbosity: int = 1,
numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
- Testing the scores calculated for the HRF dataset with all assumptions
+ Testing the scores calculated for the HRF dataset with both assumptions on the
+ region of evaluation ('fov'/'all') and both aggregation methods ('mean of scores',
Args:
imageset (str|list): 'all' or the list of identifiers of images (e.g. ['13_h', '01_g'])
@@ -208,12 +244,15 @@ def check_hrf_vessel_aggregated(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_hrf_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_hrf_vessel_aggregated
>>> scores = {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}
>>> k = 4
>>> results = check_hrf_vessel_aggregated(scores=scores,
@@ -263,7 +302,12 @@ def check_hrf_vessel_image(image_identifier: str,
Args:
image_identifier (str): the identifier of the image (like "01_g")
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -272,12 +316,15 @@ def check_hrf_vessel_image(image_identifier: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_hrf_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_hrf_vessel_image
>>> scores = {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}
>>> identifier = '13_h'
>>> k = 4
diff --git a/mlscorecheck/bundles/retina/_stare.py b/mlscorecheck/check/bundles/retina/_stare.py
similarity index 68%
rename from mlscorecheck/bundles/retina/_stare.py
rename to mlscorecheck/check/bundles/retina/_stare.py
index 0b895f5..30eea19 100644
--- a/mlscorecheck/bundles/retina/_stare.py
+++ b/mlscorecheck/check/bundles/retina/_stare.py
@@ -2,11 +2,11 @@
This module implements the tests for the STARE retina vessel segmentation dataset
"""
-from ...experiments import get_experiment
-from ...check import check_n_testsets_som_no_kfold_scores
-from ...check import check_n_testsets_mos_no_kfold_scores
-from ...check import check_1_testset_no_kfold_scores
-from ...core import NUMERICAL_TOLERANCE
+from ....experiments import get_experiment
+from ...binary import check_n_testsets_som_no_kfold
+from ...binary import check_n_testsets_mos_no_kfold
+from ...binary import check_1_testset_no_kfold
+from ....core import NUMERICAL_TOLERANCE
__all__ = ['check_stare_vessel_image',
'check_stare_vessel_aggregated',
@@ -42,14 +42,14 @@ def check_stare_vessel_aggregated_mos(imageset,
verbosity: int = 1,
numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
"""
- Checking the consistency of scores with calculated for some images of
- the STARE dataset with the mean-of-scores aggregation.
+ Checking the consistency of scores calculated for some images of
+ the STARE dataset with the mean of scores aggregation.
Args:
imageset (str|list): 'all' if all images are used, or a list of identifiers of
images (e.g. ['im0082', 'im0235'])
annotator (str): the annotation to be used ('ah'/'vk')
- scores (dict): the scores to check
+ scores (dict): the scores to check (supports only 'acc', 'sens', 'spec', 'bacc')
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
score_bounds (dict(str,tuple(float,float))): the potential bounds on the scores
of the images
@@ -64,26 +64,28 @@ def check_stare_vessel_aggregated_mos(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: the dictionary of the results of the analysis, the
- ``inconsistency`` entry indicates if inconsistencies have
- been found. The aggregated_results entry is empty if
- the execution of the linear programming based check was
- unnecessary. The result has four more keys. Under ``lp_status``
- one finds the status of the lp solver, under ``lp_configuration_scores_match``
- one finds a flag indicating if the scores from the lp configuration
- match the scores provided, ``lp_configuration_bounds_match`` indicates
- if the specified bounds match the actual figures and finally
- ``lp_configuration`` contains the actual configuration of the
- linear programming solver.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
"""
data = get_experiment('retina.stare')
testsets = _filter_stare(data, imageset, annotator)
- return check_n_testsets_mos_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_mos_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
testset_score_bounds=score_bounds,
@@ -106,7 +108,12 @@ def check_stare_vessel_aggregated_som(imageset,
imageset (str|list): 'all' if all images are used, or a list of identifiers of
images (e.g. ['im0082', 'im0235'])
annotator (str): the annotation to be used ('ah'/'vk')
- scores (dict): the scores to check
+ scores (dict): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -115,21 +122,30 @@ def check_stare_vessel_aggregated_som(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible with
- all scores. Under the key ``prefiltering_details`` one finds the results of the prefiltering
- by using the solutions for the score pairs.
-
- Raises:
- ValueError: if the problem is not specified properly
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
"""
data = get_experiment('retina.stare')
testsets = _filter_stare(data, imageset, annotator)
- return check_n_testsets_som_no_kfold_scores(testsets=testsets,
+ return check_n_testsets_som_no_kfold(testsets=testsets,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
@@ -167,12 +183,15 @@ def check_stare_vessel_aggregated(imageset,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: The summary of the results, with the following entries:
+
+ - ``'inconsistency'``:
+ All findings.
+ - ``details*``:
+ The details of the analysis for the two assumptions.
Examples:
- >>> from mlscorecheck.bundles.retina import check_stare_vessel_aggregated
+ >>> from mlscorecheck.check.bundles.retina import check_stare_vessel_aggregated
>>> scores = {'acc': 0.4964, 'sens': 0.5793, 'spec': 0.4871, 'bacc': 0.5332}
>>> results = check_stare_vessel_aggregated(imageset='all',
annotator='ah',
@@ -218,7 +237,12 @@ def check_stare_vessel_image(image_identifier: str,
Args:
image_identifier (str): the identifier of the image (like "im0235")
annotator (str): the annotation to use ('ah'/'vk')
- scores (dict(str,float)): the scores to be tested
+ scores (dict(str,float)): the scores to be tested ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float): the numerical uncertainty
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -227,12 +251,27 @@ def check_stare_vessel_image(image_identifier: str,
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. Under the ``inconsistency`` key one finds all
- findings, under the keys ``details*`` the details of the analysis can
- be found.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
Examples:
- >>> from mlscorecheck.bundles.retina import check_stare_vessel_image
+ >>> from mlscorecheck.check.bundles.retina import check_stare_vessel_image
>>> img_identifier = 'im0235'
>>> scores = {'acc': 0.4699, 'npv': 0.8993, 'f1p': 0.134}
>>> results = check_stare_vessel_image(image_identifier=img_identifier,
@@ -246,7 +285,7 @@ def check_stare_vessel_image(image_identifier: str,
image = [image for image in images[annotator]['images']
if image['identifier'] == image_identifier][0]
- return check_1_testset_no_kfold_scores(testset=image,
+ return check_1_testset_no_kfold(testset=image,
scores=scores,
eps=eps,
numerical_tolerance=numerical_tolerance,
diff --git a/mlscorecheck/bundles/skinlesion/__init__.py b/mlscorecheck/check/bundles/skinlesion/__init__.py
similarity index 100%
rename from mlscorecheck/bundles/skinlesion/__init__.py
rename to mlscorecheck/check/bundles/skinlesion/__init__.py
diff --git a/mlscorecheck/check/bundles/skinlesion/_isic2016.py b/mlscorecheck/check/bundles/skinlesion/_isic2016.py
new file mode 100644
index 0000000..f3e6bba
--- /dev/null
+++ b/mlscorecheck/check/bundles/skinlesion/_isic2016.py
@@ -0,0 +1,64 @@
+"""
+This module implements the tests for the ISIC2016 dataset
+"""
+
+from ....experiments import get_experiment
+from ...binary import check_1_testset_no_kfold
+from ....core import NUMERICAL_TOLERANCE
+
+__all__ = ['check_isic2016']
+
+def check_isic2016(*,
+ scores: dict,
+ eps: float,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE):
+ """
+ Tests if the scores are consistent with the test set of the ISIC2016
+ melanoma classification dataset
+
+ Args:
+ scores (dict): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the dataset.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
+
+ Examples:
+ >>> from mlscorecheck.check.bundles.skinlesion import check_isic2016
+ >>> scores = {'acc': 0.7916, 'sens': 0.2933, 'spec': 0.9145}
+ >>> results = check_isic2016(scores=scores, eps=1e-4)
+ >>> results['inconsistency']
+ # False
+ """
+ data = get_experiment('skinlesion.isic2016')
+ return check_1_testset_no_kfold(scores=scores,
+ testset=data,
+ eps=eps,
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=True)
diff --git a/mlscorecheck/bundles/skinlesion/_isic2017.py b/mlscorecheck/check/bundles/skinlesion/_isic2017.py
similarity index 63%
rename from mlscorecheck/bundles/skinlesion/_isic2017.py
rename to mlscorecheck/check/bundles/skinlesion/_isic2017.py
index 88aab2b..f302b16 100644
--- a/mlscorecheck/bundles/skinlesion/_isic2017.py
+++ b/mlscorecheck/check/bundles/skinlesion/_isic2017.py
@@ -2,9 +2,9 @@
This module implements the tests for the ISIC2017 dataset
"""
-from ...experiments import get_experiment
-from ...check import check_1_testset_no_kfold_scores
-from ...core import NUMERICAL_TOLERANCE
+from ....experiments import get_experiment
+from ...binary import check_1_testset_no_kfold
+from ....core import NUMERICAL_TOLERANCE
__all__ = ['check_isic2017',
'_prepare_testset_isic2017']
@@ -52,7 +52,12 @@ def check_isic2017(*,
'SK' for seborrheic keratosis and 'N' for nevus.
against (str|list): specification of the negative classes, with the encoding 'M' for
melanoma, 'SK' for seborrheic keratosis and 'N' for nevus.
- scores (dict): the scores to check
+ scores (dict): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
numerical_tolerance (float): in practice, beyond the numerical uncertainty of
the scores, some further tolerance is applied. This is
@@ -60,15 +65,27 @@ def check_isic2017(*,
scores. It does ensure that the specificity of the test
is 1, it might slightly decrease the sensitivity.
Returns:
- dict: a summary of the results. When the ``inconsistency`` flag is True, it indicates
- that the set of feasible ``tp``, ``tn`` pairs is empty. The list under the key
- ``details`` provides further details from the analysis of the scores one after the other.
- Under the key ``n_valid_tptn_pairs`` one finds the number of tp and tn pairs compatible
- with all scores. Under the key ``prefiltering_details`` one finds the results of the
- prefiltering by using the solutions for the score pairs.
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the dataset.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
Examples:
- >>> from mlscorecheck.bundles.skinlesion import check_isic2017
+ >>> from mlscorecheck.check.bundles.skinlesion import check_isic2017
>>> scores = {'acc': 0.6183, 'sens': 0.4957, 'ppv': 0.2544, 'f1p': 0.3362}
>>> results = check_isic2017(target='M',
against=['SK', 'N'],
@@ -80,7 +97,8 @@ def check_isic2017(*,
testset = _prepare_testset_isic2017(target, against)
- return check_1_testset_no_kfold_scores(scores=scores,
+ return check_1_testset_no_kfold(scores=scores,
testset=testset,
eps=eps,
- numerical_tolerance=numerical_tolerance)
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=True)
diff --git a/mlscorecheck/check/multiclass/__init__.py b/mlscorecheck/check/multiclass/__init__.py
new file mode 100644
index 0000000..ccd5b87
--- /dev/null
+++ b/mlscorecheck/check/multiclass/__init__.py
@@ -0,0 +1,10 @@
+"""
+This module brings together all multiclass test functionalities
+"""
+
+from ._check_1_testset_no_kfold_macro import *
+from ._check_1_testset_no_kfold_micro import *
+from ._check_1_dataset_known_folds_som_macro import *
+from ._check_1_dataset_known_folds_som_micro import *
+from ._check_1_dataset_known_folds_mos_micro import *
+from ._check_1_dataset_known_folds_mos_macro import *
diff --git a/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_mos_macro.py b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_mos_macro.py
new file mode 100644
index 0000000..94d8f22
--- /dev/null
+++ b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_mos_macro.py
@@ -0,0 +1,126 @@
+"""
+This module implements the multiclass tests in a k-fold MoS scenario with macro averaging
+of scores of classes.
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import transform_multiclass_fold_to_binary, create_folds_multiclass
+
+from ..binary import check_n_datasets_mos_known_folds_mos
+
+__all__ = ["check_1_dataset_known_folds_mos_macro"]
+
+
+def check_1_dataset_known_folds_mos_macro(
+ dataset: dict,
+ folding: dict,
+ scores: dict,
+ eps,
+ *,
+ class_score_bounds: dict = None,
+ fold_score_bounds: dict = None,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ Checking the consistency of scores calculated by taking the macro average
+ of class-level scores on one single multiclass dataset with k-fold cross-validation.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add ``class_score_bounds`` or ``fold_score_bounds``
+ when, for example, the minimum and the maximum scores over the classes or folds are available.
+
+ Args:
+ testset (dict): the specification of the testset
+ scores (dict(str,float)): the scores to check
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ class_score_bounds (None|dict(str,tuple(float,float))): the potential bounds on the
+ scores for the classes
+ fold_score_bounds (None|dict(str,tuple(float,float))): the potential bounds on the
+ scores in the folds
+ solver_name (None|str, optional): The solver to use. Defaults to None.
+ timeout (None|int, optional): The timeout for the linear programming solver in seconds.
+ Defaults to None.
+ verbosity (int, optional): The verbosity level of the pulp linear programming solver.
+ 0: silent, non-zero: verbose. Defaults to 1.
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_macro
+ >>> dataset = {0: 149, 1: 118, 2: 83, 3: 154}
+ >>> folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
+ >>> scores = {'acc': 0.626, 'sens': 0.2483, 'spec': 0.7509, 'f1p': 0.2469}
+ >>> result = check_1_dataset_known_folds_mos_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> results['inconsistency']
+ # False
+
+ >>> scores['acc'] = 0.656
+ >>> result = check_1_dataset_known_folds_mos_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+ """
+ folds = create_folds_multiclass(dataset, folding)
+ binary_folds = [transform_multiclass_fold_to_binary(fold) for fold in folds]
+
+ evaluations = []
+
+ for binary_folding in binary_folds:
+ folding = {"folds": binary_folding}
+ dataset = {
+ "p": sum(tmp["p"] for tmp in binary_folding),
+ "n": sum(tmp["n"] for tmp in binary_folding),
+ }
+ evaluations.append(
+ {
+ "dataset": dataset,
+ "folding": folding,
+ "fold_score_bounds": class_score_bounds,
+ }
+ )
+
+ return check_n_datasets_mos_known_folds_mos(
+ evaluations=evaluations,
+ scores=scores,
+ eps=eps,
+ dataset_score_bounds=fold_score_bounds,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_mos_micro.py b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_mos_micro.py
new file mode 100644
index 0000000..6534ebe
--- /dev/null
+++ b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_mos_micro.py
@@ -0,0 +1,120 @@
+"""
+This module implements the multiclass tests in a k-fold MoS scenario with micro
+averaging of scores.
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import transform_multiclass_fold_to_binary, create_folds_multiclass
+
+from ..binary import check_n_datasets_mos_kfold_som
+
+__all__ = ["check_1_dataset_known_folds_mos_micro"]
+
+
+def check_1_dataset_known_folds_mos_micro(
+ dataset: dict,
+ folding: dict,
+ scores: dict,
+ eps,
+ *,
+ fold_score_bounds: dict = None,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ This function checks the consistency of scores calculated by taking the micro average
+ on a single multiclass dataset with known folds.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add ``fold_score_bounds`` when, for example, the minimum
+ and the maximum scores over the folds are available.
+
+ Args:
+ dataset (dict): The specification of the dataset.
+ folding (dict): The specification of the folding strategy.
+ scores (dict(str,float)): The scores to check.
+ eps (float|dict(str,float)): The numerical uncertainty(ies) of the scores.
+ fold_score_bounds (None|dict(str,tuple(float,float))): the potential bounds on the
+ scores in the folds
+ solver_name (None|str, optional): The solver to use. Defaults to None.
+ timeout (None|int, optional): The timeout for the linear programming solver in seconds.
+ Defaults to None.
+ verbosity (int, optional): The verbosity level of the pulp linear programming solver.
+ 0: silent, non-zero: verbose. Defaults to 1.
+ numerical_tolerance (float, optional): Beyond the numerical uncertainty of
+ the scores, some further tolerance is applied.
+ This is orders of magnitude smaller than the
+ uncertainty of the scores. It ensures that the
+ specificity of the test is 1, it might slightly
+ decrease the sensitivity. Defaults to
+ NUMERICAL_TOLERANCE.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_micro
+ >>> dataset = {0: 66, 1: 178, 2: 151}
+ >>> folding = {'folds': [{0: 33, 1: 89, 2: 76}, {0: 33, 1: 89, 2: 75}]}
+ >>> scores = {'acc': 0.5646, 'sens': 0.3469, 'spec': 0.6734, 'f1p': 0.3469}
+ >>> result = check_1_dataset_known_folds_mos_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+ >>> scores['acc'] = 0.5746
+ >>> result = check_1_dataset_known_folds_mos_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+ """
+ folds = create_folds_multiclass(dataset, folding)
+ binary_folds = [transform_multiclass_fold_to_binary(fold) for fold in folds]
+
+ evaluations = []
+
+ for binary_folding in binary_folds:
+ folding = {"folds": binary_folding}
+ dataset = {
+ "p": sum(tmp["p"] for tmp in binary_folding),
+ "n": sum(tmp["n"] for tmp in binary_folding),
+ }
+ evaluations.append({"dataset": dataset, "folding": folding})
+
+ return check_n_datasets_mos_kfold_som(
+ evaluations=evaluations,
+ scores=scores,
+ eps=eps,
+ dataset_score_bounds=fold_score_bounds,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_som_macro.py b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_som_macro.py
new file mode 100644
index 0000000..a3aae1d
--- /dev/null
+++ b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_som_macro.py
@@ -0,0 +1,116 @@
+"""
+This module implements the macro-averaged multiclass tests in a k-fold scenario
+with SoM aggregation over the folds.
+"""
+
+import copy
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import create_folds_multiclass
+
+from ._check_1_testset_no_kfold_macro import check_1_testset_no_kfold_macro
+
+__all__ = ["check_1_dataset_known_folds_som_macro"]
+
+
+def check_1_dataset_known_folds_som_macro(
+ dataset: dict,
+ folding: dict,
+ scores: dict,
+ eps,
+ *,
+ class_score_bounds: dict = None,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ This function checks the consistency of scores calculated by taking the macro average
+ on a single multiclass dataset and averaging the scores across the folds in the SoM manner.
+
+ The test operates by constructing a linear program describing the experiment and checkings its
+ feasibility.
+
+ The test can only check the consistency of the 'acc', 'sens', 'spec' and 'bacc'
+ scores. For a stronger test, one can add ``class_score_bounds`` when, for example, the minimum
+ and the maximum scores over the classes are available.
+
+ Args:
+ dataset (dict): The specification of the dataset.
+ folding (dict): The specification of the folding.
+ scores (dict(str,float)): The scores to check.
+ eps (float|dict(str,float)): The numerical uncertainty(ies) of the scores.
+ class_score_bounds (None|dict(str,tuple(float,float))): the potential bounds on the
+ scores for the classes
+ solver_name (None|str, optional): The solver to use. Defaults to None.
+ timeout (None|int, optional): The timeout for the linear programming solver in seconds.
+ Defaults to None.
+ verbosity (int, optional): The verbosity level of the pulp linear programming solver.
+ 0: silent, non-zero: verbose. Defaults to 1.
+ numerical_tolerance (float, optional): In practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied.
+ This is orders of magnitude smaller than the
+ uncertainty of the scores. It does ensure that the
+ specificity of the test is 1, it might slightly
+ decrease the sensitivity. Defaults to
+ NUMERICAL_TOLERANCE.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: If the provided scores are not consistent with the dataset.
+
+ Examples:
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_macro
+ >>> dataset = {0: 129, 1: 81, 2: 135}
+ >>> folding = {'n_folds': 2, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}
+ >>> scores = {'acc': 0.5662, 'sens': 0.3577, 'spec': 0.6767, 'f1p': 0.3481}
+ >>> result = check_1_dataset_known_folds_som_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+ >>> scores['acc'] = 0.6762
+ >>> result = check_1_dataset_known_folds_som_macro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+ """
+ folds = create_folds_multiclass(dataset, folding)
+
+ testset = copy.deepcopy(folds[0])
+ for fold in folds[1:]:
+ for key in fold:
+ testset[key] += fold[key]
+
+ return check_1_testset_no_kfold_macro(
+ testset=testset,
+ scores=scores,
+ eps=eps,
+ class_score_bounds=class_score_bounds,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_som_micro.py b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_som_micro.py
new file mode 100644
index 0000000..f813aa8
--- /dev/null
+++ b/mlscorecheck/check/multiclass/_check_1_dataset_known_folds_som_micro.py
@@ -0,0 +1,101 @@
+"""
+This module implements the micro-averaged multiclass tests in a k-fold scenario
+with SoM aggregation.
+"""
+
+import copy
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import create_folds_multiclass
+
+from ._check_1_testset_no_kfold_micro import check_1_testset_no_kfold_micro
+
+__all__ = ["check_1_dataset_known_folds_som_micro"]
+
+
+def check_1_dataset_known_folds_som_micro(
+ dataset: dict,
+ folding: dict,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ prefilter_by_pairs: bool = True
+) -> dict:
+ """
+ This function checks the consistency of scores calculated by taking the micro average of
+ class level scores on a single multiclass dataset and averaging across the folds in the
+ SoM manner.
+
+ The test is performed by exhaustively testing all possible confusion matrices.
+
+ Args:
+ dataset (dict): The specification of the dataset.
+ folding (dict): The specification of the folding strategy.
+ scores (dict(str,float)): The scores to check.
+ eps (float|dict(str,float)): The numerical uncertainty(ies) of the scores.
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+ prefilter_by_pairs (bool): whether to prefilter the solution space by pair
+ solutions when possible to speed up the process
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
+
+ Raises:
+ ValueError: If the provided scores are not consistent with the dataset.
+
+ Examples:
+ >>> from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_micro
+ >>> dataset = {0: 86, 1: 96, 2: 59, 3: 105}
+ >>> folding = {'folds': [{0: 43, 1: 48, 2: 30, 3: 52}, {0: 43, 1: 48, 2: 29, 3: 53}]}
+ >>> scores = {'acc': 0.6272, 'sens': 0.2543, 'spec': 0.7514, 'f1p': 0.2543}
+ >>> result = check_1_dataset_known_folds_som_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+ >>> scores['sens'] = 0.2553
+ >>> result = check_1_dataset_known_folds_som_micro(dataset=dataset,
+ folding=folding,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+ """
+ folds = create_folds_multiclass(dataset, folding)
+
+ testset = copy.deepcopy(folds[0])
+ for fold in folds[1:]:
+ for key in fold:
+ testset[key] += fold[key]
+
+ return check_1_testset_no_kfold_micro(
+ testset=testset,
+ scores=scores,
+ eps=eps,
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
diff --git a/mlscorecheck/check/multiclass/_check_1_testset_no_kfold_macro.py b/mlscorecheck/check/multiclass/_check_1_testset_no_kfold_macro.py
new file mode 100644
index 0000000..927a7d9
--- /dev/null
+++ b/mlscorecheck/check/multiclass/_check_1_testset_no_kfold_macro.py
@@ -0,0 +1,105 @@
+"""
+This module implements the consistency testing for multiclass macro averages
+supposing one evaluation set
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import transform_multiclass_fold_to_binary
+
+from ..binary import check_1_dataset_known_folds_mos
+
+__all__ = ["check_1_testset_no_kfold_macro"]
+
+
+def check_1_testset_no_kfold_macro(
+ testset: dict,
+ scores: dict,
+ eps,
+ *,
+ class_score_bounds: dict = None,
+ solver_name: str = None,
+ timeout: int = None,
+ verbosity: int = 1,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
+ """
+ The function tests the consistency of scores calculated by taking the macro average of
+ class level scores on one single multiclass dataset.
+
+ The test operates by constructing a linear programming problem representing the experiment
+ and checking its feasibility.
+
+ Note that this test can only check the consistency of the 'acc', 'sens', 'spec'
+ and 'bacc' scores. Note that without bounds, if there is a large number of classes, it is
+ likely that there will be a configuration matching the scores provided. In order to
+ increase the strength of the test, one can add ``class_scores_bounds`` when, for example,
+ besides the average score, the minimum and the maximum scores over the classes
+ are also provided.
+
+ Args:
+ testset (dict): the specification of the testset
+ scores (dict(str,float)): the scores to check
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ class_score_bounds (None|dict(str,tuple(float,float))): bounds on the scores in the
+ classes
+ solver_name (None|str): the solver to use
+ timeout (None|int): the timeout for the linear programming solver in seconds
+ verbosity (int): the verbosity level of the pulp linear programming solver
+ 0: silent, non-zero: verbose.
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'lp_status'``:
+ The status of the lp solver.
+ - ``'lp_configuration_scores_match'``:
+ A flag indicating if the scores from the lp configuration match the scores
+ provided.
+ - ``'lp_configuration_bounds_match'``:
+ Indicates if the specified bounds match the actual figures.
+ - ``'lp_configuration'``:
+ Contains the actual configuration of the linear programming solver.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.multiclass import check_1_testset_no_kfold_macro
+ >>> testset = {0: 10, 1: 100, 2: 80}
+ >>> scores = {'acc': 0.6, 'sens': 0.3417, 'spec': 0.6928, 'f1p': 0.3308}
+ >>> results = check_1_testset_no_kfold_macro(scores=scores, testset=testset, eps=1e-4)
+ >>> results['inconsistency']
+ # False
+
+ >>> scores['acc'] = 0.6020
+ >>> results = check_1_testset_no_kfold_macro(scores=scores, testset=testset, eps=1e-4)
+ >>> results['inconsistency']
+ # True
+ """
+ folds = transform_multiclass_fold_to_binary(testset)
+ dataset = {
+ "p": sum(fold["p"] for fold in folds),
+ "n": sum(fold["n"] for fold in folds),
+ }
+
+ return check_1_dataset_known_folds_mos(
+ scores=scores,
+ eps=eps,
+ dataset=dataset,
+ folding={"folds": folds},
+ fold_score_bounds=class_score_bounds,
+ solver_name=solver_name,
+ timeout=timeout,
+ verbosity=verbosity,
+ numerical_tolerance=numerical_tolerance,
+ )
diff --git a/mlscorecheck/check/multiclass/_check_1_testset_no_kfold_micro.py b/mlscorecheck/check/multiclass/_check_1_testset_no_kfold_micro.py
new file mode 100644
index 0000000..8414f60
--- /dev/null
+++ b/mlscorecheck/check/multiclass/_check_1_testset_no_kfold_micro.py
@@ -0,0 +1,99 @@
+"""
+This module implements the consistency testing for multiclass micro averages
+supposing one evaluation set
+"""
+
+from ...core import NUMERICAL_TOLERANCE
+from ...aggregated import transform_multiclass_fold_to_binary
+
+from ..binary import check_1_dataset_kfold_som
+
+__all__ = ["check_1_testset_no_kfold_micro"]
+
+
+def check_1_testset_no_kfold_micro(
+ testset: dict,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ prefilter_by_pairs: bool = True
+) -> dict:
+ """
+ Checking the consistency of scores calculated by taking the micro average of class level
+ scores on one single multiclass dataset.
+
+ The test operates by the exhaustive enumeration of all potential confusion matrices.
+
+ Args:
+ testset (dict): the specification of the testset
+ scores (dict(str,float)): the scores to check ('acc', 'sens', 'spec',
+ 'bacc', 'npv', 'ppv', 'f1', 'fm', 'f1n',
+ 'fbp', 'fbn', 'upm', 'gm', 'mk', 'lrp', 'lrn', 'mcc',
+ 'bm', 'pt', 'dor', 'ji', 'kappa'), when using
+ f-beta positive or f-beta negative, also set
+ 'beta_positive' and 'beta_negative'.
+ eps (float|dict(str,float)): the numerical uncertainty(ies) of the scores
+ numerical_tolerance (float): in practice, beyond the numerical uncertainty of
+ the scores, some further tolerance is applied. This is
+ orders of magnitude smaller than the uncertainty of the
+ scores. It does ensure that the specificity of the test
+ is 1, it might slightly decrease the sensitivity.
+ prefilter_by_pairs (bool): whether to prefilter the solution space by pair
+ solutions when possible to speed up the process
+
+ Returns:
+ dict: A dictionary containing the results of the consistency check. The dictionary
+ includes the following keys:
+
+ - ``'inconsistency'``:
+ A boolean flag indicating whether the set of feasible true
+ positive (tp) and true negative (tn) pairs is empty. If True,
+ it indicates that the provided scores are not consistent with the experiment.
+ - ``'details'``:
+ A list providing further details from the analysis of the scores one
+ after the other.
+ - ``'n_valid_tptn_pairs'``:
+ The number of tp and tn pairs that are compatible with all
+ scores.
+ - ``'prefiltering_details'``:
+ The results of the prefiltering by using the solutions for
+ the score pairs.
+ - ``'evidence'``:
+ The evidence for satisfying the consistency constraints.
+
+ Raises:
+ ValueError: if the problem is not specified properly
+
+ Examples:
+ >>> from mlscorecheck.check.multiclass import check_1_testset_no_kfold_micro
+ >>> testset = {0: 10, 1: 100, 2: 80}
+ >>> scores = {'acc': 0.5158, 'sens': 0.2737, 'spec': 0.6368,
+ 'bacc': 0.4553, 'ppv': 0.2737, 'npv': 0.6368}
+ >>> results = check_1_testset_no_kfold_micro(testset=testset,
+ scores=scores,
+ eps=1e-4)
+ >>> results['inconsistency']
+ # False
+
+ >>> scores['acc'] = 0.5258
+ >>> results = check_1_testset_no_kfold_micro(testset=testset,
+ scores=scores,
+ eps=1e-4)
+ >>> results['inconsistency']
+ # True
+ """
+ folds = transform_multiclass_fold_to_binary(testset)
+ dataset = {
+ "p": sum(fold["p"] for fold in folds),
+ "n": sum(fold["n"] for fold in folds),
+ }
+
+ return check_1_dataset_kfold_som(
+ scores=scores,
+ eps=eps,
+ dataset=dataset,
+ folding={"folds": folds},
+ numerical_tolerance=numerical_tolerance,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
diff --git a/mlscorecheck/check/regression/__init__.py b/mlscorecheck/check/regression/__init__.py
new file mode 100644
index 0000000..25bbf9f
--- /dev/null
+++ b/mlscorecheck/check/regression/__init__.py
@@ -0,0 +1,5 @@
+"""
+This module implements the tests for regression scores.
+"""
+
+from ._check_regression import *
diff --git a/mlscorecheck/check/regression/_check_regression.py b/mlscorecheck/check/regression/_check_regression.py
new file mode 100644
index 0000000..b67bb42
--- /dev/null
+++ b/mlscorecheck/check/regression/_check_regression.py
@@ -0,0 +1,263 @@
+"""
+This module implements the testing of regression scores
+"""
+
+import numpy as np
+
+from ...individual import Interval
+from ...core import NUMERICAL_TOLERANCE, round_scores, safe_eval
+
+__all__ = [
+ "check_1_testset_no_kfold",
+ "expand_regression_scores",
+ "mean_average_error",
+ "mean_squared_error",
+ "root_mean_squared_error",
+ "r_squared",
+ "calculate_regression_scores",
+ "generate_regression_problem_and_scores",
+ "check_relations",
+ "score_formulas",
+]
+
+rules = [{"score0": "mae", "score1": "rmse", "relation": "le"}]
+
+
+def mean_average_error(y_true: np.array, y_pred: np.array) -> float:
+ """
+ The mean average error (MAE) regression performance score
+
+ Args:
+ y_true (np.array): the true labels
+ y_pred (np.array): the predicted labels
+
+ Returns:
+ float: the MAE score
+ """
+ return np.mean(np.abs(y_true - y_pred))
+
+
+def mean_squared_error(y_true: np.array, y_pred: np.array) -> float:
+ """
+ The mean squared error (MSE) regression performance score
+
+ Args:
+ y_true (np.array): the true labels
+ y_pred (np.array): the predicted labels
+
+ Returns:
+ float: the MSE score
+ """
+ return np.mean((y_true - y_pred) ** 2)
+
+
+def root_mean_squared_error(y_true: np.array, y_pred: np.array) -> float:
+ """
+ The root mean squared error (RMSE) regression performance score
+
+ Args:
+ y_true (np.array): the true labels
+ y_pred (np.array): the predicted labels
+
+ Returns:
+ float: the RMSE score
+ """
+ return np.sqrt(mean_squared_error(y_true, y_pred))
+
+
+def r_squared(y_true: np.array, y_pred: np.array) -> float:
+ """
+ The R squared (r2) regression performance score
+
+ Args:
+ y_true (np.array): the true labels
+ y_pred (np.array): the predicted labels
+
+ Returns:
+ float: the R2 score
+ """
+ return 1.0 - np.sum((y_true - y_pred) ** 2) / (np.var(y_true) * y_true.shape[0])
+
+
+regression_scores = {
+ "mae": mean_average_error,
+ "mse": mean_squared_error,
+ "rmse": root_mean_squared_error,
+ "r2": r_squared,
+}
+
+
+def calculate_regression_scores(
+ y_true: np.array, y_pred: np.array, subset=None
+) -> dict:
+ """
+ Calculate the performance scores for a regression problem
+
+ Args:
+ y_true (np.array): the true labels
+ y_pred (np.array): the predicted labels
+ subset (None|list(str)): the scores to calculate
+
+ Returns:
+ dict: the calculated scores
+ """
+ scores = {}
+
+ for key, function in regression_scores.items():
+ if subset is None or key in subset:
+ scores[key] = function(y_true, y_pred)
+
+ return scores
+
+
+def generate_regression_problem_and_scores(
+ random_state=None, max_n_samples=1000, subset=None, rounding_decimals=None
+) -> (float, int, dict):
+ """
+ Generate a regression problem and corresponding scores
+
+ Args:
+ random_state (None|int|np.random.RandomState): the random state/seed to use
+ max_n_samples (int): the maximum number of samples to be used
+ subset (None|list(str)): the list of scores to be used
+ rounding_decimals (None|int): the number of decimals to round to
+
+ Returns:
+ float, int, dict: the variance, the number of samples and the scores
+ """
+ if not isinstance(random_state, np.random.RandomState):
+ random_state = np.random.RandomState(random_state)
+
+ n_samples = random_state.randint(2, max_n_samples)
+
+ y_true = np.random.random_sample(n_samples)
+ y_pred = y_true + (np.random.random_sample(n_samples) - 0.5) / 10
+
+ scores = calculate_regression_scores(y_true, y_pred, subset)
+
+ scores = round_scores(scores, rounding_decimals)
+
+ return np.var(y_true), n_samples, scores
+
+
+score_formulas = {
+ "mse": {"rmse": "rmse**2", "r2": "((1 - r2) * (var))"},
+ "rmse": {"mse": "mse ** 0.5", "r2": "((1 - r2) * (var)) ** 0.5"},
+ "r2": {"mse": "(1 - mse / var)", "rmse": "(1 - rmse**2 / var)"},
+}
+
+
+def expand_regression_scores(
+ var: float, n_samples: int, scores: dict, eps, numerical_tolerance: float
+) -> dict:
+ """
+ Generate scores from the ones available and expand the scores to intervals given the
+ numerical uncertainty.
+
+ Args:
+ var (float): the variance of the evaluation set
+ n_samples (int): the number of samples in the evaluation set
+ scores (dict(str,float)): the scores to check ('mae', 'rmse', 'mse', 'r2')
+ eps (float,dict(str,float)): the numerical uncertainty of the scores
+ numerical_tolerance (float): the numerical tolerance of the test
+
+ Returns:
+ dict: the extended set of score intervals
+ """
+ scores = {
+ key: Interval(
+ value - eps - numerical_tolerance, value + eps + numerical_tolerance
+ )
+ for key, value in scores.items()
+ }
+
+ to_add = {}
+ for key, value in score_formulas.items():
+ if key not in scores:
+ for sol, formula in value.items():
+ if sol in scores:
+ to_add[key] = safe_eval(
+ formula, scores | {"var": var, "n_samples": n_samples}
+ )
+ break
+
+ return scores | to_add
+
+
+def check_relations(scores: dict) -> dict:
+ """
+ Check the relations of the scores
+
+ Args:
+ scores (dict(str,Interval)): the score and figure intervals
+
+ Returns:
+ dict: a summary of the analysis, with the following entries:
+
+ - ``'inconsistency'`` (bool): whether an inconsistency has been identified
+ - ``'details'`` (list(dict)): the details of the analysis, with the following entries
+ """
+ results = {"details": []}
+
+ for rule in rules:
+ if rule["score0"] in scores and rule["score1"] in scores:
+ score0 = scores[rule["score0"]]
+ score1 = scores[rule["score1"]]
+ if rule["relation"] == "le":
+ value = score0.lower_bound <= score1.upper_bound
+ results["details"].append(rule | {"value": value})
+
+ results["inconsistency"] = any(not result["value"] for result in results["details"])
+
+ return results
+
+
+def check_1_testset_no_kfold(
+ var: float,
+ n_samples: int,
+ scores: dict,
+ eps,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+) -> dict:
+ """
+ The consistency test for regression scores calculated on a single test set
+ with no k-folding
+
+ Args:
+ var (float): the variance of the evaluation set
+ n_samples (int): the number of samples in the evaluation set
+ scores (dict(str,float)): the scores to check ('mae', 'rmse', 'mse', 'r2')
+ eps (float,dict(str,float)): the numerical uncertainty of the scores
+ numerical_tolerance (float): the numerical tolerance of the test
+
+ Returns:
+ dict: a summary of the analysis, with the following entries:
+
+ - ``'inconsistency'`` (bool): whether an inconsistency has been identified
+ - ``'details'`` (list(dict)): the details of the analysis, with the following entries
+
+ Examples:
+ >>> from mlscorecheck.check.regression import check_1_testset_no_kfold
+ >>> var = 0.08316192579267838
+ >>> n_samples = 100
+ >>> scores = {'mae': 0.0254, 'r2': 0.9897}
+ >>> result = check_1_testset_no_kfold(var=var,
+ n_samples=n_samples,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # False
+
+ >>> scores['mae'] = 0.03
+ >>> result = check_1_testset_no_kfold(var=var,
+ n_samples=n_samples,
+ scores=scores,
+ eps=1e-4)
+ >>> result['inconsistency']
+ # True
+ """
+ intervals = expand_regression_scores(
+ var, n_samples, scores, eps, numerical_tolerance
+ )
+
+ return check_relations(intervals)
diff --git a/mlscorecheck/core/_dict.py b/mlscorecheck/core/_dict.py
index b6ef6ae..1edad4e 100644
--- a/mlscorecheck/core/_dict.py
+++ b/mlscorecheck/core/_dict.py
@@ -4,8 +4,8 @@
import numpy as np
-__all__ = ['dict_mean',
- 'dict_minmax']
+__all__ = ["dict_mean", "dict_minmax"]
+
def dict_mean(dicts: list) -> dict:
"""
@@ -28,6 +28,7 @@ def dict_mean(dicts: list) -> dict:
return result
+
def dict_minmax(dicts: list) -> dict:
"""
Calculates the min-max of scores in a dictionary
diff --git a/mlscorecheck/core/_load_json.py b/mlscorecheck/core/_load_json.py
index 106e24c..be74e3d 100644
--- a/mlscorecheck/core/_load_json.py
+++ b/mlscorecheck/core/_load_json.py
@@ -7,7 +7,8 @@
from importlib.resources import files
-__all__ = ['load_json']
+__all__ = ["load_json"]
+
def load_json(directory: str, file: str) -> dict:
"""
@@ -20,6 +21,6 @@ def load_json(directory: str, file: str) -> dict:
Returns:
obj: the loaded object
"""
- sio = files('mlscorecheck').joinpath(os.path.join(directory, file)).read_text()
+ sio = files("mlscorecheck").joinpath(os.path.join(directory, file)).read_text()
return json.loads(sio)
diff --git a/mlscorecheck/core/_numerical_tolerance.py b/mlscorecheck/core/_numerical_tolerance.py
index 592921e..b31cab0 100644
--- a/mlscorecheck/core/_numerical_tolerance.py
+++ b/mlscorecheck/core/_numerical_tolerance.py
@@ -4,12 +4,15 @@
import numpy as np
-__all__ = ['NUMERICAL_TOLERANCE',
- 'check_uncertainty_and_tolerance',
- 'update_uncertainty']
+__all__ = [
+ "NUMERICAL_TOLERANCE",
+ "check_uncertainty_and_tolerance",
+ "update_uncertainty",
+]
NUMERICAL_TOLERANCE = 1e-6
+
def check_uncertainty_and_tolerance(eps: float, numerical_tolerance: float):
"""
Checking the relation of the numerical tolerance and the uncertainty
@@ -23,9 +26,12 @@ def check_uncertainty_and_tolerance(eps: float, numerical_tolerance: float):
greater than the numerical tolerance
"""
eps_min = np.min(list(eps.values())) if isinstance(eps, dict) else eps
- if eps_min <= 10*numerical_tolerance:
- raise ValueError('The numerical tolerance is comparable to the numerical '\
- 'uncertainty. Please lower the numerical tolerance.')
+ if eps_min <= 10 * numerical_tolerance:
+ raise ValueError(
+ "The numerical tolerance is comparable to the numerical "
+ "uncertainty. Please lower the numerical tolerance."
+ )
+
def update_uncertainty(eps: float, numerical_tolerance: float):
"""
diff --git a/mlscorecheck/core/_random_state.py b/mlscorecheck/core/_random_state.py
index 7673130..37a0cc6 100644
--- a/mlscorecheck/core/_random_state.py
+++ b/mlscorecheck/core/_random_state.py
@@ -4,9 +4,10 @@
import numpy as np
-__all__ = ['init_random_state']
+__all__ = ["init_random_state"]
-def init_random_state(random_state = None) -> np.random.RandomState:
+
+def init_random_state(random_state=None) -> np.random.RandomState:
"""
Initializes a random state
diff --git a/mlscorecheck/core/_rounding.py b/mlscorecheck/core/_rounding.py
index 222fd7f..8bb646e 100644
--- a/mlscorecheck/core/_rounding.py
+++ b/mlscorecheck/core/_rounding.py
@@ -4,7 +4,8 @@
import numpy as np
-__all__ = ['round_scores']
+__all__ = ["round_scores"]
+
def round_scores(to_round: dict, rounding_decimals: int = None):
"""
@@ -23,4 +24,7 @@ def round_scores(to_round: dict, rounding_decimals: int = None):
if not isinstance(to_round, dict):
return float(np.round(to_round, rounding_decimals))
- return {key: float(np.round(value, rounding_decimals)) for key, value in to_round.items()}
+ return {
+ key: float(np.round(value, rounding_decimals))
+ for key, value in to_round.items()
+ }
diff --git a/mlscorecheck/core/_safe_eval.py b/mlscorecheck/core/_safe_eval.py
index 787bdc4..d7b47c4 100644
--- a/mlscorecheck/core/_safe_eval.py
+++ b/mlscorecheck/core/_safe_eval.py
@@ -2,8 +2,8 @@
This module implements functions for safe evaluations
"""
-__all__ = ['safe_eval',
- 'safe_call']
+__all__ = ["safe_eval", "safe_call"]
+
def safe_eval(expression: str, subs: dict):
"""
@@ -18,7 +18,8 @@ def safe_eval(expression: str, subs: dict):
obj: the value of the expression
"""
- return eval(expression, subs) # pylint: disable=eval-used
+ return eval(expression, subs) # pylint: disable=eval-used
+
def check_applicability(params: dict, non_applicable: list = None) -> bool:
"""
@@ -48,6 +49,7 @@ def check_applicability(params: dict, non_applicable: list = None) -> bool:
return False
return True
+
def safe_call(function, params: dict, non_applicable: list = None):
"""
Safe call to a function
@@ -65,6 +67,6 @@ def safe_call(function, params: dict, non_applicable: list = None):
if not check_applicability(params, non_applicable):
return None
- args = list(function.__code__.co_varnames[:function.__code__.co_kwonlyargcount])
+ args = list(function.__code__.co_varnames[: function.__code__.co_kwonlyargcount])
return function(**{arg: params[arg] for arg in args if arg in params})
diff --git a/mlscorecheck/experiments/_load_datasets.py b/mlscorecheck/experiments/_load_datasets.py
index f4814d6..066379c 100644
--- a/mlscorecheck/experiments/_load_datasets.py
+++ b/mlscorecheck/experiments/_load_datasets.py
@@ -4,15 +4,13 @@
import os
-__all__ = ['dataset_statistics',
- 'load_ml_datasets',
- 'lookup_dataset',
- 'resolve_pn']
+__all__ = ["dataset_statistics", "load_ml_datasets", "lookup_dataset", "resolve_pn"]
from ..core import load_json
dataset_statistics = {}
+
def resolve_pn(dataset_conf):
"""
Resolve the dataset configuration from the integrated statistics
@@ -27,15 +25,16 @@ def resolve_pn(dataset_conf):
"""
if isinstance(dataset_conf, dict):
result = {**dataset_conf}
- if result.get('dataset') is not None:
- tmp = lookup_dataset(result['dataset'])
- result['p'] = tmp['p']
- result['n'] = tmp['n']
+ if result.get("dataset") is not None:
+ tmp = lookup_dataset(result["dataset"])
+ result["p"] = tmp["p"]
+ result["n"] = tmp["n"]
elif isinstance(dataset_conf, list):
result = [resolve_pn(dataset) for dataset in dataset_conf]
return result
+
def lookup_dataset(dataset: str) -> dict:
"""
Look up a dataset
@@ -47,22 +46,34 @@ def lookup_dataset(dataset: str) -> dict:
dict: the count statistics of the dataset
"""
if dataset not in dataset_statistics:
- raise ValueError(f"No statistics about dataset {dataset} are available. "\
- "Didn't you forget to identify like 'common_datasets.ecoli1'?")
+ raise ValueError(
+ f"No statistics about dataset {dataset} are available. "
+ "Didn't you forget to identify like 'common_datasets.ecoli1'?"
+ )
return dataset_statistics[dataset]
+
def load_ml_datasets():
"""
Load the ML datasets
"""
- data = load_json(os.path.join('experiments', 'machine_learning'), 'sklearn.json')
+ data = load_json(os.path.join("experiments", "machine_learning"), "sklearn.json")
+
+ for entry in data["datasets"]:
+ dataset_statistics["sklearn." + entry["name"]] = {
+ "p": entry["p"],
+ "n": entry["n"],
+ }
- for entry in data['datasets']:
- dataset_statistics['sklearn.' + entry['name']] = {'p': entry['p'], 'n': entry['n']}
+ data = load_json(
+ os.path.join("experiments", "machine_learning"), "common_datasets.json"
+ )
- data = load_json(os.path.join('experiments', 'machine_learning'), 'common_datasets.json')
+ for entry in data["datasets"]:
+ dataset_statistics["common_datasets." + entry["name"]] = {
+ "p": entry["p"],
+ "n": entry["n"],
+ }
- for entry in data['datasets']:
- dataset_statistics['common_datasets.' + entry['name']] = {'p': entry['p'], 'n': entry['n']}
load_ml_datasets()
diff --git a/mlscorecheck/experiments/_load_experiments.py b/mlscorecheck/experiments/_load_experiments.py
index 32be3a7..921079c 100644
--- a/mlscorecheck/experiments/_load_experiments.py
+++ b/mlscorecheck/experiments/_load_experiments.py
@@ -4,17 +4,19 @@
import os
-__all__ = ['get_experiment',
- 'load_drive',
- 'load_tpehg',
- 'load_isic2016',
- 'load_isic2017',
- 'load_stare',
- 'load_chase_db1',
- 'load_diaretdb0',
- 'load_diaretdb1',
- 'load_hrf',
- 'load_drishti_gs']
+__all__ = [
+ "get_experiment",
+ "load_drive",
+ "load_tpehg",
+ "load_isic2016",
+ "load_isic2017",
+ "load_stare",
+ "load_chase_db1",
+ "load_diaretdb0",
+ "load_diaretdb1",
+ "load_hrf",
+ "load_drishti_gs",
+]
from ..core import load_json
@@ -35,28 +37,28 @@ def get_experiment(name):
if name in experiments:
return experiments[name]
- if name == 'retina.drive':
+ if name == "retina.drive":
experiments[name] = load_drive()
- elif name == 'retina.stare':
+ elif name == "retina.stare":
experiments[name] = load_stare()
- elif name == 'retina.chase_db1':
+ elif name == "retina.chase_db1":
experiments[name] = load_chase_db1()
- elif name == 'retina.diaretdb0':
+ elif name == "retina.diaretdb0":
experiments[name] = load_diaretdb0()
- elif name == 'retina.diaretdb1':
+ elif name == "retina.diaretdb1":
experiments[name] = load_diaretdb1()
- elif name == 'retina.hrf':
+ elif name == "retina.hrf":
experiments[name] = load_hrf()
- elif name == 'retina.drishti_gs':
+ elif name == "retina.drishti_gs":
experiments[name] = load_drishti_gs()
- elif name == 'ehg.tpehg':
+ elif name == "ehg.tpehg":
experiments[name] = load_tpehg()
- elif name == 'skinlesion.isic2016':
+ elif name == "skinlesion.isic2016":
experiments[name] = load_isic2016()
- elif name == 'skinlesion.isic2017':
+ elif name == "skinlesion.isic2017":
experiments[name] = load_isic2017()
else:
- raise ValueError(f'unknown dataset {name}')
+ raise ValueError(f"unknown dataset {name}")
return experiments[name]
@@ -68,9 +70,11 @@ def load_chase_db1() -> dict:
Returns:
dict: the experiment specifications
"""
- prefix = os.path.join('experiments', 'retina', 'chase_db1')
- return {'manual1': load_json(prefix, 'manual1.json'),
- 'manual2': load_json(prefix, 'manual2.json')}
+ prefix = os.path.join("experiments", "retina", "chase_db1")
+ return {
+ "manual1": load_json(prefix, "manual1.json"),
+ "manual2": load_json(prefix, "manual2.json"),
+ }
def load_diaretdb0() -> dict:
@@ -80,8 +84,8 @@ def load_diaretdb0() -> dict:
Returns:
dict: the experiment specifications
"""
- prefix = os.path.join('experiments', 'retina', 'diaretdb0')
- return load_json(prefix, 'diaretdb0.json')
+ prefix = os.path.join("experiments", "retina", "diaretdb0")
+ return load_json(prefix, "diaretdb0.json")
def load_diaretdb1() -> dict:
@@ -91,8 +95,8 @@ def load_diaretdb1() -> dict:
Returns:
dict: the experiment specifications
"""
- prefix = os.path.join('experiments', 'retina', 'diaretdb1')
- return load_json(prefix, 'diaretdb1.json')
+ prefix = os.path.join("experiments", "retina", "diaretdb1")
+ return load_json(prefix, "diaretdb1.json")
def load_drishti_gs() -> dict:
@@ -102,9 +106,11 @@ def load_drishti_gs() -> dict:
Returns:
dict: the experiment specifications
"""
- prefix = os.path.join('experiments', 'retina', 'drishti_gs')
- return {'train': load_json(prefix, 'drishti_gs_train.json')['distributions'],
- 'test': load_json(prefix, 'drishti_gs_test.json')['distributions']}
+ prefix = os.path.join("experiments", "retina", "drishti_gs")
+ return {
+ "train": load_json(prefix, "drishti_gs_train.json")["distributions"],
+ "test": load_json(prefix, "drishti_gs_test.json")["distributions"],
+ }
def load_hrf() -> dict:
@@ -114,9 +120,11 @@ def load_hrf() -> dict:
Returns:
dict: the experiment specifications
"""
- prefix = os.path.join('experiments', 'retina', 'hrf')
- return {'fov': load_json(prefix, 'with_fov.json'),
- 'all': load_json(prefix, 'without_fov.json')}
+ prefix = os.path.join("experiments", "retina", "hrf")
+ return {
+ "fov": load_json(prefix, "with_fov.json"),
+ "all": load_json(prefix, "without_fov.json"),
+ }
def load_stare() -> dict:
@@ -126,9 +134,8 @@ def load_stare() -> dict:
Returns:
dict: the experiment specifications
"""
- prefix = os.path.join('experiments', 'retina', 'stare')
- return {'ah': load_json(prefix, 'ah.json'),
- 'vk': load_json(prefix, 'vk.json')}
+ prefix = os.path.join("experiments", "retina", "stare")
+ return {"ah": load_json(prefix, "ah.json"), "vk": load_json(prefix, "vk.json")}
def load_drive() -> dict:
@@ -138,13 +145,17 @@ def load_drive() -> dict:
Returns:
dict: the drive experiments
"""
- prefix = os.path.join('experiments', 'retina', 'drive')
+ prefix = os.path.join("experiments", "retina", "drive")
results = {}
for annotator in [1, 2]:
- for assumption in ['fov', 'all']:
- tmp = {'train': load_json(prefix, f'drive_{annotator}_train_{assumption}.json'),
- 'test': load_json(prefix, f'drive_{annotator}_test_{assumption}.json')}
+ for assumption in ["fov", "all"]:
+ tmp = {
+ "train": load_json(
+ prefix, f"drive_{annotator}_train_{assumption}.json"
+ ),
+ "test": load_json(prefix, f"drive_{annotator}_test_{assumption}.json"),
+ }
results[(annotator, assumption)] = tmp
return results
@@ -156,8 +167,8 @@ def load_tpehg() -> dict:
Returns:
dict: the drive experiments
"""
- prefix = os.path.join('experiments', 'ehg')
- return load_json(prefix, 'tpehg.json')['distribution']
+ prefix = os.path.join("experiments", "ehg")
+ return load_json(prefix, "tpehg.json")["distribution"]
def load_isic2016() -> dict:
@@ -167,8 +178,8 @@ def load_isic2016() -> dict:
Returns:
dict: the testset
"""
- prefix = os.path.join('experiments', 'skinlesion', 'isic2016')
- return load_json(prefix, 'isic2016.json')['distribution']
+ prefix = os.path.join("experiments", "skinlesion", "isic2016")
+ return load_json(prefix, "isic2016.json")["distribution"]
def load_isic2017() -> dict:
@@ -179,5 +190,5 @@ def load_isic2017() -> dict:
Returns:
dict: the testset
"""
- prefix = os.path.join('experiments', 'skinlesion', 'isic2017')
- return load_json(prefix, 'isic2017.json')['distribution']
+ prefix = os.path.join("experiments", "skinlesion", "isic2017")
+ return load_json(prefix, "isic2017.json")["distribution"]
diff --git a/mlscorecheck/individual/__init__.py b/mlscorecheck/individual/__init__.py
index bd0706f..77f3e1b 100644
--- a/mlscorecheck/individual/__init__.py
+++ b/mlscorecheck/individual/__init__.py
@@ -2,7 +2,6 @@
This module brings together all the logic related to individual scores
"""
-from ._calculate_scores import *
from ._expression import *
from ._interval import *
from ._pair_solutions import *
diff --git a/mlscorecheck/individual/_check_scores_tptn.py b/mlscorecheck/individual/_check_scores_tptn.py
index c2c9ec5..bc2ca77 100644
--- a/mlscorecheck/individual/_check_scores_tptn.py
+++ b/mlscorecheck/individual/_check_scores_tptn.py
@@ -12,21 +12,47 @@
from ._tptn_solution_bundles import tptn_solutions, sens_tp, spec_tn, is_applicable_tptn
from ._pair_solutions import solution_specifications
-__all__ = ['check_scores_tptn_pairs',
- '_check_scores_tptn_pairs',
- '_check_scores_tptn_intervals']
-
-preferred_order = ['acc', 'sens', 'spec', 'bacc', 'npv', 'ppv', 'f1p', 'f1n', 'fbp', 'fbn',
- 'fm', 'bm', 'pt', 'lrp', 'lrn', 'mk', 'dor',
- 'ji', 'gm', 'upm', 'kappa', 'mcc']
-
-def iterate_tptn(*,
- score: str,
- score_value: float,
- valid_pairs: dict,
- sol_fun,
- params: dict,
- iterate_by: str) -> dict:
+__all__ = [
+ "check_scores_tptn_pairs",
+ "_check_scores_tptn_pairs",
+ "_check_scores_tptn_intervals",
+]
+
+preferred_order = [
+ "acc",
+ "sens",
+ "spec",
+ "bacc",
+ "npv",
+ "ppv",
+ "f1p",
+ "f1n",
+ "fbp",
+ "fbn",
+ "fm",
+ "bm",
+ "pt",
+ "lrp",
+ "lrn",
+ "mk",
+ "dor",
+ "ji",
+ "gm",
+ "upm",
+ "kappa",
+ "mcc",
+]
+
+
+def iterate_tptn(
+ *,
+ score: str,
+ score_value: float,
+ valid_pairs: dict,
+ sol_fun,
+ params: dict,
+ iterate_by: str
+) -> dict:
"""
Iterate through the potential values of tp or tn and construct feasible pairs
@@ -45,7 +71,9 @@ def iterate_tptn(*,
results = {}
for value in valid_pairs:
- if not is_applicable_tptn(score, score_value, 'tp' if iterate_by == 'tn' else 'tn'):
+ if not is_applicable_tptn(
+ score, score_value, "tp" if iterate_by == "tn" else "tn"
+ ):
results[value] = valid_pairs[value]
continue
@@ -62,10 +90,8 @@ def iterate_tptn(*,
return results
-def update_sens(p: int,
- valid_pairs: dict,
- score_int,
- solve_for: str) -> dict:
+
+def update_sens(p: int, valid_pairs: dict, score_int, solve_for: str) -> dict:
"""
Update sensitivity intervals
@@ -79,21 +105,26 @@ def update_sens(p: int,
dict(int,Interval|IntervalUnion): the updated intervals
"""
ints = sens_tp(sens=score_int, p=p)
- ints = ints.intersection(Interval(0, p+1)).shrink_to_integers()
-
- if solve_for == 'tp':
- valid_pairs = {key: value.intersection(ints) for key, value in valid_pairs.items()}
- valid_pairs = {key: value for key, value in valid_pairs.items() if not value.is_empty()}
+ ints = ints.intersection(Interval(0, p + 1)).shrink_to_integers()
+
+ if solve_for == "tp":
+ valid_pairs = {
+ key: value.intersection(ints) for key, value in valid_pairs.items()
+ }
+ valid_pairs = {
+ key: value for key, value in valid_pairs.items() if not value.is_empty()
+ }
else:
- valid_pairs = {value: interval for value, interval in valid_pairs.items()
- if ints.contains(value)}
+ valid_pairs = {
+ value: interval
+ for value, interval in valid_pairs.items()
+ if ints.contains(value)
+ }
return valid_pairs
-def update_spec(n: int,
- valid_pairs: dict,
- score_int,
- solve_for: str) -> dict:
+
+def update_spec(n: int, valid_pairs: dict, score_int, solve_for: str) -> dict:
"""
Update specificity intervals
@@ -107,21 +138,28 @@ def update_spec(n: int,
dict(int,Interval|IntervalUnion): the updated intervals
"""
ints = spec_tn(spec=score_int, n=n)
- ints = ints.intersection(Interval(0, n+1)).shrink_to_integers()
-
- if solve_for == 'tn':
- valid_pairs = {key: value.intersection(ints) for key, value in valid_pairs.items()}
- valid_pairs = {key: value for key, value in valid_pairs.items() if not value.is_empty()}
+ ints = ints.intersection(Interval(0, n + 1)).shrink_to_integers()
+
+ if solve_for == "tn":
+ valid_pairs = {
+ key: value.intersection(ints) for key, value in valid_pairs.items()
+ }
+ valid_pairs = {
+ key: value for key, value in valid_pairs.items() if not value.is_empty()
+ }
else:
- valid_pairs = {value: interval for value, interval in valid_pairs.items()
- if ints.contains(value)}
+ valid_pairs = {
+ value: interval
+ for value, interval in valid_pairs.items()
+ if ints.contains(value)
+ }
return valid_pairs
-def initialize_valid_pairs(p: int,
- n: int,
- iterate_by: str,
- init_tptn_intervals: dict) -> dict:
+
+def initialize_valid_pairs(
+ p: int, n: int, iterate_by: str, init_tptn_intervals: dict
+) -> dict:
"""
Initializes the valid pairs, either from the original input or the
prefiltered intervals.
@@ -136,10 +174,10 @@ def initialize_valid_pairs(p: int,
dict(int,Interval|IntervalUnion): the initialized pairs
"""
if init_tptn_intervals is not None:
- tp_int = IntervalUnion(init_tptn_intervals['tp'])
- tn_int = IntervalUnion(init_tptn_intervals['tn'])
+ tp_int = IntervalUnion(init_tptn_intervals["tp"])
+ tn_int = IntervalUnion(init_tptn_intervals["tn"])
valid_pairs = {}
- if iterate_by == 'tp':
+ if iterate_by == "tp":
for interval in tp_int.intervals:
for tp in range(interval.lower_bound, interval.upper_bound + 1):
valid_pairs[tp] = copy.deepcopy(tn_int)
@@ -149,17 +187,20 @@ def initialize_valid_pairs(p: int,
valid_pairs[tn] = copy.deepcopy(tp_int)
return valid_pairs
- init_interval = Interval(0, n+1) if iterate_by == 'tp' else Interval(0, p+1)
- return {key: init_interval for key in range(p+1 if iterate_by == 'tp' else n+1)}
-
-def _check_scores_tptn_pairs(p: int,
- n: int,
- scores: dict,
- eps,
- *,
- numerical_tolerance: float = NUMERICAL_TOLERANCE,
- solve_for: str=None,
- init_tptn_intervals: dict=None) -> dict:
+ init_interval = Interval(0, n + 1) if iterate_by == "tp" else Interval(0, p + 1)
+ return {key: init_interval for key in range(p + 1 if iterate_by == "tp" else n + 1)}
+
+
+def _check_scores_tptn_pairs(
+ p: int,
+ n: int,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ solve_for: str = None,
+ init_tptn_intervals: dict = None
+) -> dict:
"""
Check scores by iteratively reducing the set of feasible ``tp``, ``tn`` pairs.
@@ -182,13 +223,15 @@ def _check_scores_tptn_pairs(p: int,
all scores.
"""
if solve_for is None:
- solve_for = 'tn' if p < n else 'tp'
+ solve_for = "tn" if p < n else "tp"
- if solve_for not in {'tp', 'tn'}:
- raise ValueError('The specified ``solve_for`` variable needs to be either '\
- '``tp`` or ``tn``.')
+ if solve_for not in {"tp", "tn"}:
+ raise ValueError(
+ "The specified ``solve_for`` variable needs to be either "
+ "``tp`` or ``tn``."
+ )
- iterate_by = 'tn' if solve_for == 'tp' else 'tp'
+ iterate_by = "tn" if solve_for == "tp" else "tp"
# resolving aliases and complements
scores = resolve_aliases_and_complements(scores)
@@ -197,66 +240,82 @@ def _check_scores_tptn_pairs(p: int,
eps = eps if isinstance(eps, dict) else {score: eps for score in scores}
eps = update_uncertainty(eps, numerical_tolerance)
- params = {'p': p, 'n': n,
- 'beta_positive': scores.get('beta_positive'),
- 'beta_negative': scores.get('beta_negative')}
+ params = {
+ "p": p,
+ "n": n,
+ "beta_positive": scores.get("beta_positive"),
+ "beta_negative": scores.get("beta_negative"),
+ }
valid_pairs = initialize_valid_pairs(p, n, iterate_by, init_tptn_intervals)
details = []
for score in [score for score in preferred_order if score in scores]:
- logger.info('testing %s, feasible tptn pairs: %d',
- score, p*n if valid_pairs is None else len(valid_pairs))
+ logger.info(
+ "testing %s, feasible tptn pairs: %d",
+ score,
+ p * n if valid_pairs is None else len(valid_pairs),
+ )
score_int = Interval(scores[score] - eps[score], scores[score] + eps[score])
params[score] = score_int
- detail = {'testing': score,
- 'score_interval': score_int,
- 'n_tptn_pairs_before': p*n if valid_pairs is None else len(valid_pairs)}
-
- if score not in {'sens', 'spec'}:
- valid_pairs = iterate_tptn(score=score,
- score_value=scores[score],
- valid_pairs=valid_pairs,
- sol_fun=tptn_solutions[score][solve_for],
- params=params,
- iterate_by=iterate_by)
- elif score == 'sens':
- valid_pairs = update_sens(p=p,
- valid_pairs=valid_pairs,
- score_int=score_int,
- solve_for=solve_for)
+ detail = {
+ "testing": score,
+ "score_interval": score_int,
+ "n_tptn_pairs_before": p * n if valid_pairs is None else len(valid_pairs),
+ }
+
+ if score not in {"sens", "spec"}:
+ valid_pairs = iterate_tptn(
+ score=score,
+ score_value=scores[score],
+ valid_pairs=valid_pairs,
+ sol_fun=tptn_solutions[score][solve_for],
+ params=params,
+ iterate_by=iterate_by,
+ )
+ elif score == "sens":
+ valid_pairs = update_sens(
+ p=p, valid_pairs=valid_pairs, score_int=score_int, solve_for=solve_for
+ )
else:
# score == 'spec'
- valid_pairs = update_spec(n=n,
- valid_pairs=valid_pairs,
- score_int=score_int,
- solve_for=solve_for)
+ valid_pairs = update_spec(
+ n=n, valid_pairs=valid_pairs, score_int=score_int, solve_for=solve_for
+ )
- detail['n_tptn_pairs_after'] = len(valid_pairs)
- detail['decision'] = 'continue' if len(valid_pairs) > 0 else 'infeasible'
+ detail["n_tptn_pairs_after"] = len(valid_pairs)
+ detail["decision"] = "continue" if len(valid_pairs) > 0 else "infeasible"
details.append(detail)
if len(valid_pairs) == 0:
- logger.info('no more feasible tp,tn pairs left')
+ logger.info("no more feasible tp,tn pairs left")
break
total_count = sum(interval.integer_counts() for interval in valid_pairs.values())
- logger.info('constructing final tp, tn pair set')
- logger.info('final number of intervals: %d', len(valid_pairs))
- logger.info('final number of pairs: %d', total_count)
-
- return {'inconsistency': len(valid_pairs) == 0,
- 'details': details,
- 'n_valid_tptn_pairs': total_count,
- 'iterate_by': iterate_by,
- 'solve_for': solve_for,
- 'evidence': ((list(valid_pairs.keys())[0],
- valid_pairs[list(valid_pairs.keys())[0]].to_tuple())
- if len(valid_pairs) > 0 else None)}
+ logger.info("constructing final tp, tn pair set")
+ logger.info("final number of intervals: %d", len(valid_pairs))
+ logger.info("final number of pairs: %d", total_count)
+
+ return {
+ "inconsistency": len(valid_pairs) == 0,
+ "details": details,
+ "n_valid_tptn_pairs": total_count,
+ "iterate_by": iterate_by,
+ "solve_for": solve_for,
+ "evidence": (
+ {
+ iterate_by: list(valid_pairs.keys())[0],
+ solve_for: valid_pairs[list(valid_pairs.keys())[0]].representing_int(),
+ }
+ if len(valid_pairs) > 0
+ else None
+ ),
+ }
+
def check_all_negative_base(sols: list) -> bool:
"""
@@ -268,7 +327,8 @@ def check_all_negative_base(sols: list) -> bool:
Returns:
bool: True if all solutions have negative base, False otherwise
"""
- return all(sol.get('message') == 'negative base' for sol in sols)
+ return all(sol.get("message") == "negative base" for sol in sols)
+
def check_any_zero_division(sols: list) -> bool:
"""
@@ -280,7 +340,8 @@ def check_any_zero_division(sols: list) -> bool:
Returns:
bool: True if at least one solution has zero division, False otherwise
"""
- return any(sol.get('message') == 'zero division' for sol in sols)
+ return any(sol.get("message") == "zero division" for sol in sols)
+
def update_tptn(tp, tn, sols: list):
"""
@@ -294,23 +355,26 @@ def update_tptn(tp, tn, sols: list):
Returns:
IntervalUnion, IntervalUnion: the updated tp and tn intervals
"""
- tp_union = IntervalUnion([sol['tp'] for sol in sols if sol['tp'] is not None])
- tn_union = IntervalUnion([sol['tn'] for sol in sols if sol['tn'] is not None])
+ tp_union = IntervalUnion([sol["tp"] for sol in sols if sol["tp"] is not None])
+ tn_union = IntervalUnion([sol["tn"] for sol in sols if sol["tn"] is not None])
- logger.info('the tp solutions: %s', tp_union)
- logger.info('the tn solutions: %s', tn_union)
+ logger.info("the tp solutions: %s", tp_union)
+ logger.info("the tn solutions: %s", tn_union)
tp = tp.intersection(tp_union).shrink_to_integers()
tn = tn.intersection(tn_union).shrink_to_integers()
return tp, tn
-def _check_scores_tptn_intervals(p: int,
- n: int,
- scores: dict,
- eps,
- *,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
+
+def _check_scores_tptn_intervals(
+ p: int,
+ n: int,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
"""
Check scores by iteratively reducing the set of feasible ``tp``, ``tn`` pairs.
@@ -337,9 +401,12 @@ def _check_scores_tptn_intervals(p: int,
eps = eps if isinstance(eps, dict) else {score: eps for score in scores}
eps = update_uncertainty(eps, numerical_tolerance)
- params = {'p': p, 'n': n,
- 'beta_positive': scores.get('beta_positive'),
- 'beta_negative': scores.get('beta_negative')}
+ params = {
+ "p": p,
+ "n": n,
+ "beta_positive": scores.get("beta_positive"),
+ "beta_negative": scores.get("beta_negative"),
+ }
params |= create_intervals(scores, eps)
tp = Interval(0, p)
@@ -350,61 +417,94 @@ def _check_scores_tptn_intervals(p: int,
score_names = list(scores.keys())
for idx, score0 in enumerate(score_names):
- for score1 in score_names[idx+1:]:
- detail = {'base_score_0': score0,
- 'base_score_1': score1,
- 'base_score_0_interval': params[score0].to_tuple(),
- 'base_score_1_interval': params[score1].to_tuple()}
+ for score1 in score_names[idx + 1 :]:
+ detail = {
+ "base_score_0": score0,
+ "base_score_1": score1,
+ "base_score_0_interval": params[score0].to_tuple(),
+ "base_score_1_interval": params[score1].to_tuple(),
+ }
if tuple(sorted([score0, score1])) not in solution_specifications:
- logger.info('there is no solution for %s and %s', score0, score1)
- details.append(detail | {'inconsistency': False,
- 'explanation': 'there is no solution for the pair'})
+ logger.info("there is no solution for %s and %s", score0, score1)
+ details.append(
+ detail
+ | {
+ "inconsistency": False,
+ "explanation": "there is no solution for the pair",
+ }
+ )
continue
- logger.info('evaluating the tp and tn solution for %s and %s', score0, score1)
+ logger.info(
+ "evaluating the tp and tn solution for %s and %s", score0, score1
+ )
- sols = solution_specifications[(tuple(sorted([score0, score1])))]\
- .evaluate(params)
+ sols = solution_specifications[(tuple(sorted([score0, score1])))].evaluate(
+ params
+ )
if check_all_negative_base(sols):
- details.append(detail | {'inconsistency': True,
- 'explanation': 'all solutions lead to negative bases'})
+ details.append(
+ detail
+ | {
+ "inconsistency": True,
+ "explanation": "all solutions lead to negative bases",
+ }
+ )
logger.info("all negative bases - iteration finished")
break
if check_any_zero_division(sols):
- details.append(detail | {'inconsistency': False,
- 'explanation': 'zero division indicates an '\
- 'underdetermined system'})
+ details.append(
+ detail
+ | {
+ "inconsistency": False,
+ "explanation": "zero division indicates an "
+ "underdetermined system",
+ }
+ )
logger.info("all zero divisions - iteration continued")
continue
- logger.info('intervals before: %s, %s', str(tp.to_tuple()), str(tn.to_tuple()))
+ logger.info(
+ "intervals before: %s, %s", str(tp.to_tuple()), str(tn.to_tuple())
+ )
tp, tn = update_tptn(tp, tn, sols)
- logger.info('intervals after: %s, %s', str(tp.to_tuple()), str(tn.to_tuple()))
-
- details.append(detail | {
- 'tp_after': tp.to_tuple(),
- 'tn_after': tn.to_tuple(),
- 'inconsistency': (tp.is_empty()) and (tn.is_empty())})
+ logger.info(
+ "intervals after: %s, %s", str(tp.to_tuple()), str(tn.to_tuple())
+ )
+
+ details.append(
+ detail
+ | {
+ "tp_after": tp.to_tuple(),
+ "tn_after": tn.to_tuple(),
+ "inconsistency": (tp.is_empty()) and (tn.is_empty()),
+ }
+ )
else:
continue
break
- return {'inconsistency': (tp.is_empty()) and (tn.is_empty()),
- 'tp': tp.to_tuple(),
- 'tn': tn.to_tuple(),
- 'details': details}
-
-def check_scores_tptn_pairs(p: int,
- n: int,
- scores: dict,
- eps,
- *,
- numerical_tolerance: float = NUMERICAL_TOLERANCE,
- solve_for: str = None,
- prefilter_by_pairs: bool = False) -> dict:
+ return {
+ "inconsistency": (tp.is_empty()) and (tn.is_empty()),
+ "tp": tp.to_tuple(),
+ "tn": tn.to_tuple(),
+ "details": details,
+ }
+
+
+def check_scores_tptn_pairs(
+ p: int,
+ n: int,
+ scores: dict,
+ eps,
+ *,
+ numerical_tolerance: float = NUMERICAL_TOLERANCE,
+ solve_for: str = None,
+ prefilter_by_pairs: bool = False
+) -> dict:
"""
Check scores by iteratively reducing the set of feasible ``tp``, ``tn`` pairs.
@@ -429,18 +529,31 @@ def check_scores_tptn_pairs(p: int,
by using the solutions for the score pairs.
"""
if not prefilter_by_pairs:
- return _check_scores_tptn_pairs(p, n, scores, eps,
- numerical_tolerance=numerical_tolerance,
- solve_for=solve_for)
- results_interval = _check_scores_tptn_intervals(p, n, scores, eps,
- numerical_tolerance=numerical_tolerance)
-
- results = _check_scores_tptn_pairs(p, n, scores, eps,
- numerical_tolerance=numerical_tolerance,
- solve_for=solve_for,
- init_tptn_intervals={'tp': results_interval['tp'],
- 'tn': results_interval['tn']})
-
- results['prefiltering_details'] = results_interval
+ return _check_scores_tptn_pairs(
+ p,
+ n,
+ scores,
+ eps,
+ numerical_tolerance=numerical_tolerance,
+ solve_for=solve_for,
+ )
+ results_interval = _check_scores_tptn_intervals(
+ p, n, scores, eps, numerical_tolerance=numerical_tolerance
+ )
+
+ results = _check_scores_tptn_pairs(
+ p,
+ n,
+ scores,
+ eps,
+ numerical_tolerance=numerical_tolerance,
+ solve_for=solve_for,
+ init_tptn_intervals={
+ "tp": results_interval["tp"],
+ "tn": results_interval["tn"],
+ },
+ )
+
+ results["prefiltering_details"] = results_interval
return results
diff --git a/mlscorecheck/individual/_complex_interval.py b/mlscorecheck/individual/_complex_interval.py
index 85c6d4d..1e94cfb 100644
--- a/mlscorecheck/individual/_complex_interval.py
+++ b/mlscorecheck/individual/_complex_interval.py
@@ -4,12 +4,14 @@
from ._interval import Interval, IntervalUnion
-__all__ = ['ComplexInterval']
+__all__ = ["ComplexInterval"]
+
class ComplexInterval:
"""
The class represents a complex interval
"""
+
def __init__(self, real, imag):
"""
Constructor of the complex interval
@@ -88,8 +90,10 @@ def __mul__(self, other):
if not isinstance(other, ComplexInterval):
return ComplexInterval(self.real * other, self.imag * other)
- return ComplexInterval(self.real * other.real - self.imag * other.imag,
- self.real * other.imag + self.imag * other.real)
+ return ComplexInterval(
+ self.real * other.real - self.imag * other.imag,
+ self.real * other.imag + self.imag * other.real,
+ )
def __rmul__(self, other):
"""
@@ -118,8 +122,10 @@ def __truediv__(self, other):
norm = other.real**2 + other.imag**2
- return ComplexInterval((self.real * other.real + self.imag * other.imag)/norm,
- (self.imag * other.real - self.real * other.imag)/norm)
+ return ComplexInterval(
+ (self.real * other.real + self.imag * other.imag) / norm,
+ (self.imag * other.real - self.real * other.imag) / norm,
+ )
def __rtruediv__(self, other):
"""
diff --git a/mlscorecheck/individual/_expression.py b/mlscorecheck/individual/_expression.py
index c9d949e..32b669f 100644
--- a/mlscorecheck/individual/_expression.py
+++ b/mlscorecheck/individual/_expression.py
@@ -6,17 +6,17 @@
from ..core import safe_eval
from ._interval import sqrt
-__all__ = ['Expression']
+__all__ = ["Expression"]
+
class Expression:
"""
The class represents a formal expression
"""
- def __init__(self,
- expression: str,
- symbols: list,
- functional_symbols: list = None,
- **kwargs):
+
+ def __init__(
+ self, expression: str, symbols: list, functional_symbols: list = None, **kwargs
+ ):
"""
The constructor of the expression
@@ -29,7 +29,7 @@ def __init__(self,
_ = kwargs
if functional_symbols is None:
- self.functional_symbols = {'sqrt': sqrt}
+ self.functional_symbols = {"sqrt": sqrt}
self.expression = expression
self.symbols = symbols
@@ -44,8 +44,7 @@ def evaluate(self, subs: dict):
Returns:
numeric|Interval|IntervalUnion: the result of the evaluation
"""
- subs = {**{symbol: subs[symbol] for symbol in self.symbols},
- 'sqrt': sqrt}
+ subs = {**{symbol: subs[symbol] for symbol in self.symbols}, "sqrt": sqrt}
return safe_eval(self.expression, subs)
def to_dict(self) -> dict:
@@ -55,6 +54,8 @@ def to_dict(self) -> dict:
Returns:
dict: the dictionary representation
"""
- return {'expression': self.expression,
- 'symbols': self.symbols,
- 'functional_symbols': self.functional_symbols}
+ return {
+ "expression": self.expression,
+ "symbols": self.symbols,
+ "functional_symbols": self.functional_symbols,
+ }
diff --git a/mlscorecheck/individual/_interval.py b/mlscorecheck/individual/_interval.py
index 934e1a0..5e5f3dc 100644
--- a/mlscorecheck/individual/_interval.py
+++ b/mlscorecheck/individual/_interval.py
@@ -4,9 +4,8 @@
import numpy as np
-__all__ = ['Interval',
- 'IntervalUnion',
- 'sqrt']
+__all__ = ["Interval", "IntervalUnion", "sqrt"]
+
def sqrt(obj):
"""
@@ -19,14 +18,16 @@ def sqrt(obj):
Interval|IntervalUnion|numeric: the square root of the parameter
"""
- result = obj**0.5 if isinstance(obj, (Interval, IntervalUnion)) else obj**0.5
+ result = obj**0.5 if isinstance(obj, (Interval, IntervalUnion)) else obj ** 0.5
return result
+
class Interval:
"""
The interval abstraction
"""
+
def __init__(self, lower_bound, upper_bound):
"""
Constructor of the interval
@@ -80,9 +81,15 @@ def intersection(self, other):
if isinstance(other, IntervalUnion):
return IntervalUnion([self]).intersection(other)
- if self.lower_bound >= other.lower_bound and self.upper_bound <= other.upper_bound:
+ if (
+ self.lower_bound >= other.lower_bound
+ and self.upper_bound <= other.upper_bound
+ ):
return Interval(self.lower_bound, self.upper_bound)
- if self.lower_bound <= other.lower_bound and self.upper_bound >= other.upper_bound:
+ if (
+ self.lower_bound <= other.lower_bound
+ and self.upper_bound >= other.upper_bound
+ ):
return Interval(other.lower_bound, other.upper_bound)
tmp_lower = self
@@ -153,11 +160,15 @@ def __add__(self, other):
return other + self
if not isinstance(other, Interval):
- return Interval(lower_bound=self.lower_bound + other,
- upper_bound=self.upper_bound + other)
+ return Interval(
+ lower_bound=self.lower_bound + other,
+ upper_bound=self.upper_bound + other,
+ )
- return Interval(lower_bound=self.lower_bound + other.lower_bound,
- upper_bound=self.upper_bound + other.upper_bound)
+ return Interval(
+ lower_bound=self.lower_bound + other.lower_bound,
+ upper_bound=self.upper_bound + other.upper_bound,
+ )
def __radd__(self, other):
"""
@@ -185,11 +196,15 @@ def __sub__(self, other):
return (-1) * other + self
if not isinstance(other, Interval):
- return Interval(lower_bound=self.lower_bound - other,
- upper_bound=self.upper_bound - other)
+ return Interval(
+ lower_bound=self.lower_bound - other,
+ upper_bound=self.upper_bound - other,
+ )
- return Interval(lower_bound=self.lower_bound - other.upper_bound,
- upper_bound=self.upper_bound - other.lower_bound)
+ return Interval(
+ lower_bound=self.lower_bound - other.upper_bound,
+ upper_bound=self.upper_bound - other.lower_bound,
+ )
def __rsub__(self, other):
"""
@@ -217,16 +232,19 @@ def __mul__(self, other):
return other * self
if not isinstance(other, Interval):
- return Interval(lower_bound=min(self.lower_bound * other, self.upper_bound*other),
- upper_bound=max(self.upper_bound * other, self.lower_bound*other))
+ return Interval(
+ lower_bound=min(self.lower_bound * other, self.upper_bound * other),
+ upper_bound=max(self.upper_bound * other, self.lower_bound * other),
+ )
term0 = self.lower_bound * other.lower_bound
term1 = self.lower_bound * other.upper_bound
term2 = self.upper_bound * other.lower_bound
term3 = self.upper_bound * other.upper_bound
- return Interval(min(term0, term1, term2, term3),
- max(term0, term1, term2, term3))
+ return Interval(
+ min(term0, term1, term2, term3), max(term0, term1, term2, term3)
+ )
def __rmul__(self, other):
"""
@@ -251,7 +269,7 @@ def __truediv__(self, other):
Interval: the ratio
"""
if isinstance(other, IntervalUnion):
- return (1.0/other) * self
+ return (1.0 / other) * self
if not isinstance(other, Interval):
if other >= 0:
@@ -261,21 +279,19 @@ def __truediv__(self, other):
lower_bound = self.upper_bound / other
upper_bound = self.lower_bound / other
- return Interval(lower_bound=lower_bound,
- upper_bound=upper_bound)
+ return Interval(lower_bound=lower_bound, upper_bound=upper_bound)
if (other.upper_bound == 0) and (other.lower_bound != 0):
- return self * Interval(-np.inf, 1.0/other.lower_bound)
+ return self * Interval(-np.inf, 1.0 / other.lower_bound)
if (other.lower_bound == 0) and (other.upper_bound != 0):
- return self * Interval(1.0/other.upper_bound, np.inf)
+ return self * Interval(1.0 / other.upper_bound, np.inf)
if (other.lower_bound > 0) or (other.upper_bound < 0):
- return self * Interval(1.0/other.upper_bound,
- 1.0/other.lower_bound)
+ return self * Interval(1.0 / other.upper_bound, 1.0 / other.lower_bound)
- res_0 = Interval(-np.inf, 1.0/other.lower_bound)
- res_1 = Interval(1.0/other.upper_bound, np.inf)
+ res_0 = Interval(-np.inf, 1.0 / other.lower_bound)
+ res_1 = Interval(1.0 / other.upper_bound, np.inf)
return IntervalUnion([self * res_0, self * res_1])
@@ -315,7 +331,10 @@ def __eq__(self, other) -> bool:
"""
if not isinstance(other, Interval):
return False
- return self.lower_bound == other.lower_bound and self.upper_bound == other.upper_bound
+ return (
+ self.lower_bound == other.lower_bound
+ and self.upper_bound == other.upper_bound
+ )
def __ne__(self, other) -> bool:
"""
@@ -336,7 +355,7 @@ def __neg__(self):
Returns:
Interval: the negated interval
"""
- return (-1)*self
+ return (-1) * self
def __pow__(self, other):
"""
@@ -356,9 +375,13 @@ def __pow__(self, other):
if self.lower_bound > 0 or self.upper_bound < 0:
lower_bound = self.lower_bound**other
upper_bound = self.upper_bound**other
- res = Interval(min(lower_bound, upper_bound), max(lower_bound, upper_bound))
+ res = Interval(
+ min(lower_bound, upper_bound), max(lower_bound, upper_bound)
+ )
else:
- res = Interval(0, max(self.lower_bound**other, self.upper_bound**other))
+ res = Interval(
+ 0, max(self.lower_bound**other, self.upper_bound**other)
+ )
elif int(other) % 2 == 1:
lower_bound = self.lower_bound**other
upper_bound = self.upper_bound**other
@@ -366,10 +389,24 @@ def __pow__(self, other):
return res
+ def representing_int(self):
+ """
+ Returns a representative integer
+
+ Returns:
+ int: a representative element of the interval
+ """
+ shrunk = self.shrink_to_integers()
+ if not shrunk.is_empty():
+ return shrunk.lower_bound
+ return None
+
+
class IntervalUnion:
"""
The interval union abstraction
"""
+
def __init__(self, intervals):
"""
Constructor of the object
@@ -378,6 +415,7 @@ def __init__(self, intervals):
intervals (Interval|tuple|list(Interval)): a specification of one interval
or a list of intervals
"""
+
if isinstance(intervals, Interval):
self.intervals = [intervals]
elif isinstance(intervals, tuple):
@@ -386,7 +424,9 @@ def __init__(self, intervals):
if isinstance(intervals[0], Interval):
self.intervals = intervals
else:
- self.intervals = [Interval(interval[0], interval[1]) for interval in intervals]
+ self.intervals = [
+ Interval(interval[0], interval[1]) for interval in intervals
+ ]
else:
self.intervals = intervals
@@ -415,13 +455,22 @@ def simplify(self):
to_drop = []
for idx, int0 in enumerate(intervals):
for jdx, int1 in enumerate(intervals):
- if idx < jdx:
- if (int0.lower_bound <= int1.lower_bound) \
- and (int0.upper_bound >= int1.upper_bound):
+ if idx != jdx:
+ if (
+ (int0.lower_bound == int1.lower_bound)
+ and (int0.upper_bound == int1.upper_bound)
+ and idx > jdx
+ ):
+ continue
+ if (int0.lower_bound <= int1.lower_bound) and (
+ int0.upper_bound >= int1.upper_bound
+ ):
# the interval widh index jdx is contained in the interval with index idx
to_drop.append(jdx)
- intervals = [interval for idx, interval in enumerate(intervals) if not idx in to_drop]
+ intervals = [
+ interval for idx, interval in enumerate(intervals) if not idx in to_drop
+ ]
# chaining the intervals
sorted_intervals = sorted(intervals, key=lambda x: x.lower_bound)
@@ -431,7 +480,9 @@ def simplify(self):
for idx in range(1, len(sorted_intervals)):
if interval.upper_bound >= sorted_intervals[idx].lower_bound:
- interval = Interval(interval.lower_bound, sorted_intervals[idx].upper_bound)
+ interval = Interval(
+ interval.lower_bound, sorted_intervals[idx].upper_bound
+ )
else:
final_intervals.append(interval)
interval = sorted_intervals[idx]
@@ -463,9 +514,12 @@ def intersection(self, other):
is empty
"""
if isinstance(other, Interval):
- new_intervals = [other.intersection(interval) for interval in self.intervals]
- new_intervals = [interval for interval in new_intervals
- if interval != Interval(1, 0)]
+ new_intervals = [
+ other.intersection(interval) for interval in self.intervals
+ ]
+ new_intervals = [
+ interval for interval in new_intervals if interval != Interval(1, 0)
+ ]
return IntervalUnion(new_intervals)
intersections = []
@@ -473,8 +527,9 @@ def intersection(self, other):
for int1 in other.intervals:
intersections.append(int0.intersection(int1))
- return IntervalUnion([interval for interval in intersections
- if interval != Interval(1, 0)])
+ return IntervalUnion(
+ [interval for interval in intersections if interval != Interval(1, 0)]
+ )
def integer(self) -> bool:
"""
@@ -493,7 +548,9 @@ def shrink_to_integers(self):
Returns:
IntervalUnion: the shrinked interval union
"""
- return IntervalUnion([interval.shrink_to_integers() for interval in self.intervals])
+ return IntervalUnion(
+ [interval.shrink_to_integers() for interval in self.intervals]
+ )
def integer_counts(self) -> int:
"""
@@ -668,7 +725,7 @@ def __neg__(self):
Returns:
IntervalUnion: the negated interval union
"""
- return (-1)*self
+ return (-1) * self
def __repr__(self) -> str:
"""
@@ -677,8 +734,12 @@ def __repr__(self) -> str:
Returns:
str: the representation
"""
- return ' | '.join([f'({interval.lower_bound}, {interval.upper_bound})'
- for interval in self.intervals])
+ return " | ".join(
+ [
+ f"({interval.lower_bound}, {interval.upper_bound})"
+ for interval in self.intervals
+ ]
+ )
def __eq__(self, other) -> bool:
"""
@@ -728,3 +789,20 @@ def __pow__(self, other):
IntervalUnion: the return of the power operation
"""
return IntervalUnion([interval**other for interval in self.intervals])
+
+ def representing_int(self):
+ """
+ Returns a representative integer
+
+ Returns:
+ int: a representative element of the interval
+ """
+ if len(self.intervals) == 0:
+ return None
+
+ for interval in self.intervals:
+ integer = interval.representing_int()
+ if integer is not None:
+ return integer
+
+ return None
diff --git a/mlscorecheck/individual/_pair_solutions.py b/mlscorecheck/individual/_pair_solutions.py
index cdc48fe..f533425 100644
--- a/mlscorecheck/individual/_pair_solutions.py
+++ b/mlscorecheck/individual/_pair_solutions.py
@@ -13,19 +13,16 @@
from ._interval import Interval, IntervalUnion
from ._expression import Expression
-__all__ = ['load_solutions',
- 'Solution',
- 'Solutions',
- 'solution_specifications']
+__all__ = ["load_solutions", "Solution", "Solutions", "solution_specifications"]
+
class Solution:
"""
Represents one single solution (expressions for tp and tn) and corresponding
non-zero and non-negative conditions as expressions
"""
- def __init__(self,
- solution: dict,
- conditions: list = None):
+
+ def __init__(self, solution: dict, conditions: list = None):
"""
Constructor of the solution
@@ -40,12 +37,12 @@ def __init__(self,
self.all_symbols = set()
for item in self.solution.values():
- self.all_symbols = self.all_symbols.union(item['symbols'])
+ self.all_symbols = self.all_symbols.union(item["symbols"])
for cond in self.conditions:
- self.all_symbols = self.all_symbols.union(cond['symbols'])
+ self.all_symbols = self.all_symbols.union(cond["symbols"])
- self.conditions = sorted(self.conditions, key=lambda x: -x['depth'])
+ self.conditions = sorted(self.conditions, key=lambda x: -x["depth"])
def to_dict(self):
"""
@@ -54,8 +51,7 @@ def to_dict(self):
Returns:
dict: the dictionary representation
"""
- return {'solution': self.solution,
- 'conditions': self.conditions}
+ return {"solution": self.solution, "conditions": self.conditions}
def check_non_negative(self, value) -> bool:
"""
@@ -88,10 +84,9 @@ def check_non_zero(self, value) -> bool:
Returns:
bool: True if the condition fails, False otherwise
"""
- return ((isinstance(value, (Interval, IntervalUnion))
- and value.contains(0))
- or (not isinstance(value, (Interval, IntervalUnion))
- and abs(value) < 1e-8))
+ return (isinstance(value, (Interval, IntervalUnion)) and value.contains(0)) or (
+ not isinstance(value, (Interval, IntervalUnion)) and abs(value) < 1e-8
+ )
def evaluate(self, subs):
"""
@@ -108,41 +103,41 @@ def evaluate(self, subs):
message = None
term = None
for condition in self.conditions:
- if condition['mode'] == 'non-negative':
+ if condition["mode"] == "non-negative":
value = Expression(**condition).evaluate(subs)
if self.check_non_negative(value):
- message = 'negative base'
- term = condition['expression']
+ message = "negative base"
+ term = condition["expression"]
break
- elif condition['mode'] == 'non-zero':
+ elif condition["mode"] == "non-zero":
value = Expression(**condition).evaluate(subs)
if self.check_non_zero(value):
- message = 'zero division'
- term = condition['expression']
+ message = "zero division"
+ term = condition["expression"]
break
if message is not None:
- return {'tp': None,
- 'tn': None,
- 'message': message,
- 'term': term}
+ return {"tp": None, "tn": None, "message": message, "term": term}
- res = {key: Expression(**value).evaluate(subs) for key, value in self.solution.items()}
- if 'tp' in self.solution:
- res['tp_formula'] = self.solution['tp']['expression']
+ res = {
+ key: Expression(**value).evaluate(subs)
+ for key, value in self.solution.items()
+ }
+ if "tp" in self.solution:
+ res["tp_formula"] = self.solution["tp"]["expression"]
- if 'tn' in self.solution:
- res['tn_formula'] = self.solution['tn']['expression']
+ if "tn" in self.solution:
+ res["tn_formula"] = self.solution["tn"]["expression"]
return res
+
class Solutions:
"""
Represents all solutions to a particular problem
"""
- def __init__(self,
- scores: list,
- solutions: list):
+
+ def __init__(self, scores: list, solutions: list):
"""
The constructor of the object
@@ -161,8 +156,10 @@ def to_dict(self) -> dict:
dict: the dictionary representation
"""
- return {'scores': self.scores,
- 'solutions': [sol.to_dict() for sol in self.solutions]}
+ return {
+ "scores": self.scores,
+ "solutions": [sol.to_dict() for sol in self.solutions],
+ }
def evaluate(self, subs):
"""
@@ -182,6 +179,7 @@ def evaluate(self, subs):
return results
+
def load_solutions():
"""
Load the solutions
@@ -189,27 +187,32 @@ def load_solutions():
Returns:
dict: the dictionary of the solutions
"""
- sio = files('mlscorecheck').joinpath(os.path.join('individual', 'solutions.json')).read_text() # pylint: disable=unspecified-encoding
+ sio = (
+ files("mlscorecheck")
+ .joinpath(os.path.join("individual", "solutions.json"))
+ .read_text()
+ ) # pylint: disable=unspecified-encoding
solutions_dict = json.loads(sio)
results = {}
- for sol in solutions_dict['solutions']:
- scores = list(sol['scores'])
- if 'p4' not in scores:
+ for sol in solutions_dict["solutions"]:
+ scores = list(sol["scores"])
+ if "p4" not in scores:
results[tuple(sorted(scores))] = Solutions(**sol)
# removing the solutions containing complex values
- del results[('fm', 'gm')]
- del results[('fm', 'mk')]
- #del results[('fm', 'p4')] # goes to complex
- del results[('fm', 'upm')] # goes to complex
- #del results[('dor', 'p4')] # goes to complex
- del results[('dor', 'upm')] # goes to complex
- del results[('gm', 'mk')] # goes to complex
- del results[('gm', 'mcc')] # goes to complex when tn = 0 (maybe other times too)
+ del results[("fm", "gm")]
+ del results[("fm", "mk")]
+ # del results[('fm', 'p4')] # goes to complex
+ del results[("fm", "upm")] # goes to complex
+ # del results[('dor', 'p4')] # goes to complex
+ del results[("dor", "upm")] # goes to complex
+ del results[("gm", "mk")] # goes to complex
+ del results[("gm", "mcc")] # goes to complex when tn = 0 (maybe other times too)
return results
+
solution_specifications = load_solutions()
diff --git a/mlscorecheck/individual/_problem_generator.py b/mlscorecheck/individual/_problem_generator.py
index 8409c46..bac76db 100644
--- a/mlscorecheck/individual/_problem_generator.py
+++ b/mlscorecheck/individual/_problem_generator.py
@@ -4,20 +4,28 @@
import numpy as np
-from ._calculate_scores import calculate_scores
-
-__all__ = ['generate_problems',
- 'generate_1_problem',
- 'generate_problem_and_scores']
-
-def generate_problem_and_scores(*,
- max_p: int = 1000,
- max_n: int = 1000,
- zeros: list = None,
- add_complements: bool = None,
- score_subset: list = None,
- rounding_decimals: int = None,
- random_state=None) -> (dict, dict):
+from ..scores import calculate_scores
+
+__all__ = [
+ "generate_problems",
+ "generate_1_problem",
+ "generate_problem_and_scores",
+ "generate_multiclass_dataset",
+ "sample_multiclass_dataset",
+ "create_confusion_matrix",
+]
+
+
+def generate_problem_and_scores(
+ *,
+ max_p: int = 1000,
+ max_n: int = 1000,
+ zeros: list = None,
+ add_complements: bool = None,
+ score_subset: list = None,
+ rounding_decimals: int = None,
+ random_state=None
+) -> (dict, dict):
"""
Generates a random problem and random but feasible scores
@@ -33,24 +41,29 @@ def generate_problem_and_scores(*,
Returns:
dict, dict: the problem and the scores
"""
- evaluation, problem = generate_1_problem(max_p=max_p,
- max_n=max_n,
- zeros=zeros,
- add_complements=add_complements,
- random_state=random_state)
- evaluation['beta_negative'] = 2
- evaluation['beta_positive'] = 2
+ evaluation, problem = generate_1_problem(
+ max_p=max_p,
+ max_n=max_n,
+ zeros=zeros,
+ add_complements=add_complements,
+ random_state=random_state,
+ )
+ evaluation["beta_negative"] = 2
+ evaluation["beta_positive"] = 2
scores = calculate_scores(evaluation, rounding_decimals=rounding_decimals)
if score_subset is not None:
scores = {key: value for key, value in scores.items() if key in score_subset}
return problem, scores
-def generate_1_problem(*,
- max_p: int = 1000,
- max_n: int = 1000,
- zeros: list = None,
- add_complements: bool = False,
- random_state=None) -> dict:
+
+def generate_1_problem(
+ *,
+ max_p: int = 1000,
+ max_n: int = 1000,
+ zeros: list = None,
+ add_complements: bool = False,
+ random_state=None
+) -> dict:
"""
Generates a random problem
@@ -70,38 +83,41 @@ def generate_1_problem(*,
if zeros is None:
zeros = []
- p = int(random_state.randint(2, max_p+1))
- n = int(random_state.randint(2, max_n+1))
+ p = int(random_state.randint(2, max_p + 1))
+ n = int(random_state.randint(2, max_n + 1))
- if 'tp' in zeros:
+ if "tp" in zeros:
tp = 0
- elif 'fn' in zeros:
+ elif "fn" in zeros:
tp = p
else:
tp = int(random_state.randint(1, p))
- if 'tn' in zeros:
+ if "tn" in zeros:
tn = 0
- elif 'fp' in zeros:
+ elif "fp" in zeros:
tn = n
else:
tn = int(random_state.randint(1, n))
- result = {'p': p, 'n': n, 'tp': tp, 'tn': tn}
+ result = {"p": p, "n": n, "tp": tp, "tn": tn}
if add_complements:
- result['fn'] = p - tp
- result['fp'] = n - tn
-
- return result, {'p': result['p'], 'n': result['n']}
-
-def generate_problems(*,
- n_problems: int = 1,
- max_p: int = 1000,
- max_n: int = 1000,
- zeros: list = None,
- add_complements: bool = False,
- random_state=None) -> (dict, dict):
+ result["fn"] = p - tp
+ result["fp"] = n - tn
+
+ return result, {"p": result["p"], "n": result["n"]}
+
+
+def generate_problems(
+ *,
+ n_problems: int = 1,
+ max_p: int = 1000,
+ max_n: int = 1000,
+ zeros: list = None,
+ add_complements: bool = False,
+ random_state=None
+) -> (dict, dict):
"""
Generates a random problem
@@ -122,12 +138,98 @@ def generate_problems(*,
evaluations, problems = [], []
for _ in range(n_problems):
- evaluation, problem = generate_1_problem(max_p=max_p,
- max_n=max_n,
- zeros=zeros,
- add_complements=add_complements,
- random_state=random_state)
+ evaluation, problem = generate_1_problem(
+ max_p=max_p,
+ max_n=max_n,
+ zeros=zeros,
+ add_complements=add_complements,
+ random_state=random_state,
+ )
evaluations.append(evaluation)
problems.append(problem)
return (evaluations[0], problems[0]) if n_problems == 1 else (evaluations, problems)
+
+
+def generate_multiclass_dataset(
+ random_state=None,
+ max_n_classes: int = 5,
+ min_n_classes: int = 3,
+ max_class_size: int = 200,
+ min_class_size: int = 10,
+) -> dict:
+ """
+ Generates a multiclass dataset
+
+ Args:
+ random_state (None|int|np.random.RandomState): the random seed to use
+ max_n_classes (int): the maximum number of classes
+ min_n_classes (int): the minimum number of classes
+ max_class_size (int): the maximum class size
+ min_class_size (int): the minimum class size
+
+ Returns:
+ dict: a multiclass dataset
+ """
+ if not isinstance(random_state, np.random.RandomState):
+ random_state = np.random.RandomState(random_state)
+
+ n_classes = random_state.randint(min_n_classes, max_n_classes + 1)
+ classes = {
+ class_idx: random_state.randint(min_class_size, max_class_size + 1)
+ for class_idx in range(n_classes)
+ }
+
+ return classes
+
+
+def create_confusion_matrix(y_true: np.array, y_pred: np.array) -> np.array:
+ """
+ Creates a confusion matrix
+
+ Args:
+ y_true (np.ndarray): the true labels
+ y_pred (np.ndarray): the predicted labels
+
+ Returns:
+ np.ndarray: the confusion matrix
+ """
+ n_classes = max(y_true.max(), y_pred.max()) + 1
+ confusion_matrix = np.zeros(shape=(n_classes, n_classes), dtype=int)
+
+ for true, pred in zip(y_true, y_pred):
+ confusion_matrix[true, pred] += 1
+
+ return confusion_matrix
+
+
+def sample_multiclass_dataset(dataset: dict, random_state=None) -> np.array:
+ """
+ Samples a multiclass dataset
+
+ Args:
+ dataset (dict): the dataset to sample
+ random_state (None|int|np.random.RandomState): the random seed to use
+
+ Returns:
+ np.array: the confusion matrix
+ """
+ if not isinstance(random_state, np.random.RandomState):
+ random_state = np.random.RandomState(random_state)
+
+ sample = np.zeros(shape=(len(dataset), len(dataset)), dtype=int)
+
+ for class_idx, count in dataset.items():
+ sample_row = random_state.multinomial(
+ count, np.ones(len(dataset)) / len(dataset), size=1
+ )[0]
+
+ # while np.any(sample_row == 0):
+ # zidx = np.where(sample_row == 0)[0][0]
+ # nzidx = np.where(sample_row > 1)[0][0]
+ # sample_row[zidx] += 1
+ # sample_row[nzidx] -= 1
+
+ sample[class_idx, :] = sample_row
+
+ return sample
diff --git a/mlscorecheck/individual/_tptn_solution_bundles.py b/mlscorecheck/individual/_tptn_solution_bundles.py
index 6afe6bc..da65932 100644
--- a/mlscorecheck/individual/_tptn_solution_bundles.py
+++ b/mlscorecheck/individual/_tptn_solution_bundles.py
@@ -2,50 +2,53 @@
This module implements some bundles of tp, tn solutions for scores
"""
-from ._tptn_solutions import (mcc_tp,
- mcc_tn,
- acc_tp,
- acc_tn,
- sens_tp,
- spec_tn,
- ppv_tp,
- ppv_tn,
- npv_tp,
- npv_tn,
- fbp_tp,
- fbp_tn,
- f1p_tp,
- f1p_tn,
- fbn_tp,
- fbn_tn,
- f1n_tp,
- f1n_tn,
- upm_tp,
- upm_tn,
- gm_tp,
- gm_tn,
- fm_tp,
- fm_tn,
- mk_tp,
- mk_tn,
- lrp_tp,
- lrp_tn,
- lrn_tp,
- lrn_tn,
- bm_tp,
- bm_tn,
- pt_tp,
- pt_tn,
- dor_tp,
- dor_tn,
- ji_tp,
- ji_tn,
- bacc_tp,
- bacc_tn,
- kappa_tp,
- kappa_tn)
+from ._tptn_solutions import (
+ mcc_tp,
+ mcc_tn,
+ acc_tp,
+ acc_tn,
+ sens_tp,
+ spec_tn,
+ ppv_tp,
+ ppv_tn,
+ npv_tp,
+ npv_tn,
+ fbp_tp,
+ fbp_tn,
+ f1p_tp,
+ f1p_tn,
+ fbn_tp,
+ fbn_tn,
+ f1n_tp,
+ f1n_tn,
+ upm_tp,
+ upm_tn,
+ gm_tp,
+ gm_tn,
+ fm_tp,
+ fm_tn,
+ mk_tp,
+ mk_tn,
+ lrp_tp,
+ lrp_tn,
+ lrn_tp,
+ lrn_tn,
+ bm_tp,
+ bm_tn,
+ pt_tp,
+ pt_tn,
+ dor_tp,
+ dor_tn,
+ ji_tp,
+ ji_tn,
+ bacc_tp,
+ bacc_tn,
+ kappa_tp,
+ kappa_tn,
+)
+
+__all__ = ["tp_solutions", "tn_solutions", "tptn_solutions", "is_applicable_tptn"]
-__all__ = ['tp_solutions', 'tn_solutions', 'tptn_solutions', 'is_applicable_tptn']
def is_applicable_tptn(score: str, value: float, to_compute: str) -> bool:
"""
@@ -59,60 +62,86 @@ def is_applicable_tptn(score: str, value: float, to_compute: str) -> bool:
Returns:
bool: True if the setup can be solved, False otherwise
"""
- if score == 'pt' and value == 0.0 and to_compute == 'tp':
+ if score == "pt" and value == 0.0 and to_compute == "tp":
return False
- if score == 'pt' and value == 1.0 and to_compute == 'tn':
+ if score == "pt" and value == 1.0 and to_compute == "tn":
return False
return True
-tp_solutions = {'mcc': mcc_tp,
- 'acc': acc_tp,
- 'sens': sens_tp,
- 'ppv': ppv_tp,
- 'npv': npv_tp,
- 'fbp': fbp_tp,
- 'f1p': f1p_tp,
- 'fbn': fbn_tp,
- 'f1n': f1n_tp,
- 'upm': upm_tp,
- 'gm': gm_tp,
- 'fm': fm_tp,
- 'mk': mk_tp,
- 'lrp': lrp_tp,
- 'lrn': lrn_tp,
- 'bm': bm_tp,
- 'pt': pt_tp,
- 'dor': dor_tp,
- 'ji': ji_tp,
- 'bacc': bacc_tp,
- 'kappa': kappa_tp}
-tn_solutions = {'mcc': mcc_tn,
- 'acc': acc_tn,
- 'spec': spec_tn,
- 'ppv': ppv_tn,
- 'npv': npv_tn,
- 'fbp': fbp_tn,
- 'f1p': f1p_tn,
- 'fbn': fbn_tn,
- 'f1n': f1n_tn,
- 'upm': upm_tn,
- 'gm': gm_tn,
- 'fm': fm_tn,
- 'mk': mk_tn,
- 'lrp': lrp_tn,
- 'lrn': lrn_tn,
- 'bm': bm_tn,
- 'pt': pt_tn,
- 'dor': dor_tn,
- 'ji': ji_tn,
- 'bacc': bacc_tn,
- 'kappa': kappa_tn}
+tp_solutions = {
+ "mcc": mcc_tp,
+ "acc": acc_tp,
+ "sens": sens_tp,
+ "ppv": ppv_tp,
+ "npv": npv_tp,
+ "fbp": fbp_tp,
+ "f1p": f1p_tp,
+ "fbn": fbn_tp,
+ "f1n": f1n_tp,
+ "upm": upm_tp,
+ "gm": gm_tp,
+ "fm": fm_tp,
+ "mk": mk_tp,
+ "lrp": lrp_tp,
+ "lrn": lrn_tp,
+ "bm": bm_tp,
+ "pt": pt_tp,
+ "dor": dor_tp,
+ "ji": ji_tp,
+ "bacc": bacc_tp,
+ "kappa": kappa_tp,
+}
+
+tn_solutions = {
+ "mcc": mcc_tn,
+ "acc": acc_tn,
+ "spec": spec_tn,
+ "ppv": ppv_tn,
+ "npv": npv_tn,
+ "fbp": fbp_tn,
+ "f1p": f1p_tn,
+ "fbn": fbn_tn,
+ "f1n": f1n_tn,
+ "upm": upm_tn,
+ "gm": gm_tn,
+ "fm": fm_tn,
+ "mk": mk_tn,
+ "lrp": lrp_tn,
+ "lrn": lrn_tn,
+ "bm": bm_tn,
+ "pt": pt_tn,
+ "dor": dor_tn,
+ "ji": ji_tn,
+ "bacc": bacc_tn,
+ "kappa": kappa_tn,
+}
-tptn_solutions = {key: {'tp': tp_solutions.get(key),
- 'tn': tn_solutions.get(key)} for key in ['mcc', 'acc', 'spec', 'sens',
- 'ppv', 'npv', 'fbp', 'f1p', 'fbn',
- 'f1n', 'upm', 'gm', 'fm', 'mk', 'lrp',
- 'lrn', 'bm', 'pt', 'dor', 'ji',
- 'bacc', 'kappa']}
+tptn_solutions = {
+ key: {"tp": tp_solutions.get(key), "tn": tn_solutions.get(key)}
+ for key in [
+ "mcc",
+ "acc",
+ "spec",
+ "sens",
+ "ppv",
+ "npv",
+ "fbp",
+ "f1p",
+ "fbn",
+ "f1n",
+ "upm",
+ "gm",
+ "fm",
+ "mk",
+ "lrp",
+ "lrn",
+ "bm",
+ "pt",
+ "dor",
+ "ji",
+ "bacc",
+ "kappa",
+ ]
+}
diff --git a/mlscorecheck/individual/_utils.py b/mlscorecheck/individual/_utils.py
index 0e5ce00..657dc2f 100644
--- a/mlscorecheck/individual/_utils.py
+++ b/mlscorecheck/individual/_utils.py
@@ -7,20 +7,24 @@
import numpy as np
from ._pair_solutions import solution_specifications
-from ..scores import (score_functions_without_complements,
- score_functions_standardized_without_complements,
- score_function_aliases,
- score_function_complements,
- score_specifications)
+from ..scores import (
+ score_functions_without_complements,
+ score_functions_standardized_without_complements,
+ score_function_aliases,
+ score_function_complements,
+ score_specifications,
+)
from ._interval import Interval, IntervalUnion
from ..core import NUMERICAL_TOLERANCE
-__all__ = ['create_intervals',
- 'create_problems_2',
- 'resolve_aliases_and_complements',
- 'is_less_than_zero',
- 'is_zero',
- 'unify_results']
+__all__ = [
+ "create_intervals",
+ "create_problems_2",
+ "resolve_aliases_and_complements",
+ "is_less_than_zero",
+ "is_zero",
+ "unify_results",
+]
solutions = solution_specifications
score_descriptors = score_specifications
@@ -30,6 +34,7 @@
functions = score_functions_without_complements
functions_standardized = score_functions_standardized_without_complements
+
def resolve_aliases_and_complements(scores: dict) -> dict:
"""
Standardizing the scores by resolving aliases and complements
@@ -56,9 +61,10 @@ def resolve_aliases_and_complements(scores: dict) -> dict:
return complemented
-def create_intervals(scores: dict,
- eps,
- numerical_tolerance: float = NUMERICAL_TOLERANCE) -> dict:
+
+def create_intervals(
+ scores: dict, eps, numerical_tolerance: float = NUMERICAL_TOLERANCE
+) -> dict:
"""
Turns the scores into intervals using the uncertainty specifications,
the interval for a score will be (score - eps, score + eps).
@@ -83,22 +89,28 @@ def create_intervals(scores: dict,
eps = {key: eps for key in scores}
# creating the intervals
- intervals = {key: Interval(val - eps[key] - numerical_tolerance,
- val + eps[key] + numerical_tolerance)
- for key, val in scores.items()}
+ intervals = {
+ key: Interval(
+ val - eps[key] - numerical_tolerance, val + eps[key] + numerical_tolerance
+ )
+ for key, val in scores.items()
+ }
# trimming the intervals into the domains of the scores
# for example, to prevent acc - eps < 0 implying a negative subinterval for
# accuracy
for key in intervals:
if key in score_descriptors:
- lower_bound = score_descriptors[key].get('lower_bound', -np.inf)
- upper_bound = score_descriptors[key].get('upper_bound', np.inf)
+ lower_bound = score_descriptors[key].get("lower_bound", -np.inf)
+ upper_bound = score_descriptors[key].get("upper_bound", np.inf)
- intervals[key] = intervals[key].intersection(Interval(lower_bound, upper_bound))
+ intervals[key] = intervals[key].intersection(
+ Interval(lower_bound, upper_bound)
+ )
return intervals
+
def create_problems_2(scores: list) -> list:
"""
Given a set of scores, this function generates all test case specifications.
@@ -114,12 +126,13 @@ def create_problems_2(scores: list) -> list:
bases = list(itertools.combinations(scores, 2))
problems = []
for base0, base1 in bases:
- problems.extend((base0, base1, score)
- for score in scores
- if score not in {base0, base1})
+ problems.extend(
+ (base0, base1, score) for score in scores if score not in {base0, base1}
+ )
return problems
+
def is_less_than_zero(value) -> bool:
"""
Checks if the parameter is less than zero
@@ -136,6 +149,7 @@ def is_less_than_zero(value) -> bool:
return value.upper_bound < 0
return all(interval.upper_bound < 0 for interval in value.intervals)
+
def is_zero(value, tolerance: float = 1e-8) -> bool:
"""
Checks if the parameter is zero
@@ -151,6 +165,7 @@ def is_zero(value, tolerance: float = 1e-8) -> bool:
return abs(value) < tolerance
return value.contains(0.0)
+
def unify_results(value_list):
"""
Unifies the list of solutions
@@ -161,6 +176,7 @@ def unify_results(value_list):
Returns:
obj (list|IntervalUnion): the unified result
"""
+
if len(value_list) == 0:
return None
if all(value is None for value in value_list):
@@ -177,4 +193,6 @@ def unify_results(value_list):
elif interval is not None:
intervals.append(Interval(interval, interval))
- return IntervalUnion(intervals)
+ intu = IntervalUnion(intervals)
+
+ return intu
diff --git a/mlscorecheck/scores/__init__.py b/mlscorecheck/scores/__init__.py
index 583fe00..a129437 100644
--- a/mlscorecheck/scores/__init__.py
+++ b/mlscorecheck/scores/__init__.py
@@ -5,3 +5,5 @@
from ._scores import *
from ._scores_standardized import *
from ._score_bundles import *
+from ._multiclass_scores import *
+from ._calculate_scores import *
diff --git a/mlscorecheck/individual/_calculate_scores.py b/mlscorecheck/scores/_calculate_scores.py
similarity index 66%
rename from mlscorecheck/individual/_calculate_scores.py
rename to mlscorecheck/scores/_calculate_scores.py
index 4b9374e..8378398 100644
--- a/mlscorecheck/individual/_calculate_scores.py
+++ b/mlscorecheck/scores/_calculate_scores.py
@@ -5,12 +5,16 @@
import math
+import numpy as np
+
from ..core import safe_call, round_scores, logger
-from ..scores import score_functions_with_solutions
-from ..scores import score_specifications
+from ._score_bundles import score_functions_with_solutions
+from ._score_bundles import score_specifications
+from ._multiclass_scores import multiclass_score_map
__all__ = ['calculate_scores',
- 'calculate_scores_for_lp']
+ 'calculate_scores_for_lp',
+ 'calculate_multiclass_scores']
def calculate_scores_for_lp(problem: dict, score_subset: list = None) -> dict:
"""
@@ -87,3 +91,33 @@ def calculate_scores(problem: dict,
results = round_scores(results, rounding_decimals)
return results
+
+def calculate_multiclass_scores(confusion_matrix: np.array,
+ average=None,
+ *,
+ rounding_decimals: int = None,
+ additional_symbols: dict = None,
+ subset: list = None) -> dict:
+ """
+ Calculates all scores with solutions
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix to calculate scores for
+ average (str): the mode of averaging ('macro'/'micro'/'weighted')
+ rounding_decimals (None|int): the decimal places to round to
+ additional_symbols (None|dict): additional symbols for the substitution
+ subset (None|list): the subset of scores to calculate
+
+ Returns:
+ dict: the calculated scores
+ """
+ additional_symbols = {} if additional_symbols is None else additional_symbols
+ params = {'confusion_matrix': confusion_matrix,
+ 'average': average} | additional_symbols
+ results = {score: safe_call(function, params)
+ for score, function in multiclass_score_map.items()
+ if subset is None or score in subset}
+
+ results = round_scores(results, rounding_decimals)
+
+ return results
diff --git a/mlscorecheck/scores/_multiclass_scores.py b/mlscorecheck/scores/_multiclass_scores.py
new file mode 100644
index 0000000..c7e03b2
--- /dev/null
+++ b/mlscorecheck/scores/_multiclass_scores.py
@@ -0,0 +1,587 @@
+"""
+This module implements the multiclass scores
+"""
+
+import math
+
+import numpy as np
+
+from ..core import safe_call
+
+from ._scores_standardized import (accuracy_standardized,
+ sensitivity_standardized,
+ specificity_standardized,
+ balanced_accuracy_standardized,
+ positive_predictive_value_standardized,
+ negative_predictive_value_standardized,
+ f_beta_positive_standardized,
+ f_beta_negative_standardized,
+ f1_positive_standardized,
+ f1_negative_standardized,
+ unified_performance_measure_standardized,
+ geometric_mean_standardized,
+ fowlkes_mallows_index_standardized,
+ markedness_standardized,
+ positive_likelihood_ratio_standardized,
+ negative_likelihood_ratio_standardized,
+ matthews_correlation_coefficient_standardized,
+ bookmaker_informedness_standardized,
+ prevalence_threshold_standardized,
+ diagnostic_odds_ratio_standardized,
+ jaccard_index_standardized,
+ cohens_kappa_standardized)
+from ._score_bundles import score_specifications
+
+__all__ = ['multiclass_score_macro',
+ 'multiclass_score_micro',
+ 'multiclass_score_weighted',
+ 'multiclass_score',
+ 'multiclass_accuracy',
+ 'multiclass_sensitivity',
+ 'multiclass_specificity',
+ 'multiclass_balanced_accuracy',
+ 'multiclass_positive_predictive_value',
+ 'multiclass_negative_predictive_value',
+ 'multiclass_f_beta_positive',
+ 'multiclass_f_beta_negative',
+ 'multiclass_f1_positive',
+ 'multiclass_f1_negative',
+ 'multiclass_unified_performance_measure',
+ 'multiclass_geometric_mean',
+ 'multiclass_fowlkes_mallows_index',
+ 'multiclass_markedness',
+ 'multiclass_positive_likelihood_ratio',
+ 'multiclass_negative_likelihood_ratio',
+ 'multiclass_matthews_correlation_coefficient',
+ 'multiclass_bookmaker_informedness',
+ 'multiclass_prevalence_threshold',
+ 'multiclass_diagnostic_odds_ratio',
+ 'multiclass_jaccard_index',
+ 'multiclass_cohens_kappa',
+ 'multiclass_score_map']
+
+def multiclass_score_macro(confusion_matrix,
+ score_function,
+ additional_params=None,
+ name=None):
+ """
+ Calculates the multiclass macro average score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ score_function (callable): the score function to use
+ additional_params (None|dict): the additional parameters to use
+ name (None|str): the name of the score
+
+ Returns:
+ float: the score
+ """
+ additional_params = {} if additional_params is None else additional_params
+ additional_params = additional_params | {'sqrt': math.sqrt}
+
+ counts = np.sum(confusion_matrix, axis=1)
+ n_total = np.sum(counts)
+
+ scores = [safe_call(score_function, {'p': count,
+ 'n': n_total - count,
+ 'tp': confusion_matrix[idx, idx],
+ 'tn': np.sum(confusion_matrix[0:idx, 0:idx]) \
+ + np.sum(confusion_matrix[idx+1:, idx+1:]) \
+ + np.sum(confusion_matrix[:idx, idx+1:]) \
+ + np.sum(confusion_matrix[idx+1:, :idx])}
+ | additional_params,
+ score_specifications[name].get('nans'))
+ for idx, count in enumerate(counts)]
+
+ return np.mean(scores)
+
+def multiclass_score_micro(confusion_matrix,
+ score_function,
+ additional_params=None,
+ name=None):
+ """
+ Calculates the multiclass micro average score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ score_function (callable): the score function to use
+ additional_params (None|dict): the additional parameters to use
+ name (None|str): the name of the score
+
+ Returns:
+ float: the score
+ """
+ additional_params = {} if additional_params is None else additional_params
+ additional_params = additional_params | {'sqrt': math.sqrt}
+
+ counts = np.sum(confusion_matrix, axis=1)
+ n_total = np.sum(counts)
+
+ params = {'tp': 0,
+ 'tn': 0,
+ 'p': 0,
+ 'n': 0}
+
+ for idx, count in enumerate(counts):
+ params['p'] += count
+ params['n'] += n_total - count
+ params['tp'] += confusion_matrix[idx, idx]
+ params['tn'] += np.sum(confusion_matrix[0:idx, 0:idx]) \
+ + np.sum(confusion_matrix[idx+1:, idx+1:]) \
+ + np.sum(confusion_matrix[:idx, idx+1:]) \
+ + np.sum(confusion_matrix[idx+1:, :idx])
+
+ return safe_call(score_function,
+ params | additional_params,
+ score_specifications[name].get('nans'))
+
+def multiclass_score_weighted(confusion_matrix,
+ score_function,
+ additional_params=None,
+ name=None):
+ """
+ Calculates the multiclass weighted macro average score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ score_function (callable): the score function to use
+ additional_params (None|dict): the additional parameters to use
+ name (None|str): the name of the score
+
+ Returns:
+ float: the score
+ """
+ additional_params = {} if additional_params is None else additional_params
+ additional_params = additional_params | {'sqrt': math.sqrt}
+
+ counts = np.sum(confusion_matrix, axis=1)
+ n_total = np.sum(counts)
+
+ scores = [safe_call(score_function, {'p': count,
+ 'n': n_total - count,
+ 'tp': confusion_matrix[idx, idx],
+ 'tn': np.sum(confusion_matrix[0:idx, 0:idx]) \
+ + np.sum(confusion_matrix[idx+1:, idx+1:]) \
+ + np.sum(confusion_matrix[:idx, idx+1:]) \
+ + np.sum(confusion_matrix[idx+1:, :idx])}
+ | additional_params,
+ score_specifications[name].get('nans'))\
+ * count / n_total
+ for idx, count in enumerate(counts)]
+
+ return np.sum(scores)
+
+def multiclass_score(confusion_matrix,
+ score_function,
+ average,
+ additional_params=None,
+ name=None):
+ """
+ Calculates the multiclass average score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ score_function (callable): the score function to use
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+ additional_params (None|dict): the additional parameters to use
+ name (None|str): the name of the score
+
+ Returns:
+ float: the score
+ """
+ if average == 'micro':
+ return multiclass_score_micro(confusion_matrix,
+ score_function,
+ additional_params,
+ name)
+ if average == 'macro':
+ return multiclass_score_macro(confusion_matrix,
+ score_function,
+ additional_params,
+ name)
+ if average == 'weighted':
+ return multiclass_score_weighted(confusion_matrix,
+ score_function,
+ additional_params,
+ name)
+
+ raise ValueError(f'averaging {average} is not supported')
+
+def multiclass_accuracy(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ accuracy_standardized,
+ average,
+ name='acc')
+
+def multiclass_sensitivity(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ sensitivity_standardized,
+ average,
+ name='sens')
+
+def multiclass_specificity(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ specificity_standardized,
+ average,
+ name='spec')
+
+def multiclass_balanced_accuracy(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ balanced_accuracy_standardized,
+ average,
+ name='bacc')
+
+def multiclass_positive_predictive_value(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ positive_predictive_value_standardized,
+ average,
+ name='ppv')
+
+def multiclass_negative_predictive_value(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ negative_predictive_value_standardized,
+ average,
+ name='npv')
+
+def multiclass_f_beta_positive(*, beta_positive, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ beta_positive (float): the beta value to be used
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ f_beta_positive_standardized,
+ average,
+ additional_params={'beta_positive': beta_positive},
+ name='fbp')
+
+def multiclass_f_beta_negative(*, beta_negative, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ beta_negative (float): the beta value to be used
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ f_beta_negative_standardized,
+ average,
+ additional_params={'beta_negative': beta_negative},
+ name='fbn')
+
+def multiclass_f1_positive(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ f1_positive_standardized,
+ average,
+ name='f1p')
+
+def multiclass_f1_negative(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ f1_negative_standardized,
+ average,
+ name='f1n')
+
+def multiclass_unified_performance_measure(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ unified_performance_measure_standardized,
+ average,
+ name='upm')
+
+def multiclass_geometric_mean(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ geometric_mean_standardized,
+ average,
+ name='gm')
+
+def multiclass_fowlkes_mallows_index(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ fowlkes_mallows_index_standardized,
+ average,
+ name='fm')
+
+def multiclass_markedness(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ markedness_standardized,
+ average,
+ name='mk')
+
+def multiclass_positive_likelihood_ratio(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ positive_likelihood_ratio_standardized,
+ average,
+ name='lrp')
+
+def multiclass_negative_likelihood_ratio(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ negative_likelihood_ratio_standardized,
+ average,
+ name='lrn')
+
+def multiclass_matthews_correlation_coefficient(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ matthews_correlation_coefficient_standardized,
+ average,
+ name='mcc')
+
+def multiclass_bookmaker_informedness(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ bookmaker_informedness_standardized,
+ average,
+ name='bm')
+
+def multiclass_prevalence_threshold(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ prevalence_threshold_standardized,
+ average,
+ name='pt')
+
+def multiclass_diagnostic_odds_ratio(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ diagnostic_odds_ratio_standardized,
+ average,
+ name='dor')
+
+def multiclass_jaccard_index(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ jaccard_index_standardized,
+ average,
+ name='ji')
+
+def multiclass_cohens_kappa(*, confusion_matrix, average):
+ """
+ Calculates the multiclass score
+
+ Args:
+ confusion_matrix (np.array): the confusion matrix (true labels in rows)
+ average (str): the averaging to be used ('macro'/'micro'/'weighted')
+
+ Returns:
+ float: the score
+ """
+ return multiclass_score(confusion_matrix,
+ cohens_kappa_standardized,
+ average,
+ name='kappa')
+
+multiclass_score_map = {'acc': multiclass_accuracy,
+ 'sens': multiclass_sensitivity,
+ 'spec': multiclass_specificity,
+ 'bacc': multiclass_balanced_accuracy,
+ 'ppv': multiclass_positive_predictive_value,
+ 'npv': multiclass_negative_predictive_value,
+ 'fbp': multiclass_f_beta_positive,
+ 'fbn': multiclass_f_beta_negative,
+ 'f1p': multiclass_f1_positive,
+ 'f1n': multiclass_f1_negative,
+ 'upm': multiclass_unified_performance_measure,
+ 'gm': multiclass_geometric_mean,
+ 'fm': multiclass_fowlkes_mallows_index,
+ 'mk': multiclass_markedness,
+ 'lrp': multiclass_positive_likelihood_ratio,
+ 'lrn': multiclass_negative_likelihood_ratio,
+ 'mcc': multiclass_matthews_correlation_coefficient,
+ 'bm': multiclass_bookmaker_informedness,
+ 'pt': multiclass_prevalence_threshold,
+ 'dor': multiclass_diagnostic_odds_ratio,
+ 'ji': multiclass_jaccard_index,
+ 'kappa': multiclass_cohens_kappa}
diff --git a/mlscorecheck/symbolic/_algebra.py b/mlscorecheck/symbolic/_algebra.py
index 62c5889..d9f3c73 100644
--- a/mlscorecheck/symbolic/_algebra.py
+++ b/mlscorecheck/symbolic/_algebra.py
@@ -1,4 +1,3 @@
-
"""
This module implements the joint interface to the algebraic systems
to be used.
@@ -6,13 +5,14 @@
import abc
-__all__ = ['Algebra']
+__all__ = ["Algebra"]
class Algebra(metaclass=abc.ABCMeta):
"""
The base class of the algebra abstractions
"""
+
@abc.abstractmethod
def __init__(self):
"""
diff --git a/mlscorecheck/symbolic/_availability.py b/mlscorecheck/symbolic/_availability.py
index 5646a3f..16b7f38 100644
--- a/mlscorecheck/symbolic/_availability.py
+++ b/mlscorecheck/symbolic/_availability.py
@@ -4,9 +4,7 @@
import importlib
-__all__ = ['symbolic_toolkits',
- 'get_symbolic_toolkit',
- 'check_importability']
+__all__ = ["symbolic_toolkits", "get_symbolic_toolkit", "check_importability"]
symbolic_toolkits = []
@@ -28,8 +26,8 @@ def check_importability(package: str):
return None
-symbolic_toolkits.append(check_importability('sympy'))
-symbolic_toolkits.append(check_importability('sage'))
+symbolic_toolkits.append(check_importability("sympy"))
+symbolic_toolkits.append(check_importability("sage"))
symbolic_toolkits = [package for package in symbolic_toolkits if package is not None]
diff --git a/mlscorecheck/symbolic/_sage_algebra.py b/mlscorecheck/symbolic/_sage_algebra.py
index 92fa7ab..0c794ce 100644
--- a/mlscorecheck/symbolic/_sage_algebra.py
+++ b/mlscorecheck/symbolic/_sage_algebra.py
@@ -5,26 +5,28 @@
so the sage-related codes are excluded from the coverage report.
"""
-__all__ = ['SageAlgebra']
+__all__ = ["SageAlgebra"]
import importlib
from ._algebra import Algebra
-class SageAlgebra(Algebra): # pragma: no cover
+
+class SageAlgebra(Algebra): # pragma: no cover
"""
The required algebra driven by sage
"""
- def __init__(self): # pragma: no cover
+
+ def __init__(self): # pragma: no cover
"""
Constructor of the algebra
"""
- Algebra.__init__(self) # pragma: no cover
+ Algebra.__init__(self) # pragma: no cover
- self.algebra = importlib.import_module("sage.all") # pragma: no cover
- self.sqrt = self.algebra.sqrt # pragma: no cover
+ self.algebra = importlib.import_module("sage.all") # pragma: no cover
+ self.sqrt = self.algebra.sqrt # pragma: no cover
- def create_symbol(self, name, **kwargs): # pragma: no cover
+ def create_symbol(self, name, **kwargs): # pragma: no cover
"""
Create a symbol in the algebra with the specified name and assumptions
@@ -35,25 +37,25 @@ def create_symbol(self, name, **kwargs): # pragma: no cover
Returns:
object: the symbol
"""
- var = self.algebra.var(name) # pragma: no cover
- if kwargs.get('nonnegative', False): # pragma: no cover
- self.algebra.assume(var >= 0) # pragma: no cover
- if kwargs.get('positive', False): # pragma: no cover
- self.algebra.assume(var > 0) # pragma: no cover
- if kwargs.get('negative', False): # pragma: no cover
- self.algebra.assume(var < 0) # pragma: no cover
- if kwargs.get('nonpositive', False): # pragma: no cover
- self.algebra.assume(var <= 0) # pragma: no cover
- if kwargs.get('real', False): # pragma: no cover
- self.algebra.assume(var, 'real') # pragma: no cover
- if kwargs.get('upper_bound', None) is not None: # pragma: no cover
- self.algebra.assume(var <= kwargs['upper_bound']) # pragma: no cover
- if kwargs.get('lower_bound', None) is not None: # pragma: no cover
- self.algebra.assume(var >= kwargs['lower_bound']) # pragma: no cover
-
- return var # pragma: no cover
-
- def num_denom(self, expression): # pragma: no cover
+ var = self.algebra.var(name) # pragma: no cover
+ if kwargs.get("nonnegative", False): # pragma: no cover
+ self.algebra.assume(var >= 0) # pragma: no cover
+ if kwargs.get("positive", False): # pragma: no cover
+ self.algebra.assume(var > 0) # pragma: no cover
+ if kwargs.get("negative", False): # pragma: no cover
+ self.algebra.assume(var < 0) # pragma: no cover
+ if kwargs.get("nonpositive", False): # pragma: no cover
+ self.algebra.assume(var <= 0) # pragma: no cover
+ if kwargs.get("real", False): # pragma: no cover
+ self.algebra.assume(var, "real") # pragma: no cover
+ if kwargs.get("upper_bound", None) is not None: # pragma: no cover
+ self.algebra.assume(var <= kwargs["upper_bound"]) # pragma: no cover
+ if kwargs.get("lower_bound", None) is not None: # pragma: no cover
+ self.algebra.assume(var >= kwargs["lower_bound"]) # pragma: no cover
+
+ return var # pragma: no cover
+
+ def num_denom(self, expression): # pragma: no cover
"""
Extract the numerator and denominator
@@ -63,9 +65,9 @@ def num_denom(self, expression): # pragma: no cover
Returns:
object, object: the numerator and denominator
"""
- return expression.numerator(), expression.denominator() # pragma: no cover
+ return expression.numerator(), expression.denominator() # pragma: no cover
- def simplify(self, expression): # pragma: no cover
+ def simplify(self, expression): # pragma: no cover
"""
Simplify the expression
@@ -75,9 +77,9 @@ def simplify(self, expression): # pragma: no cover
Returns:
object: the symplified expression
"""
- return self.algebra.factor(expression) # pragma: no cover
+ return self.algebra.factor(expression) # pragma: no cover
- def solve(self, equation, var, **kwargs): # pragma: no cover
+ def solve(self, equation, var, **kwargs): # pragma: no cover
"""
Solve an equation for a variable
@@ -89,14 +91,14 @@ def solve(self, equation, var, **kwargs): # pragma: no cover
Returns:
list(dict): the solutions
"""
- results = self.algebra.solve(equation, var, **kwargs) # pragma: no cover
- solutions = [] # pragma: no cover
- for sol in results: # pragma: no cover
- solution = {sol.lhs(): self.algebra.factor(sol.rhs())} # pragma: no cover
- solutions.append(solution) # pragma: no cover
- return solutions # pragma: no cover
+ results = self.algebra.solve(equation, var, **kwargs) # pragma: no cover
+ solutions = [] # pragma: no cover
+ for sol in results: # pragma: no cover
+ solution = {sol.lhs(): self.algebra.factor(sol.rhs())} # pragma: no cover
+ solutions.append(solution) # pragma: no cover
+ return solutions # pragma: no cover
- def subs(self, expression, subs_dict): # pragma: no cover
+ def subs(self, expression, subs_dict): # pragma: no cover
"""
Substitute a substitution into the expression
@@ -107,9 +109,9 @@ def subs(self, expression, subs_dict): # pragma: no cover
Returns:
object: the result of the substitution
"""
- return expression.subs(subs_dict) # pragma: no cover
+ return expression.subs(subs_dict) # pragma: no cover
- def args(self, expression) -> list: # pragma: no cover
+ def args(self, expression) -> list: # pragma: no cover
"""
The list of arguments
@@ -119,9 +121,9 @@ def args(self, expression) -> list: # pragma: no cover
Returns:
list: the list of arguments
"""
- return set(expression.args()) # pragma: no cover
+ return set(expression.args()) # pragma: no cover
- def is_trivial(self, expression) -> bool: # pragma: no cover
+ def is_trivial(self, expression) -> bool: # pragma: no cover
"""
Checks if the expression is trivial
@@ -131,9 +133,11 @@ def is_trivial(self, expression) -> bool: # pragma: no cover
Returns:
bool: True if the expression is trivial, False otherwise
"""
- return True if expression is None else expression.is_trivially_equal(1) # pragma: no cover
+ return (
+ True if expression is None else expression.is_trivially_equal(1)
+ ) # pragma: no cover
- def is_root(self, expression) -> bool: # pragma: no cover
+ def is_root(self, expression) -> bool: # pragma: no cover
"""
Checks if the expression is a root
@@ -143,13 +147,13 @@ def is_root(self, expression) -> bool: # pragma: no cover
Returns:
bool: True if the expression is a root, False otherwise
"""
- if self.is_power(expression): # pragma: no cover
- _, exponent = expression.operands() # pragma: no cover
- if 0 < exponent < 1: # pragma: no cover
- return True # pragma: no cover
- return False # pragma: no cover
+ if self.is_power(expression): # pragma: no cover
+ _, exponent = expression.operands() # pragma: no cover
+ if 0 < exponent < 1: # pragma: no cover
+ return True # pragma: no cover
+ return False # pragma: no cover
- def is_power(self, expression) -> bool: # pragma: no cover
+ def is_power(self, expression) -> bool: # pragma: no cover
"""
Checks whether the expression is a power
@@ -159,10 +163,12 @@ def is_power(self, expression) -> bool: # pragma: no cover
Returns:
bool: whether the expression is a power
"""
- return bool(hasattr(expression.operator(), '__qualname__')\
- and expression.operator().__qualname__ == 'pow') # pragma: no cover
+ return bool(
+ hasattr(expression.operator(), "__qualname__")
+ and expression.operator().__qualname__ == "pow"
+ ) # pragma: no cover
- def is_division(self, expression) -> bool: # pragma: no cover
+ def is_division(self, expression) -> bool: # pragma: no cover
"""
Checks whether the expression is a division
@@ -172,27 +178,29 @@ def is_division(self, expression) -> bool: # pragma: no cover
Returns:
bool: whether the expression is a division
"""
- if self.is_power(expression): # pragma: no cover
- operands = expression.operands() # pragma: no cover
- if operands[1] < 0: # pragma: no cover
- return True # pragma: no cover
-
- if hasattr(expression.operator(), '__qualname__') \
- and expression.operator().__qualname__ == 'mul_vararg': # pragma: no cover
- operands = expression.operands() # pragma: no cover
-
- if len(operands) == 2: # pragma: no cover
- if self.is_power(operands[1]): # pragma: no cover
- _, power = operands[1].operands() # pragma: no cover
- if power < 0: # pragma: no cover
- return True # pragma: no cover
- elif self.is_power(operands[0]): # pragma: no cover
- _, power = operands[0].operands() # pragma: no cover
- if power < 0: # pragma: no cover
- return True # pragma: no cover
- return False # pragma: no cover
-
- def operands(self, expression) -> list: # pragma: no cover
+ if self.is_power(expression): # pragma: no cover
+ operands = expression.operands() # pragma: no cover
+ if operands[1] < 0: # pragma: no cover
+ return True # pragma: no cover
+
+ if (
+ hasattr(expression.operator(), "__qualname__")
+ and expression.operator().__qualname__ == "mul_vararg"
+ ): # pragma: no cover
+ operands = expression.operands() # pragma: no cover
+
+ if len(operands) == 2: # pragma: no cover
+ if self.is_power(operands[1]): # pragma: no cover
+ _, power = operands[1].operands() # pragma: no cover
+ if power < 0: # pragma: no cover
+ return True # pragma: no cover
+ elif self.is_power(operands[0]): # pragma: no cover
+ _, power = operands[0].operands() # pragma: no cover
+ if power < 0: # pragma: no cover
+ return True # pragma: no cover
+ return False # pragma: no cover
+
+ def operands(self, expression) -> list: # pragma: no cover
"""
Returns the list of operands
@@ -202,9 +210,9 @@ def operands(self, expression) -> list: # pragma: no cover
Returns:
list: the operands
"""
- return tuple(expression.operands()) # pragma: no cover
+ return tuple(expression.operands()) # pragma: no cover
- def free_symbols(self, expression) -> list: # pragma: no cover
+ def free_symbols(self, expression) -> list: # pragma: no cover
"""
Get all free symbols in an expression
@@ -214,4 +222,6 @@ def free_symbols(self, expression) -> list: # pragma: no cover
Returns:
list: the list of free symbols
"""
- return [str(var) for var in list(expression.free_variables())] # pragma: no cover
+ return [
+ str(var) for var in list(expression.free_variables())
+ ] # pragma: no cover
diff --git a/mlscorecheck/symbolic/_score_objects.py b/mlscorecheck/symbolic/_score_objects.py
index 08c1df1..ac92d4a 100644
--- a/mlscorecheck/symbolic/_score_objects.py
+++ b/mlscorecheck/symbolic/_score_objects.py
@@ -14,53 +14,59 @@
from ._symbols import Symbols
-__all__ = ['Score',
- 'PositiveLikelihoodRatio',
- 'MatthewsCorrelationCoefficient',
- 'Accuracy',
- 'ErrorRate',
- 'Sensitivity',
- 'FalseNegativeRate',
- 'FalsePositiveRate',
- 'Specificity',
- 'PositivePredictiveValue',
- 'NegativePredictiveValue',
- 'FalseDiscoveryRate',
- 'FalseOmissionRate',
- 'FBetaPositive',
- 'F1Positive',
- 'FBetaNegative',
- 'F1Negative',
- 'UnifiedPerformanceMeasure',
- 'GeometricMean',
- 'FowlkesMallowsIndex',
- 'Markedness',
- 'PositiveLikelihoodRatio',
- 'NegativeLikelihoodRatio',
- 'Informedness',
- 'PrevalenceThreshold',
- 'DiagnosticOddsRatio',
- 'JaccardIndex',
- 'BalancedAccuracy',
- 'CohensKappa',
- 'get_base_objects',
- 'get_all_objects',
- 'get_objects_without_complements']
+__all__ = [
+ "Score",
+ "PositiveLikelihoodRatio",
+ "MatthewsCorrelationCoefficient",
+ "Accuracy",
+ "ErrorRate",
+ "Sensitivity",
+ "FalseNegativeRate",
+ "FalsePositiveRate",
+ "Specificity",
+ "PositivePredictiveValue",
+ "NegativePredictiveValue",
+ "FalseDiscoveryRate",
+ "FalseOmissionRate",
+ "FBetaPositive",
+ "F1Positive",
+ "FBetaNegative",
+ "F1Negative",
+ "UnifiedPerformanceMeasure",
+ "GeometricMean",
+ "FowlkesMallowsIndex",
+ "Markedness",
+ "PositiveLikelihoodRatio",
+ "NegativeLikelihoodRatio",
+ "Informedness",
+ "PrevalenceThreshold",
+ "DiagnosticOddsRatio",
+ "JaccardIndex",
+ "BalancedAccuracy",
+ "CohensKappa",
+ "get_base_objects",
+ "get_all_objects",
+ "get_objects_without_complements",
+]
scores = score_specifications
functions = score_functions_standardized_with_complements
-class Score: # pylint: disable=too-many-instance-attributes
+
+class Score: # pylint: disable=too-many-instance-attributes
"""
The Score base class
"""
- def __init__(self,
- symbols: Symbols,
- descriptor: dict,
- *,
- function,
- expression: str = None,
- equation: str = None):
+
+ def __init__(
+ self,
+ symbols: Symbols,
+ descriptor: dict,
+ *,
+ function,
+ expression: str = None,
+ equation: str = None
+ ):
"""
Constructor of the base class
@@ -74,15 +80,17 @@ def __init__(self,
"""
self.descriptor = descriptor
- self.abbreviation = descriptor['abbreviation']
- self.name = descriptor['name']
- self.nans = descriptor.get('nans')
- self.synonyms = descriptor.get('synonyms')
- self.complement = descriptor.get('complement')
- self.args = descriptor.get('args_standardized')
- self.range = (descriptor.get('lower_bound', -np.inf),
- descriptor.get('upper_bound', np.inf))
- self.sqrt = descriptor.get('sqrt', False)
+ self.abbreviation = descriptor["abbreviation"]
+ self.name = descriptor["name"]
+ self.nans = descriptor.get("nans")
+ self.synonyms = descriptor.get("synonyms")
+ self.complement = descriptor.get("complement")
+ self.args = descriptor.get("args_standardized")
+ self.range = (
+ descriptor.get("lower_bound", -np.inf),
+ descriptor.get("upper_bound", np.inf),
+ )
+ self.sqrt = descriptor.get("sqrt", False)
# setting the base kit of symbols
@@ -91,14 +99,16 @@ def __init__(self,
# setting the symbol
kwargs = {}
if self.range[0] > -np.inf:
- kwargs['lower_bound'] = self.range[0]
+ kwargs["lower_bound"] = self.range[0]
if self.range[1] < np.inf:
- kwargs['upper_bound'] = self.range[1]
- self.symbol = self.symbols.algebra.create_symbol(self.abbreviation, real=True, **kwargs)
+ kwargs["upper_bound"] = self.range[1]
+ self.symbol = self.symbols.algebra.create_symbol(
+ self.abbreviation, real=True, **kwargs
+ )
# setting the score function
if isinstance(function, str):
- module = importlib.import_module('mlscorecheck.scores')
+ module = importlib.import_module("mlscorecheck.scores")
self.function = getattr(module, function)
else:
self.function = function
@@ -107,7 +117,7 @@ def __init__(self,
arg_symbols = {arg: getattr(symbols, arg) for arg in self.args}
if self.sqrt:
- arg_symbols['sqrt'] = symbols.sqrt
+ arg_symbols["sqrt"] = symbols.sqrt
# setting the expression
if expression is not None:
@@ -124,7 +134,7 @@ def __init__(self,
# setting the polynomial equation
subs = {**{self.abbreviation: self.symbol}, **arg_symbols}
- self.equation_polynomial = safe_eval(descriptor['polynomial_equation'], subs)
+ self.equation_polynomial = safe_eval(descriptor["polynomial_equation"], subs)
def get_algebra(self):
"""
@@ -141,16 +151,18 @@ def to_dict(self):
dict: the dictionary representation
"""
return {
- 'descriptor': self.descriptor,
- 'expression': str(self.expression),
- 'equation': str(self.equation),
- 'function': self.function.__name__
+ "descriptor": self.descriptor,
+ "expression": str(self.expression),
+ "equation": str(self.equation),
+ "function": self.function.__name__,
}
+
class Accuracy(Score):
"""
The accuracy score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -158,15 +170,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['acc'],
- function=functions['acc'])
+ Score.__init__(self, symbols, scores["acc"], function=functions["acc"])
+
class ErrorRate(Score):
"""
The error rate score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -174,15 +185,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['err'],
- function=functions['err'])
+ Score.__init__(self, symbols, scores["err"], function=functions["err"])
+
class Sensitivity(Score):
"""
The sensitivity score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -190,15 +200,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['sens'],
- function=functions['sens'])
+ Score.__init__(self, symbols, scores["sens"], function=functions["sens"])
+
class FalseNegativeRate(Score):
"""
The false negative rate
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -206,15 +215,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['fnr'],
- function=functions['fnr'])
+ Score.__init__(self, symbols, scores["fnr"], function=functions["fnr"])
+
class FalsePositiveRate(Score):
"""
The false positive rate
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -222,15 +230,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['fpr'],
- function=functions['fpr'])
+ Score.__init__(self, symbols, scores["fpr"], function=functions["fpr"])
+
class Specificity(Score):
"""
The specificity score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -238,15 +245,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['spec'],
- function=functions['spec'])
+ Score.__init__(self, symbols, scores["spec"], function=functions["spec"])
+
class PositivePredictiveValue(Score):
"""
The positive predictive value
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -254,15 +260,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['ppv'],
- function=functions['ppv'])
+ Score.__init__(self, symbols, scores["ppv"], function=functions["ppv"])
+
class FalseDiscoveryRate(Score):
"""
The false discovery rate
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -270,15 +275,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['fdr'],
- function=functions['fdr'])
+ Score.__init__(self, symbols, scores["fdr"], function=functions["fdr"])
+
class FalseOmissionRate(Score):
"""
The false omission rate
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -286,15 +290,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['for_'],
- function=functions['for_'])
+ Score.__init__(self, symbols, scores["for_"], function=functions["for_"])
+
class NegativePredictiveValue(Score):
"""
The negative predictive value
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -302,15 +305,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['npv'],
- function=functions['npv'])
+ Score.__init__(self, symbols, scores["npv"], function=functions["npv"])
+
class FBetaPositive(Score):
"""
The f-beta plus score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -318,15 +320,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['fbp'],
- function=functions['fbp'])
+ Score.__init__(self, symbols, scores["fbp"], function=functions["fbp"])
+
class F1Positive(Score):
"""
The f1-plus score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -334,15 +335,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['f1p'],
- function=functions['f1p'])
+ Score.__init__(self, symbols, scores["f1p"], function=functions["f1p"])
+
class FBetaNegative(Score):
"""
The f-beta minus score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -350,15 +350,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['fbn'],
- function=functions['fbn'])
+ Score.__init__(self, symbols, scores["fbn"], function=functions["fbn"])
+
class F1Negative(Score):
"""
The f1-minus score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -366,15 +365,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['f1n'],
- function=functions['f1n'])
+ Score.__init__(self, symbols, scores["f1n"], function=functions["f1n"])
+
class GeometricMean(Score):
"""
The geometric mean score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -382,15 +380,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['gm'],
- function=functions['gm'])
+ Score.__init__(self, symbols, scores["gm"], function=functions["gm"])
+
class FowlkesMallowsIndex(Score):
"""
The Fowlkes-Mallows-index
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -398,15 +395,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['fm'],
- function=functions['fm'])
+ Score.__init__(self, symbols, scores["fm"], function=functions["fm"])
+
class UnifiedPerformanceMeasure(Score):
"""
The unified performance measure score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -414,16 +410,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['upm'],
- function=functions['upm'])
+ Score.__init__(self, symbols, scores["upm"], function=functions["upm"])
class Markedness(Score):
"""
The markedness score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -431,15 +425,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['mk'],
- function=functions['mk'])
+ Score.__init__(self, symbols, scores["mk"], function=functions["mk"])
+
class PositiveLikelihoodRatio(Score):
"""
The positive likelihood ratio score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -447,15 +440,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['lrp'],
- function=functions['lrp'])
+ Score.__init__(self, symbols, scores["lrp"], function=functions["lrp"])
+
class NegativeLikelihoodRatio(Score):
"""
The negative likelihood ratio score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -463,15 +455,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['lrn'],
- function=functions['lrn'])
+ Score.__init__(self, symbols, scores["lrn"], function=functions["lrn"])
+
class Informedness(Score):
"""
The informedness score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -479,15 +470,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['bm'],
- function=functions['bm'])
+ Score.__init__(self, symbols, scores["bm"], function=functions["bm"])
+
class PrevalenceThreshold(Score):
"""
The prevalence threshold
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -495,15 +485,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['pt'],
- function=functions['pt'])
+ Score.__init__(self, symbols, scores["pt"], function=functions["pt"])
+
class DiagnosticOddsRatio(Score):
"""
The diagnostic odds ratio
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -511,15 +500,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['dor'],
- function=functions['dor'])
+ Score.__init__(self, symbols, scores["dor"], function=functions["dor"])
+
class JaccardIndex(Score):
"""
The Jaccard index
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -527,15 +515,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['ji'],
- function=functions['ji'])
+ Score.__init__(self, symbols, scores["ji"], function=functions["ji"])
+
class BalancedAccuracy(Score):
"""
The balanced accuracy score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -543,15 +530,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['bacc'],
- function=functions['bacc'])
+ Score.__init__(self, symbols, scores["bacc"], function=functions["bacc"])
+
class CohensKappa(Score):
"""
Cohen's kappa
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -559,15 +545,14 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['kappa'],
- function=functions['kappa'])
+ Score.__init__(self, symbols, scores["kappa"], function=functions["kappa"])
+
class MatthewsCorrelationCoefficient(Score):
"""
The MatthewsCorrelationCoefficient score
"""
+
def __init__(self, symbols):
"""
The constructor of the score
@@ -575,12 +560,10 @@ def __init__(self, symbols):
Args:
symbols (Symbols): the algebraic symbols to be used
"""
- Score.__init__(self,
- symbols,
- scores['mcc'],
- function=functions['mcc'])
+ Score.__init__(self, symbols, scores["mcc"], function=functions["mcc"])
-def get_base_objects(algebraic_system: str = 'sympy') -> dict:
+
+def get_base_objects(algebraic_system: str = "sympy") -> dict:
"""
Returns the dict of basic score objects
@@ -591,19 +574,25 @@ def get_base_objects(algebraic_system: str = 'sympy') -> dict:
dict: the dictionary of basic score objects
"""
symbols = Symbols(algebraic_system=algebraic_system)
- score_objects = [cls(symbols=symbols) for cls in [Accuracy,
- Sensitivity,
- Specificity,
- PositivePredictiveValue,
- NegativePredictiveValue,
- BalancedAccuracy,
- F1Positive,
- FowlkesMallowsIndex]]
+ score_objects = [
+ cls(symbols=symbols)
+ for cls in [
+ Accuracy,
+ Sensitivity,
+ Specificity,
+ PositivePredictiveValue,
+ NegativePredictiveValue,
+ BalancedAccuracy,
+ F1Positive,
+ FowlkesMallowsIndex,
+ ]
+ ]
score_objects = {obj.abbreviation: obj for obj in score_objects}
return score_objects
-def get_all_objects(algebraic_system: str = 'sympy') -> dict:
+
+def get_all_objects(algebraic_system: str = "sympy") -> dict:
"""
Returns the dict of all score objects
@@ -619,7 +608,8 @@ def get_all_objects(algebraic_system: str = 'sympy') -> dict:
return score_objects
-def get_objects_without_complements(algebraic_system: str = 'sympy') -> dict:
+
+def get_objects_without_complements(algebraic_system: str = "sympy") -> dict:
"""
Returns the dict of basic score objects without complements
@@ -631,7 +621,10 @@ def get_objects_without_complements(algebraic_system: str = 'sympy') -> dict:
"""
symbols = Symbols(algebraic_system=algebraic_system)
score_objects = [cls(symbols=symbols) for cls in Score.__subclasses__()]
- score_objects = {obj.abbreviation: obj for obj in score_objects
- if obj.abbreviation in score_functions_without_complements}
+ score_objects = {
+ obj.abbreviation: obj
+ for obj in score_objects
+ if obj.abbreviation in score_functions_without_complements
+ }
return score_objects
diff --git a/mlscorecheck/symbolic/_solver.py b/mlscorecheck/symbolic/_solver.py
index 3204a35..c6b9f8c 100644
--- a/mlscorecheck/symbolic/_solver.py
+++ b/mlscorecheck/symbolic/_solver.py
@@ -7,17 +7,18 @@
from ._algebra import Algebra
from ._score_objects import Score
-__all__ = ['ProblemSolver',
- 'collect_denominators_and_bases',
- '_collect_denominators_and_bases',
- 'solve_order',
- 'check_recurrent_solution']
-
-def _collect_denominators_and_bases(expression,
- denoms: list,
- bases: list,
- algebra: Algebra,
- depth: int):
+__all__ = [
+ "ProblemSolver",
+ "collect_denominators_and_bases",
+ "_collect_denominators_and_bases",
+ "solve_order",
+ "check_recurrent_solution",
+]
+
+
+def _collect_denominators_and_bases(
+ expression, denoms: list, bases: list, algebra: Algebra, depth: int
+):
"""
Recursive core of collecting all denominators and bases
@@ -35,19 +36,22 @@ def _collect_denominators_and_bases(expression,
if not algebra.is_trivial(denom):
denoms.append((denom, depth))
- _collect_denominators_and_bases(denom, denoms, bases, algebra, depth+1)
- _collect_denominators_and_bases(num, denoms, bases, algebra, depth+1)
+ _collect_denominators_and_bases(denom, denoms, bases, algebra, depth + 1)
+ _collect_denominators_and_bases(num, denoms, bases, algebra, depth + 1)
elif algebra.is_root(expression):
# fractional exponents are already checked here
base, exponent = algebra.operands(expression)
bases.append((base, float(exponent), depth))
for operand in algebra.operands(base):
- _collect_denominators_and_bases(operand, denoms, bases, algebra, depth+1)
+ _collect_denominators_and_bases(operand, denoms, bases, algebra, depth + 1)
else:
for operand in algebra.operands(expression):
if not algebra.is_trivial(operand):
- _collect_denominators_and_bases(operand, denoms, bases, algebra, depth+1)
+ _collect_denominators_and_bases(
+ operand, denoms, bases, algebra, depth + 1
+ )
+
def collect_denominators_and_bases(expression, algebra: Algebra) -> (list, list):
"""
@@ -86,6 +90,7 @@ def collect_denominators_and_bases(expression, algebra: Algebra) -> (list, list)
return denoms, bases
+
def solve_order(score0: Score, score1: Score):
"""
Determining the order the equations should be solved, based on using the shortest solution
@@ -105,11 +110,13 @@ def solve_order(score0: Score, score1: Score):
sols = {}
lens = {}
- for figure in ['tp', 'tn']:
+ for figure in ["tp", "tn"]:
for score_idx, score in enumerate([score0, score1]):
key = (figure, score_idx)
if figure in algebra.free_symbols(score.equation_polynomial):
- sols[key] = algebra.solve(score.equation_polynomial, getattr(symbols, figure))
+ sols[key] = algebra.solve(
+ score.equation_polynomial, getattr(symbols, figure)
+ )
lens[key] = len(str(sols[key][0][getattr(symbols, figure)]))
min_length = 1e10
@@ -120,26 +127,31 @@ def solve_order(score0: Score, score1: Score):
min_length = value
min_variable = key
- if ('tn', 0) not in sols or ('tp', 1) not in sols:
- return score0.equation_polynomial, score1.equation_polynomial, 'tp', 'tn'
- if ('tp', 0) not in sols or ('tn', 1) not in sols:
- return score0.equation_polynomial, score1.equation_polynomial, 'tn', 'tp'
+ if ("tn", 0) not in sols or ("tp", 1) not in sols:
+ return score0.equation_polynomial, score1.equation_polynomial, "tp", "tn"
+ if ("tp", 0) not in sols or ("tn", 1) not in sols:
+ return score0.equation_polynomial, score1.equation_polynomial, "tn", "tp"
first_variable = None
second_variable = None
- if min_variable[0] == 'tp':
+ if min_variable[0] == "tp":
first_variable = symbols.tp
second_variable = symbols.tn
if min_variable[1] == 1:
score0, score1 = score1, score0
- elif min_variable[0] == 'tn':
+ elif min_variable[0] == "tn":
first_variable = symbols.tn
second_variable = symbols.tp
if min_variable[1] == 1:
score0, score1 = score1, score0
- return (score0.equation_polynomial, score1.equation_polynomial,
- str(first_variable), str(second_variable))
+ return (
+ score0.equation_polynomial,
+ score1.equation_polynomial,
+ str(first_variable),
+ str(second_variable),
+ )
+
def check_recurrent_solution(symbol: str, symbols: list) -> bool:
"""
@@ -150,15 +162,17 @@ def check_recurrent_solution(symbol: str, symbols: list) -> bool:
symbols (list): the free symbols
"""
if symbol in symbols:
- logger.warning('recurrent %s %s', symbol, str(symbols))
+ logger.warning("recurrent %s %s", symbol, str(symbols))
return symbol in symbols
+
class ProblemSolver:
"""
The problem solver object, used to solve the individual problems with
the aid of computer algebra systems
"""
+
def __init__(self, score0: Score, score1: Score):
"""
The constructor of the object
@@ -187,10 +201,10 @@ def corner_case_solution(self, solution):
"""
var0, var1 = list(solution.keys())
flag = False
- if str(solution[var0]['expression']) in {'0', 'n', 'p'}:
+ if str(solution[var0]["expression"]) in {"0", "n", "p"}:
# triggered in the ppv-fm case
flag = True
- if str(solution[var1]['expression']) in {'0', 'n', 'p'}:
+ if str(solution[var1]["expression"]) in {"0", "n", "p"}:
# triggered in the fm-ppv case
flag = True
@@ -206,7 +220,7 @@ def solve(self, **kwargs):
Returns:
self: the solver object
"""
- logger.info('solving %s %s', self.score0.abbreviation, self.score1.abbreviation)
+ logger.info("solving %s %s", self.score0.abbreviation, self.score1.abbreviation)
self.solutions = []
@@ -222,28 +236,38 @@ def solve(self, **kwargs):
# iterating through all solutions
# substitution into the other equation if needed
- equation1_tmp = (algebra.subs(equation1, v0_sol)
- if sym0 in algebra.args(equation1)
- else equation1)
+ equation1_tmp = (
+ algebra.subs(equation1, v0_sol)
+ if sym0 in algebra.args(equation1)
+ else equation1
+ )
# solving for sym1
v1_sols = algebra.solve(equation1_tmp, sym1, **kwargs)
- logger.info('solved')
+ logger.info("solved")
# assembling all solutions
for v1_sol in v1_sols:
- v0_final = (algebra.subs(v0_sol[sym0], v1_sol)
- if sym1 in algebra.args(v0_sol[sym0])
- else v0_sol[sym0])
-
- sol = {var0: {'expression': algebra.simplify(v0_final),
- 'symbols': algebra.free_symbols(v0_final)},
- var1: {'expression': algebra.simplify(v1_sol[sym1]),
- 'symbols': algebra.free_symbols(v1_sol[sym1])}}
-
- check_recurrent_solution(str(var0), sol[var0]['symbols'])
- check_recurrent_solution(str(var1), sol[var1]['symbols'])
+ v0_final = (
+ algebra.subs(v0_sol[sym0], v1_sol)
+ if sym1 in algebra.args(v0_sol[sym0])
+ else v0_sol[sym0]
+ )
+
+ sol = {
+ var0: {
+ "expression": algebra.simplify(v0_final),
+ "symbols": algebra.free_symbols(v0_final),
+ },
+ var1: {
+ "expression": algebra.simplify(v1_sol[sym1]),
+ "symbols": algebra.free_symbols(v1_sol[sym1]),
+ },
+ }
+
+ check_recurrent_solution(str(var0), sol[var0]["symbols"])
+ check_recurrent_solution(str(var1), sol[var1]["symbols"])
if not self.corner_case_solution(sol):
self.solutions.append(sol)
@@ -265,36 +289,52 @@ def edge_cases(self):
bases_sol = {}
for _, sol in solution.items():
- denoms, bases = collect_denominators_and_bases(algebra.simplify(sol['expression']),
- algebra)
+ denoms, bases = collect_denominators_and_bases(
+ algebra.simplify(sol["expression"]), algebra
+ )
denoms = list(denoms)
bases = list(bases)
for denom, depth in denoms:
denom_str = str(denom)
- if ((denom_str not in denoms_sol) or (denoms_sol[denom_str][1] < depth)):
+ if (denom_str not in denoms_sol) or (
+ denoms_sol[denom_str][1] < depth
+ ):
denoms_sol[denom_str] = (denom, depth)
for base, exponent, depth in bases:
base_str = str((base, exponent))
- if ((base_str not in bases_sol) or (bases_sol[base_str][2] < depth)):
+ if (base_str not in bases_sol) or (bases_sol[base_str][2] < depth):
bases_sol[base_str] = (base, exponent, depth)
- denoms_sol = [{'expression': str(denom[0]),
- 'symbols': algebra.free_symbols(denom[0]),
- 'depth': denom[1],
- 'mode': 'non-zero'} for denom in list(denoms_sol.values())]
- bases_sol = [{'expression': str(base[0]),
- 'exponent': base[1],
- 'symbols': algebra.free_symbols(base[0]),
- 'depth': base[2],
- 'mode': 'non-negative'} for base in list(bases_sol.values())]
+ denoms_sol = [
+ {
+ "expression": str(denom[0]),
+ "symbols": algebra.free_symbols(denom[0]),
+ "depth": denom[1],
+ "mode": "non-zero",
+ }
+ for denom in list(denoms_sol.values())
+ ]
+ bases_sol = [
+ {
+ "expression": str(base[0]),
+ "exponent": base[1],
+ "symbols": algebra.free_symbols(base[0]),
+ "depth": base[2],
+ "mode": "non-negative",
+ }
+ for base in list(bases_sol.values())
+ ]
self.denoms.append(denoms_sol)
self.bases.append(bases_sol)
self.conditions.append(denoms_sol + bases_sol)
- tmp = {str(key): {key2: str(value2)
- if key2 == 'expression'
- else value2 for key2, value2 in value.items()}
- for key, value in solution.items()}
+ tmp = {
+ str(key): {
+ key2: str(value2) if key2 == "expression" else value2
+ for key2, value2 in value.items()
+ }
+ for key, value in solution.items()
+ }
self.str_solutions.append(tmp)
def get_solution(self) -> Solutions:
@@ -307,10 +347,11 @@ def get_solution(self) -> Solutions:
results = []
for solution, conditions in zip(self.str_solutions, self.conditions):
- results.append({'solution': solution,
- 'conditions': conditions})
+ results.append({"solution": solution, "conditions": conditions})
- solution = Solutions(scores=[self.score0.abbreviation, self.score1.abbreviation],
- solutions=results)
+ solution = Solutions(
+ scores=[self.score0.abbreviation, self.score1.abbreviation],
+ solutions=results,
+ )
return solution
diff --git a/mlscorecheck/symbolic/_symbols.py b/mlscorecheck/symbolic/_symbols.py
index 3a21f2d..2099175 100644
--- a/mlscorecheck/symbolic/_symbols.py
+++ b/mlscorecheck/symbolic/_symbols.py
@@ -2,7 +2,7 @@
This module implements the wrapper for the symbols
"""
-__all__ = ['Symbols']
+__all__ = ["Symbols"]
from ._sympy_algebra import SympyAlgebra
@@ -13,6 +13,7 @@ class Symbols: # pylint: disable=too-many-instance-attributes
"""
A symbols class representing the basic symbols to be used
"""
+
def __init__(self, algebraic_system):
"""
The constructor of the object
@@ -21,17 +22,21 @@ def __init__(self, algebraic_system):
algebraic_system ('sympy'/'sage'): the algebraic system to be used
"""
self.algebraic_system = algebraic_system
- if algebraic_system == 'sympy':
+ if algebraic_system == "sympy":
self.algebra = SympyAlgebra()
- elif algebraic_system == 'sage': # pragma: no cover
- self.algebra = SageAlgebra() # pragma: no cover
-
- self.tp = self.algebra.create_symbol('tp', nonnegative=True, real=True)
- self.tn = self.algebra.create_symbol('tn', nonnegative=True, real=True)
- self.p = self.algebra.create_symbol('p', positive=True, real=True)
- self.n = self.algebra.create_symbol('n', positive=True, real=True)
- self.beta_positive = self.algebra.create_symbol('beta_positive', positive=True, real=True)
- self.beta_negative = self.algebra.create_symbol('beta_negative', positive=True, real=True)
+ elif algebraic_system == "sage": # pragma: no cover
+ self.algebra = SageAlgebra() # pragma: no cover
+
+ self.tp = self.algebra.create_symbol("tp", nonnegative=True, real=True)
+ self.tn = self.algebra.create_symbol("tn", nonnegative=True, real=True)
+ self.p = self.algebra.create_symbol("p", positive=True, real=True)
+ self.n = self.algebra.create_symbol("n", positive=True, real=True)
+ self.beta_positive = self.algebra.create_symbol(
+ "beta_positive", positive=True, real=True
+ )
+ self.beta_negative = self.algebra.create_symbol(
+ "beta_negative", positive=True, real=True
+ )
self.sqrt = self.algebra.sqrt
def get_algebra(self):
@@ -50,10 +55,12 @@ def to_dict(self) -> dict:
Returns:
dict: the dictionary representation
"""
- return {'tp': self.tp,
- 'tn': self.tn,
- 'p': self.p,
- 'n': self.n,
- 'beta_positive': self.beta_positive,
- 'beta_negative': self.beta_negative,
- 'sqrt': self.sqrt}
+ return {
+ "tp": self.tp,
+ "tn": self.tn,
+ "p": self.p,
+ "n": self.n,
+ "beta_positive": self.beta_positive,
+ "beta_negative": self.beta_negative,
+ "sqrt": self.sqrt,
+ }
diff --git a/mlscorecheck/symbolic/_sympy_algebra.py b/mlscorecheck/symbolic/_sympy_algebra.py
index e7aeb54..1661b33 100644
--- a/mlscorecheck/symbolic/_sympy_algebra.py
+++ b/mlscorecheck/symbolic/_sympy_algebra.py
@@ -2,16 +2,18 @@
This module implements the wrapper for the sympy algebra
"""
-__all__ = ['SympyAlgebra']
+__all__ = ["SympyAlgebra"]
import importlib
from ._algebra import Algebra
+
class SympyAlgebra(Algebra):
"""
The required algebra driven by sympy
"""
+
def __init__(self):
"""
Constructor of the algebra
@@ -32,10 +34,10 @@ def create_symbol(self, name: str, **kwargs):
Returns:
object: the symbol
"""
- if 'upper_bound' in kwargs:
- del kwargs['upper_bound']
- if 'lower_bound' in kwargs:
- del kwargs['lower_bound']
+ if "upper_bound" in kwargs:
+ del kwargs["upper_bound"]
+ if "lower_bound" in kwargs:
+ del kwargs["lower_bound"]
return self.algebra.Symbol(name, **kwargs)
def num_denom(self, expression):
diff --git a/notebooks/development/038_multiclass.ipynb b/notebooks/development/038_multiclass.ipynb
new file mode 100644
index 0000000..c371ca1
--- /dev/null
+++ b/notebooks/development/038_multiclass.ipynb
@@ -0,0 +1,606 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "from sklearn.model_selection import StratifiedKFold"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "y = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "classes = {0: 3, 1: 5, 2: 4, 3: 10}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def multiclass_stratified_folds(classes, n_folds):\n",
+ " folds = []\n",
+ " y = np.hstack([np.repeat(key, value) for key, value in classes.items()])\n",
+ " for _, test in StratifiedKFold(n_splits=n_folds).split(y.reshape(-1, 1), y, y):\n",
+ " folds.append({idx: count for idx, count in enumerate(np.bincount(y[test]))})\n",
+ "\n",
+ " return folds"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[{0: 1, 1: 2, 2: 1, 3: 4}, {0: 1, 1: 2, 2: 1, 3: 3}, {0: 1, 1: 1, 2: 2, 3: 3}]"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiclass_stratified_folds(classes, 3)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def transform_multiclass_fold_to_binary(fold):\n",
+ " n_total = sum(fold.values())\n",
+ " return [{'p': value, 'n': n_total - value} for value in fold.values()]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[{'p': 1, 'n': 7}, {'p': 2, 'n': 6}, {'p': 1, 'n': 7}, {'p': 4, 'n': 4}]"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "transform_multiclass_fold_to_binary(multiclass_stratified_folds(classes, 3)[0])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def sample_multiclass_fold(fold, random_state=None):\n",
+ " if not isinstance(random_state, np.random.RandomState):\n",
+ " random_state = np.random.RandomState(random_state)\n",
+ "\n",
+ " sample = np.zeros(shape=(len(fold), len(fold)), dtype=int)\n",
+ "\n",
+ " for class_idx, count in fold.items():\n",
+ " sample[class_idx, :] = random_state.multinomial(count,\n",
+ " np.ones(len(fold))/len(fold),\n",
+ " size=1)[0]\n",
+ "\n",
+ " sums = np.sum(sample, axis=1)\n",
+ " values = np.array(list(fold.values()))\n",
+ "\n",
+ " assert np.all(sums == values)\n",
+ "\n",
+ " return sample\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "fold = multiclass_stratified_folds(classes, 3)[0]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "confusion_fold = sample_multiclass_fold(fold, random_state=42)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.core import safe_call"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 58,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def multiclass_score_macro(confusion_matrix, score_function):\n",
+ " counts = np.sum(confusion_matrix, axis=1)\n",
+ " n_total = np.sum(counts)\n",
+ "\n",
+ " scores = [safe_call(score_function, {'p': counts[idx],\n",
+ " 'n': n_total - counts[idx],\n",
+ " 'tp': confusion_matrix[idx, idx],\n",
+ " 'tn': np.sum(confusion_matrix[0:idx, 0:idx]) + np.sum(confusion_matrix[idx+1:, idx+1:]) +\n",
+ " np.sum(confusion_matrix[:idx, idx+1:]) + np.sum(confusion_matrix[idx+1:, :idx])})\n",
+ " for idx, count in enumerate(counts)]\n",
+ "\n",
+ " return np.mean(scores)\n",
+ "\n",
+ "def multiclass_score_micro(confusion_matrix, score_function):\n",
+ " counts = np.sum(confusion_matrix, axis=1)\n",
+ " n_total = np.sum(counts)\n",
+ "\n",
+ " params = {'tp': 0,\n",
+ " 'tn': 0,\n",
+ " 'p': 0,\n",
+ " 'n': 0}\n",
+ "\n",
+ " for idx, count in enumerate(counts):\n",
+ " params['p'] += counts[idx]\n",
+ " params['n'] += n_total - counts[idx]\n",
+ " params['tp'] += confusion_matrix[idx, idx]\n",
+ " params['tn'] += np.sum(confusion_matrix[0:idx, 0:idx]) + np.sum(confusion_matrix[idx+1:, idx+1:]) + \\\n",
+ " np.sum(confusion_matrix[:idx, idx+1:]) + np.sum(confusion_matrix[idx+1:, :idx])\n",
+ "\n",
+ " return safe_call(score_function, params)\n",
+ "\n",
+ "def multiclass_score_weighted(confusion_matrix, score_function):\n",
+ " counts = np.sum(confusion_matrix, axis=1)\n",
+ " n_total = np.sum(counts)\n",
+ "\n",
+ " scores = [safe_call(score_function, {'p': counts[idx],\n",
+ " 'n': n_total - counts[idx],\n",
+ " 'tp': confusion_matrix[idx, idx],\n",
+ " 'tn': np.sum(confusion_matrix[0:idx, 0:idx]) + np.sum(confusion_matrix[idx+1:, idx+1:]) +\n",
+ " np.sum(confusion_matrix[:idx, idx+1:]) + np.sum(confusion_matrix[idx+1:, :idx])})\\\n",
+ " * count / n_total\n",
+ " for idx, count in enumerate(counts)]\n",
+ "\n",
+ " return np.sum(scores)\n",
+ "\n",
+ "def multiclass_scores(confusion_matrix, score_function, average):\n",
+ " if average == 'micro':\n",
+ " return multiclass_score_micro(confusion_matrix, score_function)\n",
+ " if average == 'macro':\n",
+ " return multiclass_score_macro(confusion_matrix, score_function)\n",
+ " if average == 'weighted':\n",
+ " return multiclass_score_weighted(confusion_matrix, score_function)\n",
+ "\n",
+ " raise ValueError(f'averaging {average} is not supported')\n",
+ "\n",
+ "def multiclass_accuracy(confusion_matrix, average):\n",
+ " return multiclass_score(confusion_matrix, accuracy_standardized, average)\n",
+ "\n",
+ "def multiclass_sensitivity(confusion_matrix, average):\n",
+ " return multiclass_score(confusion_matrix, sensitivity_standardized, average)\n",
+ "\n",
+ "def multiclass_specificity(confusion_matrix, average):\n",
+ " return multiclass_score(confusion_matrix, specificity_standardized, average)\n",
+ "\n",
+ "def multiclass_balanced_accuracy(confusion_matrix, average):\n",
+ " return multiclass_score(confusion_matrix, balanced_accuracy_standardized, average)\n",
+ "\n",
+ "def multiclass_positive_predictive_value(confusion_matrix, average):\n",
+ " return multiclass_score(confusion_matrix, positive_predictive_value_standardized, average)\n",
+ "\n",
+ "def multiclass_negative_predictive_value(confusion_matrix, average):\n",
+ " return multiclass_score(confusion_matrix, negative_predictive_value_standardized, average)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 59,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.scores import accuracy_standardized, sensitivity_standardized, specificity_standardized, balanced_accuracy_standardized, positive_predictive_value_standardized"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 60,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.640625"
+ ]
+ },
+ "execution_count": 60,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiclass_accuracy(confusion_fold)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 61,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.metrics import precision_recall_fscore_support"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 62,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "y_true = np.random.randint(5, size=20)\n",
+ "y_pred = np.random.randint(5, size=20)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 63,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(0.4041666666666667, 0.25, 0.2638888888888889, None)"
+ ]
+ },
+ "execution_count": 63,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "precision_recall_fscore_support(y_true, y_pred, average='weighted')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 64,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "confusion_matrix = np.zeros(shape=(5, 5))\n",
+ "for yt, yp in zip(y_true, y_pred):\n",
+ " confusion_matrix[yt, yp] += 1"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 65,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.25"
+ ]
+ },
+ "execution_count": 65,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiclass_sensitivity(confusion_matrix)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 66,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.4041666666666667"
+ ]
+ },
+ "execution_count": 66,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiclass_positive_predictive_value(confusion_matrix)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.66"
+ ]
+ },
+ "execution_count": 40,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiclass_accuracy(confusion_matrix)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.7875"
+ ]
+ },
+ "execution_count": 41,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiclass_specificity(confusion_matrix)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.18"
+ ]
+ },
+ "execution_count": 33,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multiclass_positive_predictive_value(confusion_matrix)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "tp = np.sum()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 45,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def accuracy_multiclass(confusion_matrix):\n",
+ " counts = np.sum(confusion_matrix, axis=1)\n",
+ " n_total = np.sum(counts)\n",
+ "\n",
+ " accs = []\n",
+ "\n",
+ " for idx, count in enumerate(counts):\n",
+ " p = counts[idx]\n",
+ " n = n_total - counts[idx]\n",
+ " tp = confusion_matrix[idx, idx]\n",
+ " tn = np.sum(confusion_matrix[0:idx, 0:idx]) + np.sum(confusion_matrix[idx+1:, idx+1:]) + np.sum(confusion_matrix[:idx, idx+1:]) + np.sum(confusion_matrix[idx+1:, :idx])\n",
+ " accs.append((tp + tn) / (p + n))\n",
+ "\n",
+ " return np.mean(accs)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 46,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.6875"
+ ]
+ },
+ "execution_count": 46,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "accuracy_multiclass(confusion_fold)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 47,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check import check_1_dataset_known_folds_mos_scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 50,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "binary_folds = transform_multiclass_fold_to_binary(fold)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 52,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a065da42bbc34847a634d09fae57c662-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a065da42bbc34847a634d09fae57c662-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 7 COLUMNS\n",
+ "At line 41 RHS\n",
+ "At line 44 BOUNDS\n",
+ "At line 54 ENDATA\n",
+ "Problem MODEL has 2 rows, 9 columns and 16 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0000I Cut generators found to be infeasible! (or unbounded)\n",
+ "Pre-processing says infeasible or unbounded\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{'inconsistency': True,\n",
+ " 'lp_status': 'infeasible',\n",
+ " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 1,\n",
+ " 'n': 7,\n",
+ " 'identifier': 'fkrkv',\n",
+ " 'tp': 0.0,\n",
+ " 'tn': 7.0},\n",
+ " 'scores': {'acc': 0.875},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 2, 'n': 6, 'identifier': 'wpxvc', 'tp': 0.0, 'tn': 6.0},\n",
+ " 'scores': {'acc': 0.75},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 1, 'n': 7, 'identifier': 'ngfid', 'tp': 0.0, 'tn': 7.0},\n",
+ " 'scores': {'acc': 0.875},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 4,\n",
+ " 'n': 4,\n",
+ " 'identifier': 'yvtsr',\n",
+ " 'tp': 0.0,\n",
+ " 'tn': 1.990368},\n",
+ " 'scores': {'acc': 0.248796},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True},\n",
+ " 'scores': {'acc': 0.687199},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': {'folds': [{'fold': {'p': 1,\n",
+ " 'n': 7,\n",
+ " 'identifier': 'fkrkv',\n",
+ " 'tp': 0.0,\n",
+ " 'tn': 7.0},\n",
+ " 'scores': {'acc': 0.875},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 2, 'n': 6, 'identifier': 'wpxvc', 'tp': 0.0, 'tn': 6.0},\n",
+ " 'scores': {'acc': 0.75},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 1, 'n': 7, 'identifier': 'ngfid', 'tp': 0.0, 'tn': 7.0},\n",
+ " 'scores': {'acc': 0.875},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 4,\n",
+ " 'n': 4,\n",
+ " 'identifier': 'yvtsr',\n",
+ " 'tp': 0.0,\n",
+ " 'tn': 1.990368},\n",
+ " 'scores': {'acc': 0.248796},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True}}],\n",
+ " 'bounds_flag': True}}"
+ ]
+ },
+ "execution_count": 52,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "check_1_dataset_known_folds_mos_scores(dataset={'p': 8, 'n': 24}, folding={'folds': binary_folds}, eps=1e-4, scores={'acc': 0.6873})"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/00_check_1_testset_no_kfold.ipynb b/notebooks/illustration/00_check_1_testset_no_kfold.ipynb
deleted file mode 100644
index 058a9ac..0000000
--- a/notebooks/illustration/00_check_1_testset_no_kfold.ipynb
+++ /dev/null
@@ -1,358 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "from mlscorecheck.check import check_1_testset_no_kfold_scores\n",
- "from mlscorecheck.individual import calculate_scores, generate_problem_and_scores"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:37:54,468:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:37:54,472:INFO:calling the score check with scores {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}, uncertainty 0.01, p 530 and n 902\n",
- "2023-10-26 23:37:54,473:INFO:checking the scores {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
- "2023-10-26 23:37:54,474:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:37:54,474:INFO:intervals before: (0, 530), (0, 902)\n",
- "2023-10-26 23:37:54,475:INFO:the tp solutions: (111.29894, 121.90106)\n",
- "2023-10-26 23:37:54,476:INFO:the tn solutions: (751.6160759999999, 790.8639240000001)\n",
- "2023-10-26 23:37:54,476:INFO:intervals after: [(112, 121)], [(752, 790)]\n",
- "2023-10-26 23:37:54,477:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:37:54,478:INFO:intervals before: [(112, 121)], [(752, 790)]\n",
- "2023-10-26 23:37:54,479:INFO:the tp solutions: (88.77533199999993, 135.4646680000002)\n",
- "2023-10-26 23:37:54,480:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
- "2023-10-26 23:37:54,481:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,482:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:37:54,483:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,485:INFO:the tp solutions: (108.20512460931658, 125.45746857652338)\n",
- "2023-10-26 23:37:54,486:INFO:the tn solutions: (720.729741489652, 822.9954069390577)\n",
- "2023-10-26 23:37:54,486:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,487:INFO:evaluating the tp and tn solution for acc and fm\n",
- "2023-10-26 23:37:54,488:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,489:INFO:the tp solutions: (-16.316815204318672, 3.5854769758786067) | (101.7831884021151, 128.25349360431866)\n",
- "2023-10-26 23:37:54,489:INFO:the tn solutions: (745.2636423956811, 800.379675597885) | (866.6502819978749, 921.7663152000788)\n",
- "2023-10-26 23:37:54,491:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,492:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:37:54,492:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,493:INFO:the tp solutions: (111.29894, 121.90106)\n",
- "2023-10-26 23:37:54,493:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
- "2023-10-26 23:37:54,495:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,495:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:37:54,496:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,497:INFO:the tp solutions: (111.29894, 121.90106)\n",
- "2023-10-26 23:37:54,498:INFO:the tn solutions: (657.2586176931766, 893.5046324530509)\n",
- "2023-10-26 23:37:54,499:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,500:INFO:evaluating the tp and tn solution for sens and fm\n",
- "2023-10-26 23:37:54,501:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,502:INFO:the tp solutions: (111.29894, 121.90106)\n",
- "2023-10-26 23:37:54,502:INFO:the tn solutions: (636.7166535970434, 917.0942703288408)\n",
- "2023-10-26 23:37:54,503:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,504:INFO:evaluating the tp and tn solution for spec and f1p\n",
- "2023-10-26 23:37:54,505:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,505:INFO:the tp solutions: (109.76804841374923, 122.0385407814731)\n",
- "2023-10-26 23:37:54,508:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
- "2023-10-26 23:37:54,508:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,509:INFO:evaluating the tp and tn solution for spec and fm\n",
- "2023-10-26 23:37:54,510:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,510:INFO:the tp solutions: (-65.8536328094769, -54.25856650052427) | (106.83422730052428, 121.82197200947691)\n",
- "2023-10-26 23:37:54,511:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
- "2023-10-26 23:37:54,512:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,512:INFO:evaluating the tp and tn solution for f1p and fm\n",
- "2023-10-26 23:37:54,514:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,517:INFO:the tp solutions: (80.82924045095699, 294.0107283921828)\n",
- "2023-10-26 23:37:54,518:INFO:the tn solutions: (-420.04420024360695, 1292.9374484676016)\n",
- "2023-10-26 23:37:54,519:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
- "2023-10-26 23:37:54,520:INFO:testing acc, feasible tptn pairs: 10\n",
- "2023-10-26 23:37:54,522:INFO:testing sens, feasible tptn pairs: 10\n",
- "2023-10-26 23:37:54,522:INFO:testing spec, feasible tptn pairs: 10\n",
- "2023-10-26 23:37:54,523:INFO:testing f1p, feasible tptn pairs: 10\n",
- "2023-10-26 23:37:54,525:INFO:testing fm, feasible tptn pairs: 10\n",
- "2023-10-26 23:37:54,526:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:37:54,526:INFO:final number of intervals: 10\n",
- "2023-10-26 23:37:54,527:INFO:final number of pairs: 130\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "False"
- ]
- },
- "execution_count": 2,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "testset = {'p': 530, 'n': 902}\n",
- "scores = {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
- "\n",
- "result = check_1_testset_no_kfold_scores(testset=testset,\n",
- " scores=scores,\n",
- " eps=1e-2)\n",
- "result['inconsistency']"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:37:54,544:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:37:54,545:INFO:calling the score check with scores {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}, uncertainty 0.01, p 530 and n 902\n",
- "2023-10-26 23:37:54,547:INFO:checking the scores {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
- "2023-10-26 23:37:54,548:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:37:54,549:INFO:intervals before: (0, 530), (0, 902)\n",
- "2023-10-26 23:37:54,550:INFO:the tp solutions: (111.29894, 121.90106)\n",
- "2023-10-26 23:37:54,551:INFO:the tn solutions: (1181.216076, 1220.4639240000004)\n",
- "2023-10-26 23:37:54,551:INFO:intervals after: [(112, 121)], []\n",
- "2023-10-26 23:37:54,552:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:37:54,553:INFO:intervals before: [(112, 121)], []\n",
- "2023-10-26 23:37:54,554:INFO:the tp solutions: (518.3753319999998, 565.0646680000003)\n",
- "2023-10-26 23:37:54,555:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
- "2023-10-26 23:37:54,556:INFO:intervals after: [], []\n",
- "2023-10-26 23:37:54,557:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:37:54,559:INFO:intervals before: [], []\n",
- "2023-10-26 23:37:54,559:INFO:the tp solutions: (20.470765551173063, 28.952218416378013)\n",
- "2023-10-26 23:37:54,560:INFO:the tn solutions: (1232.0122032291965, 1355.3280073234066)\n",
- "2023-10-26 23:37:54,561:INFO:intervals after: [], []\n",
- "2023-10-26 23:37:54,561:INFO:evaluating the tp and tn solution for acc and fm\n",
- "2023-10-26 23:37:54,564:INFO:all negative bases - iteration finished\n",
- "2023-10-26 23:37:54,565:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:37:54,566:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:37:54,567:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:37:54,568:INFO:final number of intervals: 0\n",
- "2023-10-26 23:37:54,569:INFO:final number of pairs: 0\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "True"
- ]
- },
- "execution_count": 3,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "testset = {'p': 530, 'n': 902}\n",
- "scores = {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
- "\n",
- "result = check_1_testset_no_kfold_scores(testset=testset,\n",
- " scores=scores,\n",
- " eps=1e-2)\n",
- "result['inconsistency']"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:37:54,581:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:37:54,582:INFO:calling the score check with scores {'acc': 0.954, 'sens': 0.934, 'spec': 0.985, 'ppv': 0.901}, uncertainty 0.001, p 1029 and n 3118\n",
- "2023-10-26 23:37:54,583:INFO:checking the scores {'acc': 0.954, 'sens': 0.934, 'spec': 0.985, 'ppv': 0.901}\n",
- "2023-10-26 23:37:54,584:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:37:54,585:INFO:intervals before: (0, 1029), (0, 3118)\n",
- "2023-10-26 23:37:54,586:INFO:the tp solutions: (960.054942, 962.1170580000002)\n",
- "2023-10-26 23:37:54,587:INFO:the tn solutions: (2989.965647999999, 3000.3383520000007)\n",
- "2023-10-26 23:37:54,587:INFO:intervals after: [(961, 962)], [(2990, 3000)]\n",
- "2023-10-26 23:37:54,588:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:37:54,589:INFO:intervals before: [(961, 962)], [(2990, 3000)]\n",
- "2023-10-26 23:37:54,590:INFO:the tp solutions: (877.7284699999991, 892.2875300000005)\n",
- "2023-10-26 23:37:54,591:INFO:the tn solutions: (3068.105764, 3074.354236)\n",
- "2023-10-26 23:37:54,592:INFO:intervals after: [], []\n",
- "2023-10-26 23:37:54,593:INFO:evaluating the tp and tn solution for acc and ppv\n",
- "2023-10-26 23:37:54,594:INFO:intervals before: [], []\n",
- "2023-10-26 23:37:54,595:INFO:the tp solutions: (933.6679509487356, 949.8052939947056)\n",
- "2023-10-26 23:37:54,595:INFO:the tn solutions: (2988.3769250334417, 3040.8179565580203)\n",
- "2023-10-26 23:37:54,596:INFO:intervals after: [], []\n",
- "2023-10-26 23:37:54,599:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:37:54,600:INFO:intervals before: [], []\n",
- "2023-10-26 23:37:54,601:INFO:the tp solutions: (960.054942, 962.1170580000002)\n",
- "2023-10-26 23:37:54,602:INFO:the tn solutions: (3068.105764, 3074.354236)\n",
- "2023-10-26 23:37:54,602:INFO:intervals after: [], []\n",
- "2023-10-26 23:37:54,603:INFO:evaluating the tp and tn solution for sens and ppv\n",
- "2023-10-26 23:37:54,604:INFO:intervals before: [], []\n",
- "2023-10-26 23:37:54,604:INFO:the tp solutions: (960.054942, 962.1170580000002)\n",
- "2023-10-26 23:37:54,605:INFO:the tn solutions: (3002.3483691722586, 3022.4720549935846)\n",
- "2023-10-26 23:37:54,606:INFO:intervals after: [], []\n",
- "2023-10-26 23:37:54,607:INFO:evaluating the tp and tn solution for spec and ppv\n",
- "2023-10-26 23:37:54,608:INFO:intervals before: [], []\n",
- "2023-10-26 23:37:54,609:INFO:the tp solutions: (392.8031470217782, 459.241011658118)\n",
- "2023-10-26 23:37:54,610:INFO:the tn solutions: (3068.105764, 3074.354236)\n",
- "2023-10-26 23:37:54,610:INFO:intervals after: [], []\n",
- "2023-10-26 23:37:54,611:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:37:54,612:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:37:54,613:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:37:54,613:INFO:final number of intervals: 0\n",
- "2023-10-26 23:37:54,616:INFO:final number of pairs: 0\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "True"
- ]
- },
- "execution_count": 4,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "result = check_1_testset_no_kfold_scores(\n",
- " scores={'acc': 0.954, 'sens': 0.934, 'spec': 0.985, 'ppv': 0.901},\n",
- " eps=1e-3,\n",
- " testset={'name': 'common_datasets.ADA'}\n",
- " )\n",
- "result['inconsistency']"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "({'p': 112, 'n': 268},\n",
- " {'acc': 0.55, 'sens': 0.99, 'spec': 0.37, 'f1p': 0.57, 'fm': 0.63})"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "problem, scores = generate_problem_and_scores(rounding_decimals=2,\n",
- " score_subset=['acc', 'sens', 'spec',\n",
- " 'f1p', 'fm'])\n",
- "problem, scores"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:37:54,643:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:37:54,645:INFO:calling the score check with scores {'acc': 0.55, 'sens': 0.99, 'spec': 0.37, 'f1p': 0.57, 'fm': 0.63}, uncertainty 0.01, p 112 and n 268\n",
- "2023-10-26 23:37:54,646:INFO:checking the scores {'acc': 0.55, 'sens': 0.99, 'spec': 0.37, 'f1p': 0.57, 'fm': 0.63}\n",
- "2023-10-26 23:37:54,647:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:37:54,648:INFO:intervals before: (0, 112), (0, 268)\n",
- "2023-10-26 23:37:54,648:INFO:the tp solutions: (109.75977599999999, 112)\n",
- "2023-10-26 23:37:54,650:INFO:the tn solutions: (93.19923999999997, 103.04098400000004)\n",
- "2023-10-26 23:37:54,652:INFO:intervals after: [(110, 112)], [(94, 103)]\n",
- "2023-10-26 23:37:54,653:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:37:54,653:INFO:intervals before: [(110, 112)], [(94, 103)]\n",
- "2023-10-26 23:37:54,654:INFO:the tp solutions: (103.35870399999999, 116.32129600000002)\n",
- "2023-10-26 23:37:54,655:INFO:the tn solutions: (96.47946400000001, 101.84053599999999)\n",
- "2023-10-26 23:37:54,656:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,656:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:37:54,657:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,658:INFO:the tp solutions: (106.39865273512385, 120.69675379587524)\n",
- "2023-10-26 23:37:54,661:INFO:the tn solutions: (75.65062613179025, 116.5375977962754)\n",
- "2023-10-26 23:37:54,662:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,663:INFO:evaluating the tp and tn solution for acc and fm\n",
- "2023-10-26 23:37:54,665:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,666:INFO:the tp solutions: (-26.167013822737445, -20.5967483940202) | (108.09086615402019, 116.48409606273746)\n",
- "2023-10-26 23:37:54,666:INFO:the tn solutions: (88.71514393726252, 104.70989384597986) | (224.36209719312416, 240.3568471018415)\n",
- "2023-10-26 23:37:54,667:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,669:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:37:54,669:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,670:INFO:the tp solutions: (109.75977599999999, 112)\n",
- "2023-10-26 23:37:54,671:INFO:the tn solutions: (96.47946400000001, 101.84053599999999)\n",
- "2023-10-26 23:37:54,671:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,672:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:37:54,673:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,673:INFO:the tp solutions: (109.75977599999999, 112)\n",
- "2023-10-26 23:37:54,674:INFO:the tn solutions: (86.66262364689761, 117.57440562287725)\n",
- "2023-10-26 23:37:54,675:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,675:INFO:evaluating the tp and tn solution for sens and fm\n",
- "2023-10-26 23:37:54,676:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,676:INFO:the tp solutions: (109.75977599999999, 112)\n",
- "2023-10-26 23:37:54,677:INFO:the tn solutions: (81.07840207674045, 125.0908482462938)\n",
- "2023-10-26 23:37:54,681:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,682:INFO:evaluating the tp and tn solution for spec and f1p\n",
- "2023-10-26 23:37:54,684:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,684:INFO:the tp solutions: (108.17258831659396, 115.80472502149439)\n",
- "2023-10-26 23:37:54,685:INFO:the tn solutions: (96.47946400000001, 101.84053599999999)\n",
- "2023-10-26 23:37:54,686:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,686:INFO:evaluating the tp and tn solution for spec and fm\n",
- "2023-10-26 23:37:54,687:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,688:INFO:the tp solutions: (-69.4019512822148, -65.05442173085025) | (108.80148061085025, 114.5604924022148)\n",
- "2023-10-26 23:37:54,689:INFO:the tn solutions: (96.47946400000001, 101.84053599999999)\n",
- "2023-10-26 23:37:54,689:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,690:INFO:evaluating the tp and tn solution for f1p and fm\n",
- "2023-10-26 23:37:54,691:INFO:intervals before: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,692:INFO:the tp solutions: (37.132383488314474, 53.878624053768796) | (100.45698847265774, 121.5814110951982)\n",
- "2023-10-26 23:37:54,692:INFO:the tn solutions: (46.44700588745767, 161.96547642627584) | (205.3172590296215, 332.388638045686)\n",
- "2023-10-26 23:37:54,693:INFO:intervals after: [(110, 112)], [(97, 101)]\n",
- "2023-10-26 23:37:54,693:INFO:testing acc, feasible tptn pairs: 3\n",
- "2023-10-26 23:37:54,694:INFO:testing sens, feasible tptn pairs: 3\n",
- "2023-10-26 23:37:54,695:INFO:testing spec, feasible tptn pairs: 3\n",
- "2023-10-26 23:37:54,696:INFO:testing f1p, feasible tptn pairs: 3\n",
- "2023-10-26 23:37:54,697:INFO:testing fm, feasible tptn pairs: 3\n",
- "2023-10-26 23:37:54,698:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:37:54,699:INFO:final number of intervals: 3\n",
- "2023-10-26 23:37:54,700:INFO:final number of pairs: 13\n"
- ]
- }
- ],
- "source": [
- "result = check_1_testset_no_kfold_scores(scores=scores,\n",
- " eps=1e-2,\n",
- " testset=problem)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "mlscorecheck",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.0"
- },
- "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/notebooks/illustration/30_skin_lesion_analysis.ipynb b/notebooks/illustration/00_skin_lesion_analysis.ipynb
similarity index 100%
rename from notebooks/illustration/30_skin_lesion_analysis.ipynb
rename to notebooks/illustration/00_skin_lesion_analysis.ipynb
diff --git a/notebooks/illustration/40_capsule.ipynb b/notebooks/illustration/01_capsule.ipynb
similarity index 99%
rename from notebooks/illustration/40_capsule.ipynb
rename to notebooks/illustration/01_capsule.ipynb
index 16e44ef..18d5692 100644
--- a/notebooks/illustration/40_capsule.ipynb
+++ b/notebooks/illustration/01_capsule.ipynb
@@ -517,7 +517,7 @@
}
],
"source": [
- "from mlscorecheck.check import check_1_dataset_som_scores\n",
+ "from mlscorecheck.check import check_1_dataset_kfold_som_scores\n",
"\n",
"dataset = {'dataset_name': 'common_datasets.monk-2'}\n",
"folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}\n",
@@ -525,7 +525,7 @@
"scores = {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667,\n",
" 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
"\n",
- "result = check_1_dataset_som_scores(dataset=dataset,\n",
+ "result = check_1_dataset_kfold_som_scores(dataset=dataset,\n",
" folding=folding,\n",
" scores=scores,\n",
" eps=1e-3)\n",
@@ -566,7 +566,7 @@
"scores = {'spec': 0.668, 'npv': 0.754, 'ppv': 0.667,\n",
" 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
"\n",
- "result = check_1_dataset_som_scores(dataset=dataset,\n",
+ "result = check_1_dataset_kfold_som_scores(dataset=dataset,\n",
" folding=folding,\n",
" scores=scores,\n",
" eps=1e-3)\n",
diff --git a/notebooks/illustration/01_check_1_dataset_som.ipynb b/notebooks/illustration/01_check_1_dataset_som.ipynb
deleted file mode 100644
index 3473cab..0000000
--- a/notebooks/illustration/01_check_1_dataset_som.ipynb
+++ /dev/null
@@ -1,182 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "from mlscorecheck.check import check_1_dataset_som_scores\n",
- "from mlscorecheck.aggregated import Dataset, Evaluation, generate_evaluation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "({'dataset': {'p': 195, 'n': 402},\n",
- " 'folding': {'n_folds': 10, 'n_repeats': 5, 'strategy': 'stratified_sklearn'},\n",
- " 'fold_score_bounds': None,\n",
- " 'aggregation': 'mos'},\n",
- " {'acc': 0.487, 'sens': 0.467, 'spec': 0.496, 'bacc': 0.481})"
- ]
- },
- "execution_count": 2,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "dataset, scores = generate_evaluation(return_scores=True, rounding_decimals=3)\n",
- "dataset, scores"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:38:05,126:INFO:testing spec, feasible tptn pairs: 613\n",
- "2023-10-26 23:38:05,129:INFO:testing bacc, feasible tptn pairs: 613\n",
- "2023-10-26 23:38:05,144:INFO:testing npv, feasible tptn pairs: 2\n",
- "2023-10-26 23:38:05,145:INFO:testing ppv, feasible tptn pairs: 1\n",
- "2023-10-26 23:38:05,146:INFO:testing f1p, feasible tptn pairs: 1\n",
- "2023-10-26 23:38:05,148:INFO:testing fm, feasible tptn pairs: 1\n",
- "2023-10-26 23:38:05,148:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:38:05,150:INFO:final number of intervals: 1\n",
- "2023-10-26 23:38:05,151:INFO:final number of pairs: 1\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "False"
- ]
- },
- "execution_count": 3,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "dataset = {'dataset_name': 'common_datasets.monk-2'}\n",
- "folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}\n",
- "scores = {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667, 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
- "result = check_1_dataset_som_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
- "result['inconsistency']"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:38:05,163:INFO:testing spec, feasible tptn pairs: 613\n",
- "2023-10-26 23:38:05,166:INFO:testing bacc, feasible tptn pairs: 613\n",
- "2023-10-26 23:38:05,184:INFO:testing npv, feasible tptn pairs: 2\n",
- "2023-10-26 23:38:05,185:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:38:05,186:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:38:05,187:INFO:final number of intervals: 0\n",
- "2023-10-26 23:38:05,187:INFO:final number of pairs: 0\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "True"
- ]
- },
- "execution_count": 4,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "dataset = {'dataset_name': 'common_datasets.monk-2'}\n",
- "folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}\n",
- "scores = {'spec': 0.668, 'npv': 0.544, 'ppv': 0.667, 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
- "result = check_1_dataset_som_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
- "result['inconsistency']"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:38:05,199:INFO:testing acc, feasible tptn pairs: 11\n",
- "2023-10-26 23:38:05,202:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:38:05,203:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:38:05,204:INFO:final number of intervals: 0\n",
- "2023-10-26 23:38:05,205:INFO:final number of pairs: 0\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "True"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "dataset = {'p': 10, 'n': 20}\n",
- "folding = {'n_folds': 5, 'n_repeats': 1}\n",
- "scores = {'acc': 0.428, 'npv': 0.392, 'bacc': 0.442, 'f1p': 0.391}\n",
- "\n",
- "result = check_1_dataset_som_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
- "result['inconsistency']"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "mlscorecheck",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.0"
- },
- "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/notebooks/illustration/08_check_n_testsets_mos_no_kfold.ipynb b/notebooks/illustration/08_check_n_testsets_mos_no_kfold.ipynb
deleted file mode 100644
index d86ccc3..0000000
--- a/notebooks/illustration/08_check_n_testsets_mos_no_kfold.ipynb
+++ /dev/null
@@ -1,431 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "from mlscorecheck.check import check_n_testsets_mos_no_kfold_scores"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [],
- "source": [
- "testsets = [{'p': 349, 'n': 50},\n",
- " {'p': 478, 'n': 323},\n",
- " {'p': 324, 'n': 83},\n",
- " {'p': 123, 'n': 145}]\n",
- "\n",
- "scores = {'acc': 0.6441, 'sens': 0.6706, 'spec': 0.3796, 'bacc': 0.5251}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Welcome to the CBC MILP Solver \n",
- "Version: 2.10.3 \n",
- "Build Date: Dec 15 2019 \n",
- "\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d8f56a21f91e4a1bb0c6fd59d083cfc1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d8f56a21f91e4a1bb0c6fd59d083cfc1-pulp.sol (default strategy 1)\n",
- "At line 2 NAME MODEL\n",
- "At line 3 ROWS\n",
- "At line 13 COLUMNS\n",
- "At line 79 RHS\n",
- "At line 88 BOUNDS\n",
- "At line 98 ENDATA\n",
- "Problem MODEL has 8 rows, 9 columns and 48 elements\n",
- "Coin0008I MODEL read with 0 errors\n",
- "Option for timeMode changed from cpu to elapsed\n",
- "Continuous objective value is 0 - 0.00 seconds\n",
- "Cgl0004I processed model has 4 rows, 8 columns (8 integer (0 of which binary)) and 24 elements\n",
- "Cbc0031I 2 added rows had average density of 8\n",
- "Cbc0013I At root node, 2 cuts changed objective from 0 to 0 in 28 passes\n",
- "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.003 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 1 (Gomory) - 51 row cuts average 7.5 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 2 row cuts average 3.5 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 6 (TwoMirCuts) - 6 row cuts average 3.3 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
- "Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.03 seconds)\n",
- "Cbc0016I Integer solution of 0 found by strong branching after 114 iterations and 26 nodes (0.03 seconds)\n",
- "Cbc0001I Search completed - best objective 0, took 114 iterations and 26 nodes (0.03 seconds)\n",
- "Cbc0032I Strong branching done 138 times (203 iterations), fathomed 1 nodes and fixed 31 variables\n",
- "Cbc0035I Maximum depth 22, 0 variables fixed on reduced cost\n",
- "Cuts at root node changed objective from 0 to 0\n",
- "Probing was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.003 seconds)\n",
- "Gomory was tried 28 times and created 51 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
- "Knapsack was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "Clique was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "MixedIntegerRounding2 was tried 28 times and created 2 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
- "FlowCover was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "TwoMirCuts was tried 28 times and created 6 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
- "ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "\n",
- "Result - Optimal solution found\n",
- "\n",
- "Objective value: 0.00000000\n",
- "Enumerated nodes: 26\n",
- "Total iterations: 114\n",
- "Time (CPU seconds): 0.02\n",
- "Time (Wallclock seconds): 0.03\n",
- "\n",
- "Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.03 (Wallclock seconds): 0.05\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "{'inconsistency': False,\n",
- " 'lp_scores': {'acc': 0.6440088293794702,\n",
- " 'sens': 0.6706134825883379,\n",
- " 'spec': 0.3795180722891566,\n",
- " 'bacc': 0.5250657774387473},\n",
- " 'lp_status': 'feasible',\n",
- " 'lp_configuration_scores_match': True,\n",
- " 'lp_configuration_bounds_match': True,\n",
- " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 349,\n",
- " 'n': 50,\n",
- " 'identifier': 'bslgx_0',\n",
- " 'tp': 343.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.8596491228070176,\n",
- " 'sens': 0.982808022922636,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.491404011461318},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.8596491228070176,\n",
- " 'sens': 0.982808022922636,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.491404011461318},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 349,\n",
- " 'n': 50,\n",
- " 'identifier': 'bslgx_0',\n",
- " 'tp': 343.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.8596491228070176,\n",
- " 'sens': 0.982808022922636,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.491404011461318},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}},\n",
- " {'folds': {'folds': [{'fold': {'p': 478,\n",
- " 'n': 323,\n",
- " 'identifier': 'fiufl_1',\n",
- " 'tp': 272.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.33957553058676654,\n",
- " 'sens': 0.5690376569037656,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.2845188284518828},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.33957553058676654,\n",
- " 'sens': 0.5690376569037656,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.2845188284518828},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 478,\n",
- " 'n': 323,\n",
- " 'identifier': 'fiufl_1',\n",
- " 'tp': 272.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.33957553058676654,\n",
- " 'sens': 0.5690376569037656,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.2845188284518828},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}},\n",
- " {'folds': {'folds': [{'fold': {'p': 324,\n",
- " 'n': 83,\n",
- " 'identifier': 'bnafx_2',\n",
- " 'tp': 203.0,\n",
- " 'tn': 43.0},\n",
- " 'scores': {'acc': 0.6044226044226044,\n",
- " 'sens': 0.6265432098765432,\n",
- " 'spec': 0.5180722891566265,\n",
- " 'bacc': 0.5723077495165849},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.6044226044226044,\n",
- " 'sens': 0.6265432098765432,\n",
- " 'spec': 0.5180722891566265,\n",
- " 'bacc': 0.5723077495165849},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 324,\n",
- " 'n': 83,\n",
- " 'identifier': 'bnafx_2',\n",
- " 'tp': 203.0,\n",
- " 'tn': 43.0},\n",
- " 'scores': {'acc': 0.6044226044226044,\n",
- " 'sens': 0.6265432098765432,\n",
- " 'spec': 0.5180722891566265,\n",
- " 'bacc': 0.5723077495165849},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}},\n",
- " {'folds': {'folds': [{'fold': {'p': 123,\n",
- " 'n': 145,\n",
- " 'identifier': 'fuluc_3',\n",
- " 'tp': 62.0,\n",
- " 'tn': 145.0},\n",
- " 'scores': {'acc': 0.7723880597014925,\n",
- " 'sens': 0.5040650406504066,\n",
- " 'spec': 1.0,\n",
- " 'bacc': 0.7520325203252033},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.7723880597014925,\n",
- " 'sens': 0.5040650406504066,\n",
- " 'spec': 1.0,\n",
- " 'bacc': 0.7520325203252033},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 123,\n",
- " 'n': 145,\n",
- " 'identifier': 'fuluc_3',\n",
- " 'tp': 62.0,\n",
- " 'tn': 145.0},\n",
- " 'scores': {'acc': 0.7723880597014925,\n",
- " 'sens': 0.5040650406504066,\n",
- " 'spec': 1.0,\n",
- " 'bacc': 0.7520325203252033},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}}],\n",
- " 'bounds_flag': True}}"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "check_n_testsets_mos_no_kfold_scores(testsets=testsets,\n",
- " scores=scores,\n",
- " eps=0.0001)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [],
- "source": [
- "scores['sens'] = 0.6756"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Welcome to the CBC MILP Solver \n",
- "Version: 2.10.3 \n",
- "Build Date: Dec 15 2019 \n",
- "\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1807329629f84607bf51059a96b16fd1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1807329629f84607bf51059a96b16fd1-pulp.sol (default strategy 1)\n",
- "At line 2 NAME MODEL\n",
- "At line 3 ROWS\n",
- "At line 13 COLUMNS\n",
- "At line 79 RHS\n",
- "At line 88 BOUNDS\n",
- "At line 98 ENDATA\n",
- "Problem MODEL has 8 rows, 9 columns and 48 elements\n",
- "Coin0008I MODEL read with 0 errors\n",
- "Option for timeMode changed from cpu to elapsed\n",
- "Problem is infeasible - 0.00 seconds\n",
- "Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "{'inconsistency': True,\n",
- " 'lp_status': 'infeasible',\n",
- " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 349,\n",
- " 'n': 50,\n",
- " 'identifier': 'ihkbo_0',\n",
- " 'tp': 244.9966,\n",
- " 'tn': 30.326935},\n",
- " 'scores': {'acc': 0.6900339223057643,\n",
- " 'sens': 0.701995988538682,\n",
- " 'spec': 0.6065387,\n",
- " 'bacc': 0.654267344269341},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.6900339223057643,\n",
- " 'sens': 0.701995988538682,\n",
- " 'spec': 0.6065387,\n",
- " 'bacc': 0.654267344269341},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 349,\n",
- " 'n': 50,\n",
- " 'identifier': 'ihkbo_0',\n",
- " 'tp': 244.9966,\n",
- " 'tn': 30.326935},\n",
- " 'scores': {'acc': 0.6900339223057643,\n",
- " 'sens': 0.701995988538682,\n",
- " 'spec': 0.6065387,\n",
- " 'bacc': 0.654267344269341},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}},\n",
- " {'folds': {'folds': [{'fold': {'p': 478,\n",
- " 'n': 323,\n",
- " 'identifier': 'gabnd_1',\n",
- " 'tp': 478.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.5967540574282147,\n",
- " 'sens': 0.9999999999999999,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.49999999999999994},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.5967540574282147,\n",
- " 'sens': 0.9999999999999999,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.49999999999999994},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 478,\n",
- " 'n': 323,\n",
- " 'identifier': 'gabnd_1',\n",
- " 'tp': 478.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.5967540574282147,\n",
- " 'sens': 0.9999999999999999,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.49999999999999994},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}},\n",
- " {'folds': {'folds': [{'fold': {'p': 324,\n",
- " 'n': 83,\n",
- " 'identifier': 'agobc_2',\n",
- " 'tp': 324.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.7960687960687961,\n",
- " 'sens': 1.0,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.5},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.7960687960687961,\n",
- " 'sens': 1.0,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.5},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 324,\n",
- " 'n': 83,\n",
- " 'identifier': 'agobc_2',\n",
- " 'tp': 324.0,\n",
- " 'tn': 0.0},\n",
- " 'scores': {'acc': 0.7960687960687961,\n",
- " 'sens': 1.0,\n",
- " 'spec': 0.0,\n",
- " 'bacc': 0.5},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}},\n",
- " {'folds': {'folds': [{'fold': {'p': 123,\n",
- " 'n': 145,\n",
- " 'identifier': 'qorcw_3',\n",
- " 'tp': 0.0,\n",
- " 'tn': 132.16131},\n",
- " 'scores': {'acc': 0.49313921641791036,\n",
- " 'sens': 0.0,\n",
- " 'spec': 0.9114573103448275,\n",
- " 'bacc': 0.45572865517241373},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True},\n",
- " 'scores': {'acc': 0.49313921641791036,\n",
- " 'sens': 0.0,\n",
- " 'spec': 0.9114573103448275,\n",
- " 'bacc': 0.45572865517241373},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': {'folds': [{'fold': {'p': 123,\n",
- " 'n': 145,\n",
- " 'identifier': 'qorcw_3',\n",
- " 'tp': 0.0,\n",
- " 'tn': 132.16131},\n",
- " 'scores': {'acc': 0.49313921641791036,\n",
- " 'sens': 0.0,\n",
- " 'spec': 0.9114573103448275,\n",
- " 'bacc': 0.45572865517241373},\n",
- " 'score_bounds': None,\n",
- " 'bounds_flag': True}],\n",
- " 'bounds_flag': True}}],\n",
- " 'bounds_flag': True}}"
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "check_n_testsets_mos_no_kfold_scores(testsets=testsets,\n",
- " scores=scores,\n",
- " eps=0.0001)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "mlscorecheck",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.0"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/notebooks/illustration/23_retina_hrf.ipynb b/notebooks/illustration/23_retina_hrf.ipynb
deleted file mode 100644
index f72a965..0000000
--- a/notebooks/illustration/23_retina_hrf.ipynb
+++ /dev/null
@@ -1,227 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Testing the consistency of performance scored reported for the HRF retinal vessel segmentation dataset"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "from mlscorecheck.bundles.retina import (check_hrf_vessel_image,\n",
- " check_hrf_vessel_aggregated)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Aggregated scores"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "scores = {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}\n",
- "k = 4"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:42:33,586:INFO:checking the scores {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}\n",
- "2023-10-26 23:42:33,588:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:33,590:INFO:intervals before: (0, 28395044), (0, 282712163)\n",
- "2023-10-26 23:42:33,591:INFO:the tp solutions: (16082896.131512, 16088688.720488)\n",
- "2023-10-26 23:42:33,592:INFO:the tn solutions: (134486577.25309798, 134555835.712302)\n",
- "2023-10-26 23:42:33,593:INFO:intervals after: [(16082897, 16088688)], [(134486578, 134555835)]\n",
- "2023-10-26 23:42:33,594:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:33,595:INFO:intervals before: [(16082897, 16088688)], [(134486578, 134555835)]\n",
- "2023-10-26 23:42:33,596:INFO:the tp solutions: (16258151.907959998, 16379291.059439987)\n",
- "2023-10-26 23:42:33,597:INFO:the tn solutions: (134259440.784374, 134317114.065626)\n",
- "2023-10-26 23:42:33,598:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:33,599:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:33,600:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:33,601:INFO:the tp solutions: (16082896.131512, 16088688.720488)\n",
- "2023-10-26 23:42:33,602:INFO:the tn solutions: (134259440.784374, 134317114.065626)\n",
- "2023-10-26 23:42:33,603:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:33,605:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:33,605:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:33,606:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:33,607:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:33,608:INFO:final number of pairs: 0\n",
- "2023-10-26 23:42:33,646:INFO:checking the scores {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}\n",
- "2023-10-26 23:42:33,648:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:33,649:INFO:intervals before: (0, 28399210), (0, 339941270)\n",
- "2023-10-26 23:42:33,650:INFO:the tp solutions: (16085255.745579999, 16091049.18442)\n",
- "2023-10-26 23:42:33,651:INFO:the tn solutions: (162185006.45462, 162265941.35138002)\n",
- "2023-10-26 23:42:33,653:INFO:intervals after: [(16085256, 16091049)], [(162185007, 162265941)]\n",
- "2023-10-26 23:42:33,654:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:33,655:INFO:intervals before: [(16085256, 16091049)], [(162185007, 162265941)]\n",
- "2023-10-26 23:42:33,657:INFO:the tp solutions: (16769278.379500002, 16913767.8565)\n",
- "2023-10-26 23:42:33,660:INFO:the tn solutions: (161437429.24046, 161506777.25954)\n",
- "2023-10-26 23:42:33,661:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:33,662:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:33,664:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:33,666:INFO:the tp solutions: (16085255.745579999, 16091049.18442)\n",
- "2023-10-26 23:42:33,668:INFO:the tn solutions: (161437429.24046, 161506777.25954)\n",
- "2023-10-26 23:42:33,669:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:33,670:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:33,671:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:33,672:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:33,676:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:33,677:INFO:final number of pairs: 0\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "{'inconsistency_fov_mos': False,\n",
- " 'inconsistency_fov_som': True,\n",
- " 'inconsistency_all_mos': False,\n",
- " 'inconsistency_all_som': True}"
- ]
- },
- "execution_count": 3,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "results = check_hrf_vessel_aggregated(scores=scores,\n",
- " eps=10**(-k),\n",
- " imageset='all',\n",
- " verbosity=0)\n",
- "results['inconsistency']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Image level scores"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "img_identifier = '13_h'\n",
- "scores = {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:42:33,710:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:33,713:INFO:calling the score check with scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}, uncertainty 0.0001, p 712528 and n 6199936\n",
- "2023-10-26 23:42:33,715:INFO:checking the scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}\n",
- "2023-10-26 23:42:33,716:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:33,717:INFO:intervals before: (0, 712528), (0, 6199936)\n",
- "2023-10-26 23:42:33,718:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
- "2023-10-26 23:42:33,720:INFO:the tn solutions: (3484179.340416, 3485734.8387840004)\n",
- "2023-10-26 23:42:33,721:INFO:intervals after: [(359683, 359828)], [(3484180, 3485734)]\n",
- "2023-10-26 23:42:33,722:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:33,724:INFO:intervals before: [(359683, 359828)], [(3484180, 3485734)]\n",
- "2023-10-26 23:42:33,725:INFO:the tp solutions: (358390.98639999935, 361065.91600000067)\n",
- "2023-10-26 23:42:33,726:INFO:the tn solutions: (3484351.632128, 3485616.419072001)\n",
- "2023-10-26 23:42:33,727:INFO:intervals after: [(359683, 359828)], [(3484352, 3485616)]\n",
- "2023-10-26 23:42:33,729:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:33,731:INFO:intervals before: [(359683, 359828)], [(3484352, 3485616)]\n",
- "2023-10-26 23:42:33,733:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
- "2023-10-26 23:42:33,734:INFO:the tn solutions: (3484351.632128, 3485616.419072001)\n",
- "2023-10-26 23:42:33,735:INFO:intervals after: [(359683, 359828)], [(3484352, 3485616)]\n",
- "2023-10-26 23:42:33,745:INFO:testing acc, feasible tptn pairs: 146\n",
- "2023-10-26 23:42:33,757:INFO:testing sens, feasible tptn pairs: 146\n",
- "2023-10-26 23:42:33,759:INFO:testing spec, feasible tptn pairs: 144\n",
- "2023-10-26 23:42:33,765:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:33,767:INFO:final number of intervals: 144\n",
- "2023-10-26 23:42:33,769:INFO:final number of pairs: 180054\n",
- "2023-10-26 23:42:33,772:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:33,774:INFO:calling the score check with scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}, uncertainty 0.0001, p 712528 and n 7472816\n",
- "2023-10-26 23:42:33,777:INFO:checking the scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}\n",
- "2023-10-26 23:42:33,779:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:33,781:INFO:intervals before: (0, 712528), (0, 7472816)\n",
- "2023-10-26 23:42:33,784:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
- "2023-10-26 23:42:33,786:INFO:the tn solutions: (4192025.362656, 4193840.528544001)\n",
- "2023-10-26 23:42:33,787:INFO:intervals after: [(359683, 359828)], [(4192026, 4193840)]\n",
- "2023-10-26 23:42:33,789:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:33,790:INFO:intervals before: [(359683, 359828)], [(4192026, 4193840)]\n",
- "2023-10-26 23:42:33,791:INFO:the tp solutions: (350621.3268799996, 353815.5915200012)\n",
- "2023-10-26 23:42:33,792:INFO:the tn solutions: (4199707.646368, 4201232.100832)\n",
- "2023-10-26 23:42:33,793:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:33,794:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:33,795:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:33,796:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
- "2023-10-26 23:42:33,798:INFO:the tn solutions: (4199707.646368, 4201232.100832)\n",
- "2023-10-26 23:42:33,799:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:33,803:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:33,804:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:33,806:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:33,807:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:33,808:INFO:final number of pairs: 0\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "{'inconsistency_fov': False, 'inconsistency_all': True}"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "results = check_hrf_vessel_image(image_identifier=img_identifier,\n",
- " scores=scores,\n",
- " eps=10**(-k))\n",
- "results['inconsistency']"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "mlscorecheck",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.0"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/notebooks/illustration/26_drishti_gs.ipynb b/notebooks/illustration/26_drishti_gs.ipynb
deleted file mode 100644
index 8e407d5..0000000
--- a/notebooks/illustration/26_drishti_gs.ipynb
+++ /dev/null
@@ -1,294 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Testing the consistency of performance scored reported for the DRISHTI_GS retinal image dataset"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "metadata": {},
- "outputs": [],
- "source": [
- "from mlscorecheck.bundles.retina import (check_drishti_gs_segmentation_image,\n",
- " check_drishti_gs_segmentation_aggregated)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Image level scores, optic disk segmentation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 25,
- "metadata": {},
- "outputs": [],
- "source": [
- "scores = {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:42:54,160:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:54,167:INFO:calling the score check with scores {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}, uncertainty 0.0001, p 118933 and n 3487307\n",
- "2023-10-26 23:42:54,173:INFO:checking the scores {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}\n",
- "2023-10-26 23:42:54,189:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:54,197:INFO:intervals before: (0, 118933), (0, 3487307)\n",
- "2023-10-26 23:42:54,199:INFO:the tp solutions: (35667.768834, 35692.031166)\n",
- "2023-10-26 23:42:54,201:INFO:the tn solutions: (2115422.916354, 2116182.851646)\n",
- "2023-10-26 23:42:54,207:INFO:intervals after: [(35668, 35692)], [(2115423, 2116182)]\n",
- "2023-10-26 23:42:54,209:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:54,211:INFO:intervals before: [(35668, 35692)], [(2115423, 2116182)]\n",
- "2023-10-26 23:42:54,212:INFO:the tp solutions: (35010.0853059995, 36457.16889400035)\n",
- "2023-10-26 23:42:54,213:INFO:the tn solutions: (2115393.4515859997, 2116104.8622140004)\n",
- "2023-10-26 23:42:54,216:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,219:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:42:54,224:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,226:INFO:the tp solutions: (35622.01445335527, 35803.38811508206)\n",
- "2023-10-26 23:42:54,227:INFO:the tn solutions: (2114840.8000603416, 2116699.5836843182)\n",
- "2023-10-26 23:42:54,228:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,230:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:54,232:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,234:INFO:the tp solutions: (35667.768834, 35692.031166)\n",
- "2023-10-26 23:42:54,235:INFO:the tn solutions: (2115393.4515859997, 2116104.8622140004)\n",
- "2023-10-26 23:42:54,235:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,236:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:42:54,238:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,241:INFO:the tp solutions: (35667.768834, 35692.031166)\n",
- "2023-10-26 23:42:54,242:INFO:the tn solutions: (2104083.976312527, 2130248.7999003753)\n",
- "2023-10-26 23:42:54,243:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,245:INFO:evaluating the tp and tn solution for spec and f1p\n",
- "2023-10-26 23:42:54,247:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,249:INFO:the tp solutions: (35624.9728226002, 35801.4215434727)\n",
- "2023-10-26 23:42:54,250:INFO:the tn solutions: (2115393.4515859997, 2116104.8622140004)\n",
- "2023-10-26 23:42:54,251:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
- "2023-10-26 23:42:54,262:INFO:testing acc, feasible tptn pairs: 25\n",
- "2023-10-26 23:42:54,271:INFO:testing sens, feasible tptn pairs: 25\n",
- "2023-10-26 23:42:54,273:INFO:testing spec, feasible tptn pairs: 24\n",
- "2023-10-26 23:42:54,275:INFO:testing f1p, feasible tptn pairs: 24\n",
- "2023-10-26 23:42:54,279:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:54,281:INFO:final number of intervals: 24\n",
- "2023-10-26 23:42:54,282:INFO:final number of pairs: 15900\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'053': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3477881, 3219, 3590, 2617, 118933]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3505385, 22544, 7243, 7555, 63513]}}}\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "False"
- ]
- },
- "execution_count": 26,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "results = check_drishti_gs_segmentation_image(image_identifier='053',\n",
- " confidence=0.75,\n",
- " target='OD',\n",
- " scores=scores,\n",
- " eps=1e-4)\n",
- "results['inconsistency']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Aggregated scores, optic disk segmentation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {},
- "outputs": [],
- "source": [
- "scores = {'acc': 0.4767, 'sens': 0.4845, 'spec': 0.4765, 'f1p': 0.0512}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'053': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3477881, 3219, 3590, 2617, 118933]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3505385, 22544, 7243, 7555, 63513]}}, '086': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3661951, 8149, 8698, 4933, 76184]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3621348, 6545, 6160, 6846, 119016]}}, '079': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3508324, 13024, 6458, 11059, 55375]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3478285, 5415, 3027, 3578, 103935]}}, '052': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3460984, 4391, 4663, 5758, 122830]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3500423, 20101, 16820, 10694, 50588]}}, '082': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3495438, 4230, 4911, 4596, 95305]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3532354, 8797, 13359, 12241, 37729]}}, '003': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3485894, 5649, 3436, 7462, 98232]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3525707, 11625, 7097, 6819, 49425]}}, '020': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3489439, 14680, 15992, 10659, 67566]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3464685, 5263, 4686, 5838, 117864]}}, '077': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3506050, 21067, 13645, 17434, 48044]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3468009, 9077, 9049, 9181, 110924]}}, '027': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3510404, 3474, 3959, 7989, 85985]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3541050, 17683, 10632, 6316, 36130]}}, '039': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3503439, 11221, 12223, 8684, 60428]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3472465, 2680, 3853, 3597, 113400]}}, '072': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3552462, 11643, 10162, 4806, 23647]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3491680, 4073, 3871, 5145, 97951]}}, '013': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [4257502, 16045, 8020, 5049, 45801]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [4205765, 5256, 4783, 4878, 111735]}}, '009': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3476565, 7402, 3649, 3963, 100613]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3554212, 6394, 4096, 11459, 16031]}}, '067': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3476383, 3440, 2331, 3929, 105814]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3529008, 10509, 9310, 6708, 36362]}}, '025': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [4213796, 5044, 2468, 3012, 111956]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [4240127, 9969, 13053, 10728, 62399]}}, '001': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3464573, 9238, 5717, 10344, 97927]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3408681, 5831, 7793, 5558, 159936]}}, '093': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3551531, 2434, 1619, 9150, 15170]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3462006, 1234, 8029, 3851, 104784]}}, '065': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3465192, 4194, 3191, 5083, 105180]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3497362, 9735, 10890, 6815, 58038]}}, '073': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3684750, 7680, 7666, 11903, 56541]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3647265, 6093, 5362, 8176, 101644]}}, '054': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3544335, 6275, 7469, 3526, 36439]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3516203, 3277, 4215, 2817, 71532]}}, '005': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3398781, 6015, 4961, 3776, 187140]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3441715, 24143, 6491, 8771, 119553]}}, '055': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [4150128, 4008, 3103, 9715, 47095]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [4124376, 3621, 2716, 3229, 80107]}}, '019': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3439928, 6341, 5752, 4202, 150017]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3483926, 11207, 7465, 5421, 98221]}}, '050': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3520247, 4180, 3139, 5193, 83441]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3551833, 10269, 5164, 6325, 42609]}}, '006': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3444493, 4913, 4728, 7288, 132818]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3505093, 27315, 10992, 7558, 43282]}}, '011': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3521540, 5899, 11719, 3984, 46706]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3495819, 2568, 4250, 4182, 83029]}}, '034': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3511418, 22193, 15046, 8420, 46823]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3464088, 12118, 9740, 7015, 110939]}}, '056': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3503443, 2555, 2160, 6644, 89389]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3576587, 2423, 2879, 3356, 18946]}}, '070': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3527918, 7956, 3874, 4331, 64210]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3495585, 4522, 3158, 4430, 100594]}}, '095': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3487320, 5532, 3699, 6622, 80827]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3553364, 9035, 3230, 5359, 13012]}}, '083': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3525669, 6113, 18140, 3155, 49065]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3479762, 2750, 3895, 6735, 109000]}}, '021': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3523590, 6699, 5704, 6774, 62000]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3488319, 3908, 4365, 8637, 99538]}}, '087': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3500909, 4799, 1978, 3827, 95015]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3534433, 9279, 7343, 8887, 46586]}}, '023': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3504092, 10545, 3216, 3972, 74170]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3541710, 7115, 3839, 5885, 37446]}}, '099': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3517075, 8720, 11227, 5832, 35302]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3440950, 4506, 5653, 3105, 123942]}}, '014': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3514523, 5107, 5625, 12318, 66907]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3478973, 4047, 5795, 6485, 109180]}}, '007': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3442011, 5248, 7394, 11640, 133800]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3492784, 12081, 30033, 5792, 59403]}}, '074': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3494295, 2146, 1448, 3120, 101423]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3513128, 13307, 372, 9310, 66315]}}, '030': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3484311, 5033, 5851, 6467, 102818]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3521164, 15506, 4217, 20226, 43367]}}, '029': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3491452, 3821, 2175, 5221, 103571]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3520641, 14741, 5148, 11953, 53757]}}, '048': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3512052, 18930, 9537, 9802, 54159]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3451073, 5436, 3679, 4689, 139603]}}, '085': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3507466, 10468, 16404, 8686, 50922]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3467366, 5062, 4672, 6890, 109956]}}, '096': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3554148, 7737, 6819, 12781, 2216]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3477329, 12771, 5830, 2431, 85340]}}, '043': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3480618, 4456, 2393, 3392, 93438]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3501576, 19707, 9003, 6902, 47109]}}, '071': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3533440, 7800, 8170, 11186, 43595]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3492197, 2605, 3227, 3656, 102506]}}, '059': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3530646, 7495, 6097, 9295, 44803]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3490981, 4098, 2772, 3729, 96756]}}, '078': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3550802, 15998, 4450, 4679, 32647]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3496545, 5547, 3280, 2737, 100467]}}, '097': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3533819, 8238, 5262, 6206, 30176]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3430080, 14459, 9522, 5696, 123944]}}, '028': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3500043, 6798, 3601, 4301, 91785]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3525252, 6527, 6825, 7390, 60534]}}, '100': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3567101, 3029, 4134, 2503, 6934]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3454615, 28483, 17462, 3221, 79920]}}, '091': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3490047, 4057, 1805, 3532, 82511]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3552600, 6092, 2373, 7464, 13423]}}}\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:42:54,499:INFO:checking the scores {'acc': 0.4767, 'sens': 0.4845, 'spec': 0.4765, 'f1p': 0.0512}\n",
- "2023-10-26 23:42:54,501:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:54,503:INFO:intervals before: (0, 5414553), (0, 180494984)\n",
- "2023-10-26 23:42:54,504:INFO:the tp solutions: (2622798.644094, 2623903.2129059997)\n",
- "2023-10-26 23:42:54,505:INFO:the tn solutions: (85980210.30222, 86019240.41657999)\n",
- "2023-10-26 23:42:54,507:INFO:intervals after: [(2622799, 2623903)], [(85980211, 86019240)]\n",
- "2023-10-26 23:42:54,508:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:54,510:INFO:intervals before: [(2622799, 2623903)], [(85980211, 86019240)]\n",
- "2023-10-26 23:42:54,511:INFO:the tp solutions: (2579843.150758013, 2654589.6730419993)\n",
- "2023-10-26 23:42:54,513:INFO:the tn solutions: (85987449.387632, 86024270.36436799)\n",
- "2023-10-26 23:42:54,515:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,516:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:42:54,518:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,519:INFO:the tp solutions: (2618908.280150996, 2630954.653441351)\n",
- "2023-10-26 23:42:54,521:INFO:the tn solutions: (85953650.68004246, 86042649.8249844)\n",
- "2023-10-26 23:42:54,522:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,523:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:54,524:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,526:INFO:the tp solutions: (2622798.644094, 2623903.2129059997)\n",
- "2023-10-26 23:42:54,527:INFO:the tn solutions: (85987449.387632, 86024270.36436799)\n",
- "2023-10-26 23:42:54,528:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,530:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:42:54,531:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,533:INFO:the tp solutions: (2622798.644094, 2623903.2129059997)\n",
- "2023-10-26 23:42:54,534:INFO:the tn solutions: (85490212.09562817, 86628542.44030103)\n",
- "2023-10-26 23:42:54,535:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,537:INFO:evaluating the tp and tn solution for spec and f1p\n",
- "2023-10-26 23:42:54,538:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
- "2023-10-26 23:42:54,539:INFO:the tp solutions: (2618878.4015550935, 2630578.4368279246)\n",
- "2023-10-26 23:42:54,541:INFO:the tn solutions: (85987449.387632, 86024270.36436799)\n",
- "2023-10-26 23:42:54,543:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Welcome to the CBC MILP Solver \n",
- "Version: 2.10.3 \n",
- "Build Date: Dec 15 2019 \n",
- "\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7f87319971994323a66f89b2f2f6c8b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7f87319971994323a66f89b2f2f6c8b8-pulp.sol (default strategy 1)\n",
- "At line 2 NAME MODEL\n",
- "At line 3 ROWS\n",
- "At line 11 COLUMNS\n",
- "At line 625 RHS\n",
- "At line 632 BOUNDS\n",
- "At line 736 ENDATA\n",
- "Problem MODEL has 6 rows, 103 columns and 408 elements\n",
- "Coin0008I MODEL read with 0 errors\n",
- "Option for timeMode changed from cpu to elapsed\n",
- "Continuous objective value is 0 - 0.00 seconds\n",
- "Cgl0004I processed model has 3 rows, 102 columns (102 integer (0 of which binary)) and 204 elements\n",
- "Cbc0012I Integer solution of 0 found by DiveCoefficient after 0 iterations and 0 nodes (0.00 seconds)\n",
- "Cbc0001I Search completed - best objective 0, took 0 iterations and 0 nodes (0.00 seconds)\n",
- "Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
- "Cuts at root node changed objective from 0 to 0\n",
- "Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "\n",
- "Result - Optimal solution found\n",
- "\n",
- "Objective value: 0.00000000\n",
- "Enumerated nodes: 0\n",
- "Total iterations: 0\n",
- "Time (CPU seconds): 0.01\n",
- "Time (Wallclock seconds): 0.01\n",
- "\n",
- "Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
- "\n",
- "{'053': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3477881, 3219, 3590, 2617, 118933]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3505385, 22544, 7243, 7555, 63513]}}, '086': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3661951, 8149, 8698, 4933, 76184]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3621348, 6545, 6160, 6846, 119016]}}, '079': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3508324, 13024, 6458, 11059, 55375]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3478285, 5415, 3027, 3578, 103935]}}, '052': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3460984, 4391, 4663, 5758, 122830]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3500423, 20101, 16820, 10694, 50588]}}, '082': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3495438, 4230, 4911, 4596, 95305]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3532354, 8797, 13359, 12241, 37729]}}, '003': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3485894, 5649, 3436, 7462, 98232]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3525707, 11625, 7097, 6819, 49425]}}, '020': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3489439, 14680, 15992, 10659, 67566]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3464685, 5263, 4686, 5838, 117864]}}, '077': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3506050, 21067, 13645, 17434, 48044]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3468009, 9077, 9049, 9181, 110924]}}, '027': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3510404, 3474, 3959, 7989, 85985]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3541050, 17683, 10632, 6316, 36130]}}, '039': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3503439, 11221, 12223, 8684, 60428]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3472465, 2680, 3853, 3597, 113400]}}, '072': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3552462, 11643, 10162, 4806, 23647]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3491680, 4073, 3871, 5145, 97951]}}, '013': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [4257502, 16045, 8020, 5049, 45801]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [4205765, 5256, 4783, 4878, 111735]}}, '009': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3476565, 7402, 3649, 3963, 100613]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3554212, 6394, 4096, 11459, 16031]}}, '067': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3476383, 3440, 2331, 3929, 105814]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3529008, 10509, 9310, 6708, 36362]}}, '025': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [4213796, 5044, 2468, 3012, 111956]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [4240127, 9969, 13053, 10728, 62399]}}, '001': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3464573, 9238, 5717, 10344, 97927]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3408681, 5831, 7793, 5558, 159936]}}, '093': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3551531, 2434, 1619, 9150, 15170]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3462006, 1234, 8029, 3851, 104784]}}, '065': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3465192, 4194, 3191, 5083, 105180]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3497362, 9735, 10890, 6815, 58038]}}, '073': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3684750, 7680, 7666, 11903, 56541]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3647265, 6093, 5362, 8176, 101644]}}, '054': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3544335, 6275, 7469, 3526, 36439]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3516203, 3277, 4215, 2817, 71532]}}, '005': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3398781, 6015, 4961, 3776, 187140]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3441715, 24143, 6491, 8771, 119553]}}, '055': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [4150128, 4008, 3103, 9715, 47095]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [4124376, 3621, 2716, 3229, 80107]}}, '019': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3439928, 6341, 5752, 4202, 150017]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3483926, 11207, 7465, 5421, 98221]}}, '050': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3520247, 4180, 3139, 5193, 83441]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3551833, 10269, 5164, 6325, 42609]}}, '006': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3444493, 4913, 4728, 7288, 132818]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3505093, 27315, 10992, 7558, 43282]}}, '011': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3521540, 5899, 11719, 3984, 46706]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3495819, 2568, 4250, 4182, 83029]}}, '034': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3511418, 22193, 15046, 8420, 46823]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3464088, 12118, 9740, 7015, 110939]}}, '056': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3503443, 2555, 2160, 6644, 89389]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3576587, 2423, 2879, 3356, 18946]}}, '070': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3527918, 7956, 3874, 4331, 64210]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3495585, 4522, 3158, 4430, 100594]}}, '095': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3487320, 5532, 3699, 6622, 80827]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3553364, 9035, 3230, 5359, 13012]}}, '083': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3525669, 6113, 18140, 3155, 49065]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3479762, 2750, 3895, 6735, 109000]}}, '021': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3523590, 6699, 5704, 6774, 62000]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3488319, 3908, 4365, 8637, 99538]}}, '087': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3500909, 4799, 1978, 3827, 95015]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3534433, 9279, 7343, 8887, 46586]}}, '023': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3504092, 10545, 3216, 3972, 74170]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3541710, 7115, 3839, 5885, 37446]}}, '099': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3517075, 8720, 11227, 5832, 35302]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3440950, 4506, 5653, 3105, 123942]}}, '014': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3514523, 5107, 5625, 12318, 66907]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3478973, 4047, 5795, 6485, 109180]}}, '007': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3442011, 5248, 7394, 11640, 133800]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3492784, 12081, 30033, 5792, 59403]}}, '074': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3494295, 2146, 1448, 3120, 101423]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3513128, 13307, 372, 9310, 66315]}}, '030': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3484311, 5033, 5851, 6467, 102818]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3521164, 15506, 4217, 20226, 43367]}}, '029': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3491452, 3821, 2175, 5221, 103571]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3520641, 14741, 5148, 11953, 53757]}}, '048': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3512052, 18930, 9537, 9802, 54159]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3451073, 5436, 3679, 4689, 139603]}}, '085': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3507466, 10468, 16404, 8686, 50922]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3467366, 5062, 4672, 6890, 109956]}}, '096': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3554148, 7737, 6819, 12781, 2216]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3477329, 12771, 5830, 2431, 85340]}}, '043': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3480618, 4456, 2393, 3392, 93438]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3501576, 19707, 9003, 6902, 47109]}}, '071': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3533440, 7800, 8170, 11186, 43595]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3492197, 2605, 3227, 3656, 102506]}}, '059': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3530646, 7495, 6097, 9295, 44803]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3490981, 4098, 2772, 3729, 96756]}}, '078': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3550802, 15998, 4450, 4679, 32647]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3496545, 5547, 3280, 2737, 100467]}}, '097': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3533819, 8238, 5262, 6206, 30176]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3430080, 14459, 9522, 5696, 123944]}}, '028': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3500043, 6798, 3601, 4301, 91785]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3525252, 6527, 6825, 7390, 60534]}}, '100': {'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3567101, 3029, 4134, 2503, 6934]}, 'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3454615, 28483, 17462, 3221, 79920]}}, '091': {'OD': {'values': [0, 64, 128, 191, 255], 'counts': [3490047, 4057, 1805, 3532, 82511]}, 'OC': {'values': [0, 64, 128, 191, 255], 'counts': [3552600, 6092, 2373, 7464, 13423]}}}\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:42:54,611:INFO:testing acc, feasible tptn pairs: 1105\n",
- "2023-10-26 23:42:54,727:INFO:testing sens, feasible tptn pairs: 1105\n",
- "2023-10-26 23:42:54,729:INFO:testing spec, feasible tptn pairs: 1093\n",
- "2023-10-26 23:42:54,751:INFO:testing f1p, feasible tptn pairs: 1093\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2023-10-26 23:42:54,851:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:54,853:INFO:final number of intervals: 1093\n",
- "2023-10-26 23:42:54,855:INFO:final number of pairs: 33744189\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "{'inconsistency_som': False, 'inconsistency_mos': False}"
- ]
- },
- "execution_count": 28,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "results = check_drishti_gs_segmentation_aggregated(subset='test',\n",
- " confidence=0.75,\n",
- " target='OD',\n",
- " scores=scores,\n",
- " eps=1e-4)\n",
- "results['inconsistency']"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "mlscorecheck",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.0"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/notebooks/illustration/binary/00_check_1_testset_no_kfold.ipynb b/notebooks/illustration/binary/00_check_1_testset_no_kfold.ipynb
new file mode 100644
index 0000000..7de27f8
--- /dev/null
+++ b/notebooks/illustration/binary/00_check_1_testset_no_kfold.ipynb
@@ -0,0 +1,358 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.binary import check_1_testset_no_kfold\n",
+ "from mlscorecheck.individual import generate_problem_and_scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 06:29:16,066:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 06:29:16,068:INFO:calling the score check with scores {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}, uncertainty 0.01, p 530 and n 902\n",
+ "2023-11-03 06:29:16,069:INFO:checking the scores {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
+ "2023-11-03 06:29:16,070:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 06:29:16,072:INFO:intervals before: (0, 530), (0, 902)\n",
+ "2023-11-03 06:29:16,072:INFO:the tp solutions: (111.29894, 121.90106)\n",
+ "2023-11-03 06:29:16,073:INFO:the tn solutions: (751.6160759999999, 790.8639240000001)\n",
+ "2023-11-03 06:29:16,075:INFO:intervals after: [(112, 121)], [(752, 790)]\n",
+ "2023-11-03 06:29:16,076:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 06:29:16,077:INFO:intervals before: [(112, 121)], [(752, 790)]\n",
+ "2023-11-03 06:29:16,078:INFO:the tp solutions: (88.77533199999993, 135.4646680000002)\n",
+ "2023-11-03 06:29:16,079:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
+ "2023-11-03 06:29:16,080:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,081:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 06:29:16,083:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,084:INFO:the tp solutions: (108.20512460931658, 125.45746857652338)\n",
+ "2023-11-03 06:29:16,084:INFO:the tn solutions: (720.729741489652, 822.9954069390577)\n",
+ "2023-11-03 06:29:16,085:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,090:INFO:evaluating the tp and tn solution for acc and fm\n",
+ "2023-11-03 06:29:16,092:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,094:INFO:the tp solutions: (-16.316815204318672, 3.5854769758786067) | (101.7831884021151, 128.25349360431866)\n",
+ "2023-11-03 06:29:16,095:INFO:the tn solutions: (745.2636423956811, 800.379675597885) | (866.6502819978749, 921.7663152000788)\n",
+ "2023-11-03 06:29:16,096:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,097:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 06:29:16,098:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,099:INFO:the tp solutions: (111.29894, 121.90106)\n",
+ "2023-11-03 06:29:16,100:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
+ "2023-11-03 06:29:16,101:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,102:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 06:29:16,103:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,104:INFO:the tp solutions: (111.29894, 121.90106)\n",
+ "2023-11-03 06:29:16,105:INFO:the tn solutions: (657.2586176931766, 893.5046324530509)\n",
+ "2023-11-03 06:29:16,106:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,107:INFO:evaluating the tp and tn solution for sens and fm\n",
+ "2023-11-03 06:29:16,114:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,115:INFO:the tp solutions: (111.29894, 121.90106)\n",
+ "2023-11-03 06:29:16,116:INFO:the tn solutions: (636.7166535970434, 917.0942703288408)\n",
+ "2023-11-03 06:29:16,117:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,118:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 06:29:16,119:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,120:INFO:the tp solutions: (109.76804841374923, 122.0385407814731)\n",
+ "2023-11-03 06:29:16,121:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
+ "2023-11-03 06:29:16,123:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,123:INFO:evaluating the tp and tn solution for spec and fm\n",
+ "2023-11-03 06:29:16,125:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,127:INFO:the tp solutions: (-65.8536328094769, -54.25856650052427) | (106.83422730052428, 121.82197200947691)\n",
+ "2023-11-03 06:29:16,128:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
+ "2023-11-03 06:29:16,129:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,130:INFO:evaluating the tp and tn solution for f1p and fm\n",
+ "2023-11-03 06:29:16,133:INFO:intervals before: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,134:INFO:the tp solutions: (80.82924045095699, 294.0107283921828)\n",
+ "2023-11-03 06:29:16,135:INFO:the tn solutions: (-420.04420024360695, 1292.9374484676016)\n",
+ "2023-11-03 06:29:16,136:INFO:intervals after: [(112, 121)], [(767, 784)]\n",
+ "2023-11-03 06:29:16,137:INFO:testing acc, feasible tptn pairs: 10\n",
+ "2023-11-03 06:29:16,139:INFO:testing sens, feasible tptn pairs: 10\n",
+ "2023-11-03 06:29:16,140:INFO:testing spec, feasible tptn pairs: 10\n",
+ "2023-11-03 06:29:16,148:INFO:testing f1p, feasible tptn pairs: 10\n",
+ "2023-11-03 06:29:16,150:INFO:testing fm, feasible tptn pairs: 10\n",
+ "2023-11-03 06:29:16,152:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:29:16,154:INFO:final number of intervals: 10\n",
+ "2023-11-03 06:29:16,155:INFO:final number of pairs: 130\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "testset = {'p': 530, 'n': 902}\n",
+ "scores = {'acc': 0.62, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
+ "\n",
+ "result = check_1_testset_no_kfold(testset=testset,\n",
+ " scores=scores,\n",
+ " eps=1e-2)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 06:29:16,176:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 06:29:16,179:INFO:calling the score check with scores {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}, uncertainty 0.01, p 530 and n 902\n",
+ "2023-11-03 06:29:16,180:INFO:checking the scores {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
+ "2023-11-03 06:29:16,182:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 06:29:16,183:INFO:intervals before: (0, 530), (0, 902)\n",
+ "2023-11-03 06:29:16,184:INFO:the tp solutions: (111.29894, 121.90106)\n",
+ "2023-11-03 06:29:16,185:INFO:the tn solutions: (1181.216076, 1220.4639240000004)\n",
+ "2023-11-03 06:29:16,187:INFO:intervals after: [(112, 121)], []\n",
+ "2023-11-03 06:29:16,188:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 06:29:16,190:INFO:intervals before: [(112, 121)], []\n",
+ "2023-11-03 06:29:16,191:INFO:the tp solutions: (518.3753319999998, 565.0646680000003)\n",
+ "2023-11-03 06:29:16,192:INFO:the tn solutions: (766.6981959999999, 784.741804)\n",
+ "2023-11-03 06:29:16,194:INFO:intervals after: [], []\n",
+ "2023-11-03 06:29:16,196:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 06:29:16,198:INFO:intervals before: [], []\n",
+ "2023-11-03 06:29:16,200:INFO:the tp solutions: (20.470765551173063, 28.952218416378013)\n",
+ "2023-11-03 06:29:16,201:INFO:the tn solutions: (1232.0122032291965, 1355.3280073234066)\n",
+ "2023-11-03 06:29:16,202:INFO:intervals after: [], []\n",
+ "2023-11-03 06:29:16,203:INFO:evaluating the tp and tn solution for acc and fm\n",
+ "2023-11-03 06:29:16,205:INFO:all negative bases - iteration finished\n",
+ "2023-11-03 06:29:16,206:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 06:29:16,207:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 06:29:16,208:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:29:16,209:INFO:final number of intervals: 0\n",
+ "2023-11-03 06:29:16,210:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "testset = {'p': 530, 'n': 902}\n",
+ "scores = {'acc': 0.92, 'sens': 0.22, 'spec': 0.86, 'f1p': 0.3, 'fm': 0.32}\n",
+ "\n",
+ "result = check_1_testset_no_kfold(testset=testset,\n",
+ " scores=scores,\n",
+ " eps=1e-2)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 06:29:16,234:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 06:29:16,236:INFO:calling the score check with scores {'acc': 0.954, 'sens': 0.934, 'spec': 0.985, 'ppv': 0.901}, uncertainty 0.001, p 1029 and n 3118\n",
+ "2023-11-03 06:29:16,238:INFO:checking the scores {'acc': 0.954, 'sens': 0.934, 'spec': 0.985, 'ppv': 0.901}\n",
+ "2023-11-03 06:29:16,240:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 06:29:16,242:INFO:intervals before: (0, 1029), (0, 3118)\n",
+ "2023-11-03 06:29:16,243:INFO:the tp solutions: (960.054942, 962.1170580000002)\n",
+ "2023-11-03 06:29:16,244:INFO:the tn solutions: (2989.965647999999, 3000.3383520000007)\n",
+ "2023-11-03 06:29:16,246:INFO:intervals after: [(961, 962)], [(2990, 3000)]\n",
+ "2023-11-03 06:29:16,248:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 06:29:16,249:INFO:intervals before: [(961, 962)], [(2990, 3000)]\n",
+ "2023-11-03 06:29:16,250:INFO:the tp solutions: (877.7284699999991, 892.2875300000005)\n",
+ "2023-11-03 06:29:16,252:INFO:the tn solutions: (3068.105764, 3074.354236)\n",
+ "2023-11-03 06:29:16,254:INFO:intervals after: [], []\n",
+ "2023-11-03 06:29:16,255:INFO:evaluating the tp and tn solution for acc and ppv\n",
+ "2023-11-03 06:29:16,257:INFO:intervals before: [], []\n",
+ "2023-11-03 06:29:16,258:INFO:the tp solutions: (933.6679509487356, 949.8052939947056)\n",
+ "2023-11-03 06:29:16,259:INFO:the tn solutions: (2988.3769250334417, 3040.8179565580203)\n",
+ "2023-11-03 06:29:16,262:INFO:intervals after: [], []\n",
+ "2023-11-03 06:29:16,264:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 06:29:16,265:INFO:intervals before: [], []\n",
+ "2023-11-03 06:29:16,266:INFO:the tp solutions: (960.054942, 962.1170580000002)\n",
+ "2023-11-03 06:29:16,268:INFO:the tn solutions: (3068.105764, 3074.354236)\n",
+ "2023-11-03 06:29:16,270:INFO:intervals after: [], []\n",
+ "2023-11-03 06:29:16,271:INFO:evaluating the tp and tn solution for sens and ppv\n",
+ "2023-11-03 06:29:16,273:INFO:intervals before: [], []\n",
+ "2023-11-03 06:29:16,274:INFO:the tp solutions: (960.054942, 962.1170580000002)\n",
+ "2023-11-03 06:29:16,275:INFO:the tn solutions: (3002.3483691722586, 3022.4720549935846)\n",
+ "2023-11-03 06:29:16,276:INFO:intervals after: [], []\n",
+ "2023-11-03 06:29:16,277:INFO:evaluating the tp and tn solution for spec and ppv\n",
+ "2023-11-03 06:29:16,279:INFO:intervals before: [], []\n",
+ "2023-11-03 06:29:16,280:INFO:the tp solutions: (392.8031470217782, 459.241011658118)\n",
+ "2023-11-03 06:29:16,281:INFO:the tn solutions: (3068.105764, 3074.354236)\n",
+ "2023-11-03 06:29:16,283:INFO:intervals after: [], []\n",
+ "2023-11-03 06:29:16,284:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 06:29:16,285:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 06:29:16,286:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:29:16,286:INFO:final number of intervals: 0\n",
+ "2023-11-03 06:29:16,287:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_testset_no_kfold(\n",
+ " scores={'acc': 0.954, 'sens': 0.934, 'spec': 0.985, 'ppv': 0.901},\n",
+ " eps=1e-3,\n",
+ " testset={'name': 'common_datasets.ADA'}\n",
+ " )\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "({'p': 80, 'n': 257},\n",
+ " {'acc': 0.65, 'sens': 0.54, 'spec': 0.68, 'f1p': 0.42, 'fm': 0.43})"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "problem, scores = generate_problem_and_scores(rounding_decimals=2,\n",
+ " score_subset=['acc', 'sens', 'spec',\n",
+ " 'f1p', 'fm'])\n",
+ "problem, scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 06:29:16,337:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 06:29:16,339:INFO:calling the score check with scores {'acc': 0.65, 'sens': 0.54, 'spec': 0.68, 'f1p': 0.42, 'fm': 0.43}, uncertainty 0.01, p 80 and n 257\n",
+ "2023-11-03 06:29:16,340:INFO:checking the scores {'acc': 0.65, 'sens': 0.54, 'spec': 0.68, 'f1p': 0.42, 'fm': 0.43}\n",
+ "2023-11-03 06:29:16,341:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 06:29:16,343:INFO:intervals before: (0, 80), (0, 257)\n",
+ "2023-11-03 06:29:16,344:INFO:the tp solutions: (42.39984, 44.00016000000001)\n",
+ "2023-11-03 06:29:16,350:INFO:the tn solutions: (171.67916599999995, 180.02083400000004)\n",
+ "2023-11-03 06:29:16,351:INFO:intervals after: [(43, 44)], [(172, 180)]\n",
+ "2023-11-03 06:29:16,353:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 06:29:16,355:INFO:intervals before: [(43, 44)], [(172, 180)]\n",
+ "2023-11-03 06:29:16,357:INFO:the tp solutions: (38.34881199999995, 50.23118800000003)\n",
+ "2023-11-03 06:29:16,358:INFO:the tn solutions: (172.189486, 177.33051400000002)\n",
+ "2023-11-03 06:29:16,362:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,364:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 06:29:16,366:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,367:INFO:the tp solutions: (39.81113157357772, 45.76168027023603)\n",
+ "2023-11-03 06:29:16,369:INFO:the tn solutions: (161.69999706666417, 191.4427153241345)\n",
+ "2023-11-03 06:29:16,370:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,372:INFO:evaluating the tp and tn solution for acc and fm\n",
+ "2023-11-03 06:29:16,375:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,377:INFO:the tp solutions: (-14.87848661500831, -11.429044835794995) | (40.324907235794996, 45.15062421500831)\n",
+ "2023-11-03 06:29:16,379:INFO:the tn solutions: (170.52870178499165, 182.095766764205) | (226.404226835155, 237.97129181436833)\n",
+ "2023-11-03 06:29:16,380:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,381:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 06:29:16,383:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,384:INFO:the tp solutions: (42.39984, 44.00016000000001)\n",
+ "2023-11-03 06:29:16,385:INFO:the tn solutions: (172.189486, 177.33051400000002)\n",
+ "2023-11-03 06:29:16,393:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,395:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 06:29:16,397:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,400:INFO:the tp solutions: (42.39984, 44.00016000000001)\n",
+ "2023-11-03 06:29:16,401:INFO:the tn solutions: (157.09893349407676, 192.75984468294968)\n",
+ "2023-11-03 06:29:16,402:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,405:INFO:evaluating the tp and tn solution for sens and fm\n",
+ "2023-11-03 06:29:16,407:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,408:INFO:the tp solutions: (42.39984, 44.00016000000001)\n",
+ "2023-11-03 06:29:16,411:INFO:the tn solutions: (147.7954166561049, 202.96298930938647)\n",
+ "2023-11-03 06:29:16,412:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,414:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 06:29:16,415:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,417:INFO:the tp solutions: (41.17238212343633, 45.13945281524434)\n",
+ "2023-11-03 06:29:16,418:INFO:the tn solutions: (172.189486, 177.33051400000002)\n",
+ "2023-11-03 06:29:16,419:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,420:INFO:evaluating the tp and tn solution for spec and fm\n",
+ "2023-11-03 06:29:16,421:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,422:INFO:the tp solutions: (-29.669137693676408, -26.872724954638574) | (41.32065615463858, 44.80520649367641)\n",
+ "2023-11-03 06:29:16,424:INFO:the tn solutions: (172.189486, 177.33051400000002)\n",
+ "2023-11-03 06:29:16,425:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,427:INFO:evaluating the tp and tn solution for f1p and fm\n",
+ "2023-11-03 06:29:16,430:INFO:intervals before: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,432:INFO:the tp solutions: (20.339541998791276, 51.48691044561233)\n",
+ "2023-11-03 06:29:16,433:INFO:the tn solutions: (109.3333440897702, 309.2883177266594)\n",
+ "2023-11-03 06:29:16,435:INFO:intervals after: [(43, 44)], [(173, 177)]\n",
+ "2023-11-03 06:29:16,436:INFO:testing acc, feasible tptn pairs: 2\n",
+ "2023-11-03 06:29:16,437:INFO:testing sens, feasible tptn pairs: 2\n",
+ "2023-11-03 06:29:16,439:INFO:testing spec, feasible tptn pairs: 2\n",
+ "2023-11-03 06:29:16,440:INFO:testing f1p, feasible tptn pairs: 2\n",
+ "2023-11-03 06:29:16,441:INFO:testing fm, feasible tptn pairs: 2\n",
+ "2023-11-03 06:29:16,443:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:29:16,444:INFO:final number of intervals: 2\n",
+ "2023-11-03 06:29:16,445:INFO:final number of pairs: 9\n"
+ ]
+ }
+ ],
+ "source": [
+ "result = check_1_testset_no_kfold(scores=scores,\n",
+ " eps=1e-2,\n",
+ " testset=problem)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/binary/01_check_1_dataset_kfold_som.ipynb b/notebooks/illustration/binary/01_check_1_dataset_kfold_som.ipynb
new file mode 100644
index 0000000..8b50627
--- /dev/null
+++ b/notebooks/illustration/binary/01_check_1_dataset_kfold_som.ipynb
@@ -0,0 +1,370 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.binary import check_1_dataset_kfold_som\n",
+ "from mlscorecheck.aggregated import Dataset, Evaluation, generate_evaluation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "({'dataset': {'p': 53, 'n': 17},\n",
+ " 'folding': {'n_folds': 9, 'n_repeats': 2, 'strategy': 'stratified_sklearn'},\n",
+ " 'fold_score_bounds': None,\n",
+ " 'aggregation': 'mos'},\n",
+ " {'acc': 0.416, 'sens': 0.38, 'spec': 0.5, 'bacc': 0.44})"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset, scores = generate_evaluation(return_scores=True, rounding_decimals=3)\n",
+ "dataset, scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 06:28:59,468:INFO:checking the scores {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667, 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
+ "2023-11-03 06:28:59,470:INFO:evaluating the tp and tn solution for spec and npv\n",
+ "2023-11-03 06:28:59,471:INFO:intervals before: (0, 612), (0, 684)\n",
+ "2023-11-03 06:28:59,472:INFO:the tp solutions: (451.13017565420745, 458.4474606321096)\n",
+ "2023-11-03 06:28:59,473:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,475:INFO:intervals after: [(452, 458)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,476:INFO:evaluating the tp and tn solution for spec and ppv\n",
+ "2023-11-03 06:28:59,477:INFO:intervals before: [(452, 458)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,478:INFO:the tp solutions: (451.4455006459121, 458.2951263885206)\n",
+ "2023-11-03 06:28:59,482:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,483:INFO:intervals after: [(452, 458)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,484:INFO:evaluating the tp and tn solution for spec and bacc\n",
+ "2023-11-03 06:28:59,486:INFO:intervals before: [(452, 458)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,486:INFO:the tp solutions: (453.4883279999998, 457.16767200000004)\n",
+ "2023-11-03 06:28:59,487:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,488:INFO:intervals after: [(454, 457)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,489:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 06:28:59,490:INFO:intervals before: [(454, 457)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,491:INFO:the tp solutions: (453.43302310684874, 456.17518747616583)\n",
+ "2023-11-03 06:28:59,494:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,495:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,496:INFO:evaluating the tp and tn solution for spec and fm\n",
+ "2023-11-03 06:28:59,498:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,499:INFO:the tp solutions: (-152.34158237393794, -150.5769147898986) | (453.46219709389857, 456.090284069938)\n",
+ "2023-11-03 06:28:59,500:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,501:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,502:INFO:evaluating the tp and tn solution for npv and ppv\n",
+ "2023-11-03 06:28:59,503:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,504:INFO:the tp solutions: (449.7868395395296, 459.8031631145148)\n",
+ "2023-11-03 06:28:59,507:INFO:the tn solutions: (451.7913140724638, 462.1800665561134)\n",
+ "2023-11-03 06:28:59,508:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,509:INFO:evaluating the tp and tn solution for npv and bacc\n",
+ "2023-11-03 06:28:59,510:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,512:INFO:the tp solutions: (443.7154761128823, 465.2752188090839)\n",
+ "2023-11-03 06:28:59,514:INFO:the tn solutions: (452.9617561982833, 462.89386265812294)\n",
+ "2023-11-03 06:28:59,515:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,516:INFO:evaluating the tp and tn solution for npv and f1p\n",
+ "2023-11-03 06:28:59,518:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,519:INFO:the tp solutions: (440.147036110388, 469.8382788367301)\n",
+ "2023-11-03 06:28:59,520:INFO:the tn solutions: (442.00190657831143, 472.519981444259)\n",
+ "2023-11-03 06:28:59,522:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,523:INFO:evaluating the tp and tn solution for npv and fm\n",
+ "2023-11-03 06:28:59,525:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,526:INFO:the tp solutions: (394.3507743984165, 792.5978000014203)\n",
+ "2023-11-03 06:28:59,527:INFO:the tn solutions: (-527.6344214333369, 642.5102448871437)\n",
+ "2023-11-03 06:28:59,528:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,532:INFO:evaluating the tp and tn solution for ppv and bacc\n",
+ "2023-11-03 06:28:59,533:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,534:INFO:the tp solutions: (450.4921584429077, 460.9834496704991)\n",
+ "2023-11-03 06:28:59,535:INFO:the tn solutions: (444.46160710915257, 468.645191592148)\n",
+ "2023-11-03 06:28:59,536:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,537:INFO:evaluating the tp and tn solution for ppv and f1p\n",
+ "2023-11-03 06:28:59,539:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,540:INFO:the tp solutions: (451.30158006461744, 458.2976005032661)\n",
+ "2023-11-03 06:28:59,543:INFO:the tn solutions: (449.24903166602184, 464.72652816818044)\n",
+ "2023-11-03 06:28:59,545:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,546:INFO:evaluating the tp and tn solution for ppv and fm\n",
+ "2023-11-03 06:28:59,547:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,549:INFO:the tp solutions: (452.77437351751627, 456.729638591179)\n",
+ "2023-11-03 06:28:59,550:INFO:the tn solutions: (449.644434645183, 464.3373716283757)\n",
+ "2023-11-03 06:28:59,551:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,552:INFO:evaluating the tp and tn solution for bacc and f1p\n",
+ "2023-11-03 06:28:59,554:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,555:INFO:the tp solutions: (449.6000702850551, 458.43196680714374)\n",
+ "2023-11-03 06:28:59,556:INFO:the tn solutions: (444.57627428771997, 472.337195306036)\n",
+ "2023-11-03 06:28:59,557:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,558:INFO:evaluating the tp and tn solution for bacc and fm\n",
+ "2023-11-03 06:28:59,560:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,562:INFO:the tp solutions: (184.881771974585, 191.6700592910953) | (450.2788987340362, 457.82373989853176)\n",
+ "2023-11-03 06:28:59,563:INFO:the tn solutions: (452.7519076428173, 463.9258491796066) | (749.8034974913054, 760.9774390280948)\n",
+ "2023-11-03 06:28:59,564:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,565:INFO:evaluating the tp and tn solution for f1p and fm\n",
+ "2023-11-03 06:28:59,566:INFO:intervals before: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,567:INFO:the tp solutions: (389.8902833715994, 473.26546244312186)\n",
+ "2023-11-03 06:28:59,568:INFO:the tn solutions: (376.3573072940828, 624.3053618675568)\n",
+ "2023-11-03 06:28:59,570:INFO:intervals after: [(454, 456)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,571:INFO:testing spec, feasible tptn pairs: 3\n",
+ "2023-11-03 06:28:59,572:INFO:testing bacc, feasible tptn pairs: 3\n",
+ "2023-11-03 06:28:59,574:INFO:testing npv, feasible tptn pairs: 2\n",
+ "2023-11-03 06:28:59,575:INFO:testing ppv, feasible tptn pairs: 1\n",
+ "2023-11-03 06:28:59,576:INFO:testing f1p, feasible tptn pairs: 1\n",
+ "2023-11-03 06:28:59,576:INFO:testing fm, feasible tptn pairs: 1\n",
+ "2023-11-03 06:28:59,583:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:28:59,583:INFO:final number of intervals: 1\n",
+ "2023-11-03 06:28:59,584:INFO:final number of pairs: 1\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset = {'dataset_name': 'common_datasets.monk-2'}\n",
+ "folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}\n",
+ "scores = {'spec': 0.668, 'npv': 0.744, 'ppv': 0.667, 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
+ "result = check_1_dataset_kfold_som(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 06:28:59,604:INFO:checking the scores {'spec': 0.668, 'npv': 0.544, 'ppv': 0.667, 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
+ "2023-11-03 06:28:59,606:INFO:evaluating the tp and tn solution for spec and npv\n",
+ "2023-11-03 06:28:59,607:INFO:intervals before: (0, 612), (0, 684)\n",
+ "2023-11-03 06:28:59,609:INFO:the tp solutions: (224.67359151477578, 233.34537651102963)\n",
+ "2023-11-03 06:28:59,610:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,611:INFO:intervals after: [(225, 233)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,613:INFO:evaluating the tp and tn solution for spec and ppv\n",
+ "2023-11-03 06:28:59,614:INFO:intervals before: [(225, 233)], [(457, 457)]\n",
+ "2023-11-03 06:28:59,616:INFO:the tp solutions: (451.4455006459121, 458.2951263885206)\n",
+ "2023-11-03 06:28:59,617:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,618:INFO:intervals after: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,619:INFO:evaluating the tp and tn solution for spec and bacc\n",
+ "2023-11-03 06:28:59,621:INFO:intervals before: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,621:INFO:the tp solutions: (453.4883279999998, 457.16767200000004)\n",
+ "2023-11-03 06:28:59,628:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,630:INFO:intervals after: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,631:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 06:28:59,633:INFO:intervals before: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,634:INFO:the tp solutions: (453.43302310684874, 456.17518747616583)\n",
+ "2023-11-03 06:28:59,635:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,636:INFO:intervals after: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,637:INFO:evaluating the tp and tn solution for spec and fm\n",
+ "2023-11-03 06:28:59,639:INFO:intervals before: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,640:INFO:the tp solutions: (-152.34158237393794, -150.5769147898986) | (453.46219709389857, 456.090284069938)\n",
+ "2023-11-03 06:28:59,643:INFO:the tn solutions: (456.226632, 457.5973680000001)\n",
+ "2023-11-03 06:28:59,645:INFO:intervals after: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,646:INFO:evaluating the tp and tn solution for npv and ppv\n",
+ "2023-11-03 06:28:59,647:INFO:intervals before: [], [(457, 457)]\n",
+ "2023-11-03 06:28:59,651:INFO:the tp solutions: (61.67528439458414, 71.34842820524828)\n",
+ "2023-11-03 06:28:59,652:INFO:the tn solutions: (640.1989553115615, 661.6570656911333)\n",
+ "2023-11-03 06:28:59,653:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,654:INFO:evaluating the tp and tn solution for npv and bacc\n",
+ "2023-11-03 06:28:59,656:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,657:INFO:the tp solutions: (-3449.3578113967096, -2845.2966863250767)\n",
+ "2023-11-03 06:28:59,658:INFO:the tn solutions: (4174.848806456365, 4788.330979831333)\n",
+ "2023-11-03 06:28:59,660:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,660:INFO:evaluating the tp and tn solution for npv and f1p\n",
+ "2023-11-03 06:28:59,662:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,663:INFO:the tp solutions: (848.1226422631049, 888.4347620980816)\n",
+ "2023-11-03 06:28:59,664:INFO:the tn solutions: (-315.46432513389715, -295.5981674463546)\n",
+ "2023-11-03 06:28:59,665:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,666:INFO:evaluating the tp and tn solution for npv and fm\n",
+ "2023-11-03 06:28:59,668:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,670:INFO:the tp solutions: (12.846926170011079, 30.911087774236353) | (632.0777110753301, 654.6422058656685)\n",
+ "2023-11-03 06:28:59,671:INFO:the tn solutions: (-51.077339859078506, -23.855818920884683) | (686.3871737976272, 721.9244118372484)\n",
+ "2023-11-03 06:28:59,672:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,674:INFO:evaluating the tp and tn solution for ppv and bacc\n",
+ "2023-11-03 06:28:59,676:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,678:INFO:the tp solutions: (450.4921584429077, 460.9834496704991)\n",
+ "2023-11-03 06:28:59,680:INFO:the tn solutions: (444.46160710915257, 468.645191592148)\n",
+ "2023-11-03 06:28:59,681:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,682:INFO:evaluating the tp and tn solution for ppv and f1p\n",
+ "2023-11-03 06:28:59,683:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,683:INFO:the tp solutions: (451.30158006461744, 458.2976005032661)\n",
+ "2023-11-03 06:28:59,684:INFO:the tn solutions: (449.24903166602184, 464.72652816818044)\n",
+ "2023-11-03 06:28:59,685:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,686:INFO:evaluating the tp and tn solution for ppv and fm\n",
+ "2023-11-03 06:28:59,688:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,689:INFO:the tp solutions: (452.77437351751627, 456.729638591179)\n",
+ "2023-11-03 06:28:59,690:INFO:the tn solutions: (449.644434645183, 464.3373716283757)\n",
+ "2023-11-03 06:28:59,690:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,691:INFO:evaluating the tp and tn solution for bacc and f1p\n",
+ "2023-11-03 06:28:59,693:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,694:INFO:the tp solutions: (449.6000702850551, 458.43196680714374)\n",
+ "2023-11-03 06:28:59,695:INFO:the tn solutions: (444.57627428771997, 472.337195306036)\n",
+ "2023-11-03 06:28:59,696:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,697:INFO:evaluating the tp and tn solution for bacc and fm\n",
+ "2023-11-03 06:28:59,699:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,700:INFO:the tp solutions: (184.881771974585, 191.6700592910953) | (450.2788987340362, 457.82373989853176)\n",
+ "2023-11-03 06:28:59,701:INFO:the tn solutions: (452.7519076428173, 463.9258491796066) | (749.8034974913054, 760.9774390280948)\n",
+ "2023-11-03 06:28:59,702:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,704:INFO:evaluating the tp and tn solution for f1p and fm\n",
+ "2023-11-03 06:28:59,706:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,707:INFO:the tp solutions: (389.8902833715994, 473.26546244312186)\n",
+ "2023-11-03 06:28:59,708:INFO:the tn solutions: (376.3573072940828, 624.3053618675568)\n",
+ "2023-11-03 06:28:59,709:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,710:INFO:testing spec, feasible tptn pairs: 0\n",
+ "2023-11-03 06:28:59,712:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 06:28:59,714:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:28:59,715:INFO:final number of intervals: 0\n",
+ "2023-11-03 06:28:59,716:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset = {'dataset_name': 'common_datasets.monk-2'}\n",
+ "folding = {'n_folds': 4, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}\n",
+ "scores = {'spec': 0.668, 'npv': 0.544, 'ppv': 0.667, 'bacc': 0.706, 'f1p': 0.703, 'fm': 0.704}\n",
+ "result = check_1_dataset_kfold_som(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 06:28:59,754:INFO:checking the scores {'acc': 0.428, 'npv': 0.392, 'bacc': 0.442, 'f1p': 0.391}\n",
+ "2023-11-03 06:28:59,757:INFO:evaluating the tp and tn solution for acc and npv\n",
+ "2023-11-03 06:28:59,759:INFO:intervals before: (0, 10), (0, 20)\n",
+ "2023-11-03 06:28:59,760:INFO:the tp solutions: (17.53160804333868, 18.464920278322932)\n",
+ "2023-11-03 06:28:59,763:INFO:the tn solutions: (-5.270843006972088, -5.039728262417209)\n",
+ "2023-11-03 06:28:59,765:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,766:INFO:evaluating the tp and tn solution for acc and bacc\n",
+ "2023-11-03 06:28:59,767:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,769:INFO:the tp solutions: (4.769860000000006, 4.910139999999998)\n",
+ "2023-11-03 06:28:59,770:INFO:the tn solutions: (7.899800000000002, 8.100199999999994)\n",
+ "2023-11-03 06:28:59,770:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,771:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 06:28:59,773:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,774:INFO:the tp solutions: (5.47591839052987, 5.541579002003296)\n",
+ "2023-11-03 06:28:59,774:INFO:the tn solutions: (7.2251654419821625, 7.437795041990264)\n",
+ "2023-11-03 06:28:59,775:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,776:INFO:evaluating the tp and tn solution for npv and bacc\n",
+ "2023-11-03 06:28:59,777:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,778:INFO:the tp solutions: (8.156982077324718, 8.420191328852582)\n",
+ "2023-11-03 06:28:59,781:INFO:the tn solutions: (1.0779174034529388, 1.1297477937232172)\n",
+ "2023-11-03 06:28:59,782:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,783:INFO:evaluating the tp and tn solution for npv and f1p\n",
+ "2023-11-03 06:28:59,785:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,785:INFO:the tp solutions: (6.725966043917559, 6.848200935967899)\n",
+ "2023-11-03 06:28:59,786:INFO:the tn solutions: (2.0399370054806383, 2.1037139445786694)\n",
+ "2023-11-03 06:28:59,787:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,788:INFO:evaluating the tp and tn solution for bacc and f1p\n",
+ "2023-11-03 06:28:59,789:INFO:intervals before: [], []\n",
+ "2023-11-03 06:28:59,790:INFO:the tp solutions: (5.770011590470432, 5.8801108747879205)\n",
+ "2023-11-03 06:28:59,790:INFO:the tn solutions: (5.799251089558401, 6.263087449471712)\n",
+ "2023-11-03 06:28:59,791:INFO:intervals after: [], []\n",
+ "2023-11-03 06:28:59,792:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 06:28:59,795:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 06:28:59,796:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:28:59,798:INFO:final number of intervals: 0\n",
+ "2023-11-03 06:28:59,799:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset = {'p': 10, 'n': 20}\n",
+ "folding = {'n_folds': 5, 'n_repeats': 1}\n",
+ "scores = {'acc': 0.428, 'npv': 0.392, 'bacc': 0.442, 'f1p': 0.391}\n",
+ "\n",
+ "result = check_1_dataset_kfold_som(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/02_check_1_dataset_known_folds_mos.ipynb b/notebooks/illustration/binary/02_check_1_dataset_known_folds_mos.ipynb
similarity index 69%
rename from notebooks/illustration/02_check_1_dataset_known_folds_mos.ipynb
rename to notebooks/illustration/binary/02_check_1_dataset_known_folds_mos.ipynb
index 228a538..0089920 100644
--- a/notebooks/illustration/02_check_1_dataset_known_folds_mos.ipynb
+++ b/notebooks/illustration/binary/02_check_1_dataset_known_folds_mos.ipynb
@@ -2,16 +2,16 @@
"cells": [
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.check import check_1_dataset_known_folds_mos_scores"
+ "from mlscorecheck.check.binary import check_1_dataset_known_folds_mos"
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 6,
"metadata": {},
"outputs": [
{
@@ -22,7 +22,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73370e5df8d04dc1ac3e0726173e9d46-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73370e5df8d04dc1ac3e0726173e9d46-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/92066f26c3b745b4a934509f1143b9db-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/92066f26c3b745b4a934509f1143b9db-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -36,37 +36,38 @@
"Cgl0003I 0 fixed, 9 tightened bounds, 0 strengthened rows, 0 substitutions\n",
"Cgl0003I 0 fixed, 2 tightened bounds, 0 strengthened rows, 0 substitutions\n",
"Cgl0004I processed model has 3 rows, 4 columns (4 integer (0 of which binary)) and 10 elements\n",
- "Cbc0012I Integer solution of 0 found by DiveCoefficient after 24 iterations and 0 nodes (0.00 seconds)\n",
- "Cbc0013I At root node, 0 cuts changed objective from 0 to 0 in 12 passes\n",
- "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 4 column cuts (4 active) in 0.000 seconds - new frequency is 1\n",
- "Cbc0014I Cut generator 1 (Gomory) - 25 row cuts average 3.1 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is 1\n",
+ "Cbc0012I Integer solution of 0 found by DiveCoefficient after 97 iterations and 0 nodes (0.01 seconds)\n",
+ "Cbc0031I 3 added rows had average density of 3.3333333\n",
+ "Cbc0013I At root node, 3 cuts changed objective from 0 to 0 in 100 passes\n",
+ "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 178 row cuts average 3.9 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 6 (TwoMirCuts) - 10 row cuts average 2.6 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0001I Search completed - best objective 0, took 24 iterations and 0 nodes (0.00 seconds)\n",
+ "Cbc0014I Cut generator 6 (TwoMirCuts) - 5 row cuts average 2.4 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0001I Search completed - best objective 0, took 97 iterations and 0 nodes (0.01 seconds)\n",
"Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
"Cuts at root node changed objective from 0 to 0\n",
- "Probing was tried 12 times and created 4 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "Gomory was tried 12 times and created 25 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "Knapsack was tried 12 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "Clique was tried 12 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "MixedIntegerRounding2 was tried 12 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "FlowCover was tried 12 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "TwoMirCuts was tried 12 times and created 10 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Probing was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Gomory was tried 100 times and created 178 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Knapsack was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Clique was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 100 times and created 5 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
"ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
"\n",
"Result - Optimal solution found\n",
"\n",
"Objective value: 0.00000000\n",
"Enumerated nodes: 0\n",
- "Total iterations: 24\n",
- "Time (CPU seconds): 0.00\n",
- "Time (Wallclock seconds): 0.00\n",
+ "Total iterations: 97\n",
+ "Time (CPU seconds): 0.01\n",
+ "Time (Wallclock seconds): 0.01\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n"
]
},
@@ -76,7 +77,7 @@
"False"
]
},
- "execution_count": 2,
+ "execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
@@ -85,16 +86,16 @@
"dataset = {'p': 126, 'n': 131}\n",
"folding = {'folds': [{'p': 52, 'n': 94}, {'p': 74, 'n': 37}]}\n",
"scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}\n",
- "result = check_1_dataset_known_folds_mos_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_1_dataset_known_folds_mos(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": 7,
"metadata": {},
"outputs": [
{
@@ -103,7 +104,7 @@
"'feasible'"
]
},
- "execution_count": 3,
+ "execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
@@ -114,7 +115,7 @@
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 8,
"metadata": {},
"outputs": [
{
@@ -122,7 +123,7 @@
"text/plain": [
"{'evaluations': [{'folds': {'folds': [{'fold': {'p': 52,\n",
" 'n': 94,\n",
- " 'identifier': 'txehx',\n",
+ " 'identifier': 'ercuu',\n",
" 'tp': 49.0,\n",
" 'tn': 13.0},\n",
" 'scores': {'acc': 0.4246575342465753,\n",
@@ -132,7 +133,7 @@
" 'bounds_flag': True},\n",
" {'fold': {'p': 74,\n",
" 'n': 37,\n",
- " 'identifier': 'damfo',\n",
+ " 'identifier': 'lptyi',\n",
" 'tp': 44.0,\n",
" 'tn': 36.0},\n",
" 'scores': {'acc': 0.7207207207207207,\n",
@@ -147,7 +148,7 @@
" 'score_bounds': None,\n",
" 'bounds_flag': {'folds': [{'fold': {'p': 52,\n",
" 'n': 94,\n",
- " 'identifier': 'txehx',\n",
+ " 'identifier': 'ercuu',\n",
" 'tp': 49.0,\n",
" 'tn': 13.0},\n",
" 'scores': {'acc': 0.4246575342465753,\n",
@@ -157,7 +158,7 @@
" 'bounds_flag': True},\n",
" {'fold': {'p': 74,\n",
" 'n': 37,\n",
- " 'identifier': 'damfo',\n",
+ " 'identifier': 'lptyi',\n",
" 'tp': 44.0,\n",
" 'tn': 36.0},\n",
" 'scores': {'acc': 0.7207207207207207,\n",
@@ -169,7 +170,7 @@
" 'bounds_flag': True}"
]
},
- "execution_count": 4,
+ "execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
@@ -180,7 +181,7 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 9,
"metadata": {},
"outputs": [
{
@@ -191,7 +192,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a20a9c211004a378101d618439cc6ba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a20a9c211004a378101d618439cc6ba-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c25d6bb5ba95490a9a104c1e1d65d94b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c25d6bb5ba95490a9a104c1e1d65d94b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -212,7 +213,7 @@
"Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 6 (TwoMirCuts) - 12 row cuts average 2.3 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.01 seconds)\n",
+ "Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.00 seconds)\n",
"Cbc0001I Search completed - best objective 1e+50, took 77 iterations and 4 nodes (0.01 seconds)\n",
"Cbc0032I Strong branching done 16 times (23 iterations), fathomed 3 nodes and fixed 1 variables\n",
"Cbc0035I Maximum depth 1, 0 variables fixed on reduced cost\n",
@@ -245,7 +246,7 @@
"True"
]
},
- "execution_count": 5,
+ "execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
@@ -254,16 +255,16 @@
"dataset = {'p': 126, 'n': 131}\n",
"folding = {'folds': [{'p': 52, 'n': 94}, {'p': 74, 'n': 37}]}\n",
"scores = {'acc': 0.573, 'sens': 0.568, 'bacc': 0.662}\n",
- "result = check_1_dataset_known_folds_mos_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_1_dataset_known_folds_mos(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 10,
"metadata": {},
"outputs": [
{
@@ -274,7 +275,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/58ea6fb792f44257869adcc5799a5328-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/58ea6fb792f44257869adcc5799a5328-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dba90c9fd5214080af63a2f2f1e00508-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dba90c9fd5214080af63a2f2f1e00508-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -296,7 +297,7 @@
"True"
]
},
- "execution_count": 6,
+ "execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
@@ -305,38 +306,38 @@
"dataset = {'p': 398, 'n': 569}\n",
"folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}\n",
"scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6}\n",
- "result = check_1_dataset_known_folds_mos_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-2)\n",
+ "result = check_1_dataset_known_folds_mos(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-2)\n",
"result['inconsistency']"
]
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:38:18,485:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_0: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,488:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_0: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:18,489:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_1: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,490:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_1: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:18,491:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_2: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,492:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_2: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:18,492:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_3: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,494:INFO:common_datasets.glass_0_1_6_vs_2_ooa_0_3: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:18,495:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_0: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,496:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_0: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:18,497:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_1: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,498:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_1: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:18,499:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_2: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,500:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_2: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:18,501:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_3: adding lower bound 0.8 for acc\n",
- "2023-10-26 23:38:18,502:INFO:common_datasets.glass_0_1_6_vs_2_ooa_1_3: adding upper bound 1.0 for acc\n"
+ "2023-11-03 06:48:55,774:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_0: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,776:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_0: adding upper bound 1.0 for acc\n",
+ "2023-11-03 06:48:55,778:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_1: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,779:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_1: adding upper bound 1.0 for acc\n",
+ "2023-11-03 06:48:55,780:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_2: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,781:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_2: adding upper bound 1.0 for acc\n",
+ "2023-11-03 06:48:55,781:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_3: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,782:INFO:common_datasets.glass_0_1_6_vs_2_adb_0_3: adding upper bound 1.0 for acc\n",
+ "2023-11-03 06:48:55,783:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_0: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,783:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_0: adding upper bound 1.0 for acc\n",
+ "2023-11-03 06:48:55,784:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_1: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,785:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_1: adding upper bound 1.0 for acc\n",
+ "2023-11-03 06:48:55,786:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_2: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,786:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_2: adding upper bound 1.0 for acc\n",
+ "2023-11-03 06:48:55,787:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_3: adding lower bound 0.8 for acc\n",
+ "2023-11-03 06:48:55,789:INFO:common_datasets.glass_0_1_6_vs_2_adb_1_3: adding upper bound 1.0 for acc\n"
]
},
{
@@ -347,7 +348,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c5e80281b70c49c8ab5a42ca5afea3ae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c5e80281b70c49c8ab5a42ca5afea3ae-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9ed2fbdc772b4fa3984a4234b8f9d296-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9ed2fbdc772b4fa3984a4234b8f9d296-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 29 COLUMNS\n",
@@ -369,7 +370,7 @@
"True"
]
},
- "execution_count": 7,
+ "execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
@@ -380,12 +381,12 @@
"\n",
"scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6, 'bacc': 0.1, 'f1': 0.95}\n",
"\n",
- "result = check_1_dataset_known_folds_mos_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " fold_score_bounds={'acc': (0.8, 1.0)},\n",
- " scores=scores,\n",
- " eps=1e-2,\n",
- " numerical_tolerance=1e-6)\n",
+ "result = check_1_dataset_known_folds_mos(dataset=dataset,\n",
+ " folding=folding,\n",
+ " fold_score_bounds={'acc': (0.8, 1.0)},\n",
+ " scores=scores,\n",
+ " eps=1e-2,\n",
+ " numerical_tolerance=1e-6)\n",
"result['inconsistency']"
]
}
diff --git a/notebooks/illustration/03_check_1_dataset_unknown_folds_mos.ipynb b/notebooks/illustration/binary/03_check_1_dataset_unknown_folds_mos.ipynb
similarity index 84%
rename from notebooks/illustration/03_check_1_dataset_unknown_folds_mos.ipynb
rename to notebooks/illustration/binary/03_check_1_dataset_unknown_folds_mos.ipynb
index 010b0e9..53f0768 100644
--- a/notebooks/illustration/03_check_1_dataset_unknown_folds_mos.ipynb
+++ b/notebooks/illustration/binary/03_check_1_dataset_unknown_folds_mos.ipynb
@@ -2,16 +2,16 @@
"cells": [
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.check import check_1_dataset_unknown_folds_mos_scores"
+ "from mlscorecheck.check.binary import check_1_dataset_unknown_folds_mos"
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 6,
"metadata": {},
"outputs": [
{
@@ -22,7 +22,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee40e7bf89814022b0d4bc1c9e4e720e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee40e7bf89814022b0d4bc1c9e4e720e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e5ead28b04f4c6bb0f1a79ce53e5336-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e5ead28b04f4c6bb0f1a79ce53e5336-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -43,7 +43,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/43879cfc884849c49e8e463882ff2de7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/43879cfc884849c49e8e463882ff2de7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/90f5b9b6e3554c9bbeca886ed8732065-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/90f5b9b6e3554c9bbeca886ed8732065-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -63,7 +63,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7473d840fff7483993c55d736d7dcc11-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7473d840fff7483993c55d736d7dcc11-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/08764cefdf5942c9bfc7d9e467caba89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/08764cefdf5942c9bfc7d9e467caba89-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -113,7 +113,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1749e11eadac45a89ddfd38c030d9c3a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1749e11eadac45a89ddfd38c030d9c3a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6e382ec918f44865b3411a53df667931-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6e382ec918f44865b3411a53df667931-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -134,7 +134,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6bc67e384bd842cfb5ffd6370d65431d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6bc67e384bd842cfb5ffd6370d65431d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/da79df8b43a7423791598d70e0540f6b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/da79df8b43a7423791598d70e0540f6b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -178,7 +178,7 @@
"False"
]
},
- "execution_count": 2,
+ "execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
@@ -187,13 +187,313 @@
"dataset = {'p': 126, 'n': 131}\n",
"folding = {'n_folds': 2, 'n_repeats': 1}\n",
"scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.662}\n",
- "result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_1_dataset_unknown_folds_mos(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "dict_keys(['details', 'inconsistency'])"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result.keys()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[{'folds': [{'p': 1, 'n': 128, 'identifier': 'zugsza_f0_k0_r0'},\n",
+ " {'p': 125, 'n': 3, 'identifier': 'zugsza_f1_k0_r0'}],\n",
+ " 'details': {'inconsistency': True,\n",
+ " 'lp_status': 'infeasible',\n",
+ " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 1,\n",
+ " 'n': 128,\n",
+ " 'identifier': 'zugsza_f0_k0_r0',\n",
+ " 'tp': 0.538002,\n",
+ " 'tn': 18.121093},\n",
+ " 'scores': {'acc': 0.14464414728682168,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.33978651953125},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 125,\n",
+ " 'n': 3,\n",
+ " 'identifier': 'zugsza_f1_k0_r0',\n",
+ " 'tp': 125.0,\n",
+ " 'tn': 2.9172929},\n",
+ " 'scores': {'acc': 0.99935385078125,\n",
+ " 'sens': 1.0,\n",
+ " 'bacc': 0.9862154833333333},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True},\n",
+ " 'scores': {'acc': 0.5719989990340358,\n",
+ " 'sens': 0.769001,\n",
+ " 'bacc': 0.6630010014322917},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': {'folds': [{'fold': {'p': 1,\n",
+ " 'n': 128,\n",
+ " 'identifier': 'zugsza_f0_k0_r0',\n",
+ " 'tp': 0.538002,\n",
+ " 'tn': 18.121093},\n",
+ " 'scores': {'acc': 0.14464414728682168,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.33978651953125},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 125,\n",
+ " 'n': 3,\n",
+ " 'identifier': 'zugsza_f1_k0_r0',\n",
+ " 'tp': 125.0,\n",
+ " 'tn': 2.9172929},\n",
+ " 'scores': {'acc': 0.99935385078125,\n",
+ " 'sens': 1.0,\n",
+ " 'bacc': 0.9862154833333333},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True}}],\n",
+ " 'bounds_flag': True}},\n",
+ " 'configuration_id': 0},\n",
+ " {'folds': [{'p': 2, 'n': 127, 'identifier': 'zugsza_f0_k1_r0'},\n",
+ " {'p': 124, 'n': 4, 'identifier': 'zugsza_f1_k1_r0'}],\n",
+ " 'details': {'inconsistency': True,\n",
+ " 'lp_status': 'infeasible',\n",
+ " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 2,\n",
+ " 'n': 127,\n",
+ " 'identifier': 'zugsza_f0_k1_r0',\n",
+ " 'tp': 1.076004,\n",
+ " 'tn': 17.59879},\n",
+ " 'scores': {'acc': 0.14476584496124031,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.3382875748031496},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 124,\n",
+ " 'n': 4,\n",
+ " 'identifier': 'zugsza_f1_k1_r0',\n",
+ " 'tp': 124.0,\n",
+ " 'tn': 3.9017154},\n",
+ " 'scores': {'acc': 0.9992321515625,\n",
+ " 'sens': 1.0,\n",
+ " 'bacc': 0.9877144250000001},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True},\n",
+ " 'scores': {'acc': 0.5719989982618702,\n",
+ " 'sens': 0.769001,\n",
+ " 'bacc': 0.6630009999015748},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': {'folds': [{'fold': {'p': 2,\n",
+ " 'n': 127,\n",
+ " 'identifier': 'zugsza_f0_k1_r0',\n",
+ " 'tp': 1.076004,\n",
+ " 'tn': 17.59879},\n",
+ " 'scores': {'acc': 0.14476584496124031,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.3382875748031496},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 124,\n",
+ " 'n': 4,\n",
+ " 'identifier': 'zugsza_f1_k1_r0',\n",
+ " 'tp': 124.0,\n",
+ " 'tn': 3.9017154},\n",
+ " 'scores': {'acc': 0.9992321515625,\n",
+ " 'sens': 1.0,\n",
+ " 'bacc': 0.9877144250000001},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True}}],\n",
+ " 'bounds_flag': True}},\n",
+ " 'configuration_id': 1},\n",
+ " {'folds': [{'p': 3, 'n': 126, 'identifier': 'zugsza_f0_k2_r0'},\n",
+ " {'p': 123, 'n': 5, 'identifier': 'zugsza_f1_k2_r0'}],\n",
+ " 'details': {'inconsistency': True,\n",
+ " 'lp_status': 'infeasible',\n",
+ " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 3,\n",
+ " 'n': 126,\n",
+ " 'identifier': 'zugsza_f0_k2_r0',\n",
+ " 'tp': 1.614006,\n",
+ " 'tn': 17.069944},\n",
+ " 'scores': {'acc': 0.14483682170542636,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.336738873015873},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 123,\n",
+ " 'n': 5,\n",
+ " 'identifier': 'zugsza_f1_k2_r0',\n",
+ " 'tp': 123.0,\n",
+ " 'tn': 4.8926313},\n",
+ " 'scores': {'acc': 0.99916118203125, 'sens': 1.0, 'bacc': 0.98926313},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True},\n",
+ " 'scores': {'acc': 0.5719990018683382,\n",
+ " 'sens': 0.769001,\n",
+ " 'bacc': 0.6630010015079365},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': {'folds': [{'fold': {'p': 3,\n",
+ " 'n': 126,\n",
+ " 'identifier': 'zugsza_f0_k2_r0',\n",
+ " 'tp': 1.614006,\n",
+ " 'tn': 17.069944},\n",
+ " 'scores': {'acc': 0.14483682170542636,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.336738873015873},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 123,\n",
+ " 'n': 5,\n",
+ " 'identifier': 'zugsza_f1_k2_r0',\n",
+ " 'tp': 123.0,\n",
+ " 'tn': 4.8926313},\n",
+ " 'scores': {'acc': 0.99916118203125, 'sens': 1.0, 'bacc': 0.98926313},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True}}],\n",
+ " 'bounds_flag': True}},\n",
+ " 'configuration_id': 2},\n",
+ " {'folds': [{'p': 4, 'n': 125, 'identifier': 'zugsza_f0_k3_r0'},\n",
+ " {'p': 122, 'n': 6, 'identifier': 'zugsza_f1_k3_r0'}],\n",
+ " 'details': {'inconsistency': True,\n",
+ " 'lp_status': 'infeasible',\n",
+ " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 4,\n",
+ " 'n': 125,\n",
+ " 'identifier': 'zugsza_f0_k3_r0',\n",
+ " 'tp': 2.152008,\n",
+ " 'tn': 16.534221},\n",
+ " 'scores': {'acc': 0.144854488372093,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.335137884},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 122,\n",
+ " 'n': 6,\n",
+ " 'identifier': 'zugsza_f1_k3_r0',\n",
+ " 'tp': 122.0,\n",
+ " 'tn': 5.8903694},\n",
+ " 'scores': {'acc': 0.9991435109375,\n",
+ " 'sens': 1.0,\n",
+ " 'bacc': 0.9908641166666666},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True},\n",
+ " 'scores': {'acc': 0.5719989996547965,\n",
+ " 'sens': 0.769001,\n",
+ " 'bacc': 0.6630010003333333},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': {'folds': [{'fold': {'p': 4,\n",
+ " 'n': 125,\n",
+ " 'identifier': 'zugsza_f0_k3_r0',\n",
+ " 'tp': 2.152008,\n",
+ " 'tn': 16.534221},\n",
+ " 'scores': {'acc': 0.144854488372093,\n",
+ " 'sens': 0.538002,\n",
+ " 'bacc': 0.335137884},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 122,\n",
+ " 'n': 6,\n",
+ " 'identifier': 'zugsza_f1_k3_r0',\n",
+ " 'tp': 122.0,\n",
+ " 'tn': 5.8903694},\n",
+ " 'scores': {'acc': 0.9991435109375,\n",
+ " 'sens': 1.0,\n",
+ " 'bacc': 0.9908641166666666},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True}}],\n",
+ " 'bounds_flag': True}},\n",
+ " 'configuration_id': 3},\n",
+ " {'folds': [{'p': 5, 'n': 124, 'identifier': 'zugsza_f0_k4_r0'},\n",
+ " {'p': 121, 'n': 7, 'identifier': 'zugsza_f1_k4_r0'}],\n",
+ " 'details': {'inconsistency': False,\n",
+ " 'lp_scores': {'acc': 0.5726138565891473,\n",
+ " 'sens': 0.7677685950413223,\n",
+ " 'bacc': 0.661246048672735},\n",
+ " 'lp_status': 'feasible',\n",
+ " 'lp_configuration_scores_match': True,\n",
+ " 'lp_configuration_bounds_match': True,\n",
+ " 'lp_configuration': {'evaluations': [{'folds': {'folds': [{'fold': {'p': 5,\n",
+ " 'n': 124,\n",
+ " 'identifier': 'zugsza_f0_k4_r0',\n",
+ " 'tp': 4.0,\n",
+ " 'tn': 49.0},\n",
+ " 'scores': {'acc': 0.4108527131782946,\n",
+ " 'sens': 0.8,\n",
+ " 'bacc': 0.5975806451612904},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 121,\n",
+ " 'n': 7,\n",
+ " 'identifier': 'zugsza_f1_k4_r0',\n",
+ " 'tp': 89.0,\n",
+ " 'tn': 5.0},\n",
+ " 'scores': {'acc': 0.734375,\n",
+ " 'sens': 0.7355371900826446,\n",
+ " 'bacc': 0.7249114521841794},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True},\n",
+ " 'scores': {'acc': 0.5726138565891473,\n",
+ " 'sens': 0.7677685950413223,\n",
+ " 'bacc': 0.661246048672735},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': {'folds': [{'fold': {'p': 5,\n",
+ " 'n': 124,\n",
+ " 'identifier': 'zugsza_f0_k4_r0',\n",
+ " 'tp': 4.0,\n",
+ " 'tn': 49.0},\n",
+ " 'scores': {'acc': 0.4108527131782946,\n",
+ " 'sens': 0.8,\n",
+ " 'bacc': 0.5975806451612904},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True},\n",
+ " {'fold': {'p': 121,\n",
+ " 'n': 7,\n",
+ " 'identifier': 'zugsza_f1_k4_r0',\n",
+ " 'tp': 89.0,\n",
+ " 'tn': 5.0},\n",
+ " 'scores': {'acc': 0.734375,\n",
+ " 'sens': 0.7355371900826446,\n",
+ " 'bacc': 0.7249114521841794},\n",
+ " 'score_bounds': None,\n",
+ " 'bounds_flag': True}],\n",
+ " 'bounds_flag': True}}],\n",
+ " 'bounds_flag': True}},\n",
+ " 'configuration_id': 4}]"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result['details']"
+ ]
+ },
{
"cell_type": "code",
"execution_count": 3,
@@ -207,7 +507,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/20dd81cc408a40b08b70188d841bdb3f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/20dd81cc408a40b08b70188d841bdb3f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a64bda5696f34ec889bb61703d8a9e6f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a64bda5696f34ec889bb61703d8a9e6f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -225,7 +525,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dff6b7554ff7438fa133a9c1efa7a6ce-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dff6b7554ff7438fa133a9c1efa7a6ce-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/47e881a9789d46eabbc8c84384811b7b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/47e881a9789d46eabbc8c84384811b7b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -243,7 +543,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/59d6aae601b44848b0f32a14871e2587-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/59d6aae601b44848b0f32a14871e2587-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bd533a9143774f628f95c5fb84bf9caa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bd533a9143774f628f95c5fb84bf9caa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -261,7 +561,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/43bb65636b914f21a10dabe1e8bc648b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/43bb65636b914f21a10dabe1e8bc648b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a64acd60d51a4fb19d30161e72582dad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a64acd60d51a4fb19d30161e72582dad-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -279,7 +579,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dbd3ee2f9a864a30a0c8af0cce2b1f28-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dbd3ee2f9a864a30a0c8af0cce2b1f28-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a2e6241f2744f6488d9f71ac8631833-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a2e6241f2744f6488d9f71ac8631833-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -297,7 +597,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1aedfa2f46a040f8a689556d7d67afca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1aedfa2f46a040f8a689556d7d67afca-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/490a29dca5ab4395b1bc7787aae91f1e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/490a29dca5ab4395b1bc7787aae91f1e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -315,7 +615,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/208b45147edc4dc590109ecf89b0a325-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/208b45147edc4dc590109ecf89b0a325-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fe8af1388e2d45d89eca0c1ee8f00d9c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fe8af1388e2d45d89eca0c1ee8f00d9c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -333,7 +633,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fdf4759f63284b3bbf7282553f83f5df-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fdf4759f63284b3bbf7282553f83f5df-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3659ab64aef04fed920ef1b27b6d9d99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3659ab64aef04fed920ef1b27b6d9d99-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -351,7 +651,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/103f8fd203104ecbab7072c9bcb224da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/103f8fd203104ecbab7072c9bcb224da-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d4f53bdc5bf04de0bdeada45b74c1018-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d4f53bdc5bf04de0bdeada45b74c1018-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -369,7 +669,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f5ad7710312c4e2ea618cfa14da1e4e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f5ad7710312c4e2ea618cfa14da1e4e9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cbac2968a885462eb17916b6e3948dc5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cbac2968a885462eb17916b6e3948dc5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -387,7 +687,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9cd4e7017604407b871cb7b6647fc14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9cd4e7017604407b871cb7b6647fc14-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db0021c0aaec4529bd40423e9811fb31-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db0021c0aaec4529bd40423e9811fb31-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -405,7 +705,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8821a36570d345baaf7e85c7a2c913ad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8821a36570d345baaf7e85c7a2c913ad-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fdca9b1099f34d4792ca6c9edafc6de1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fdca9b1099f34d4792ca6c9edafc6de1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -423,7 +723,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fba7b1cb10c14ecf9a43f706eb192f29-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fba7b1cb10c14ecf9a43f706eb192f29-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb669d5233fe44838e96c01f36a9e365-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb669d5233fe44838e96c01f36a9e365-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -441,7 +741,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/af7aa975b9e7472b85835c98d0326545-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/af7aa975b9e7472b85835c98d0326545-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0bb3fb2c482d4d428878a78acf55bed0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0bb3fb2c482d4d428878a78acf55bed0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -459,7 +759,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/65178c991c2d4caca58befb801dd3b39-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/65178c991c2d4caca58befb801dd3b39-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64c7ed7fd4bd4adbaecb8d1535b57aba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64c7ed7fd4bd4adbaecb8d1535b57aba-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -477,7 +777,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/085fbd28462a4e89b6a41c6c085d0b4b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/085fbd28462a4e89b6a41c6c085d0b4b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea1b904bfec84ab1b688b562fce97f81-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea1b904bfec84ab1b688b562fce97f81-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -495,7 +795,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4fbbfbea8b8b4866b83d46b1dc7256ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4fbbfbea8b8b4866b83d46b1dc7256ca-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/67ebc52be3ac45de832a1a012050f246-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/67ebc52be3ac45de832a1a012050f246-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -513,7 +813,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f5b2604981d34ee9bc7e5cf32e6ea618-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f5b2604981d34ee9bc7e5cf32e6ea618-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/133248800cda48d58d399939beacd828-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/133248800cda48d58d399939beacd828-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -531,7 +831,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b989dbc0715d4cc6aafad9ae8efadece-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b989dbc0715d4cc6aafad9ae8efadece-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fcc95c9962114e8f9ecf46c39c00d281-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fcc95c9962114e8f9ecf46c39c00d281-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -549,7 +849,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0015bb6cc97247b9b36dee39fe070d02-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0015bb6cc97247b9b36dee39fe070d02-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/edc97d52b5bc4d0a911d3dca969d5479-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/edc97d52b5bc4d0a911d3dca969d5479-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -567,7 +867,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/83225eb9f5a34cbba0a48d7617e34e89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/83225eb9f5a34cbba0a48d7617e34e89-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7b64c98bf69248bca4085476e4dab003-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7b64c98bf69248bca4085476e4dab003-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -585,7 +885,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1fb1abc603894baab80188fcdd6e92cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1fb1abc603894baab80188fcdd6e92cf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1b5b397e1e924d26b9fb4dd012b9f93f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1b5b397e1e924d26b9fb4dd012b9f93f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -603,7 +903,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b65104d4248948c1974c01ba1de1a4bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b65104d4248948c1974c01ba1de1a4bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/be3540a197aa4aa2875f8934e5b89ac7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/be3540a197aa4aa2875f8934e5b89ac7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -621,7 +921,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e15d00cef4c34ae4829906f9d98e6bbd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e15d00cef4c34ae4829906f9d98e6bbd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8d8edd9a2c0b40478632239c8699e3e5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8d8edd9a2c0b40478632239c8699e3e5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -639,7 +939,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ca16e255373d4388a135bb25a5df2e4e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ca16e255373d4388a135bb25a5df2e4e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87cd97c3da8b4a38b1828cd6d41e9979-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87cd97c3da8b4a38b1828cd6d41e9979-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -657,7 +957,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/acb36cbabd33464aad1a2974cc3f4d77-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/acb36cbabd33464aad1a2974cc3f4d77-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3826897ed3fe44d3929da36a57b6a85f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3826897ed3fe44d3929da36a57b6a85f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -675,7 +975,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/11db332e9bf4412691c63967d83e3130-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/11db332e9bf4412691c63967d83e3130-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e8bd38b8c8fb4440b0f7ae4d08620be8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e8bd38b8c8fb4440b0f7ae4d08620be8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -693,7 +993,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/65eda315835045338dc8211c60c7892f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/65eda315835045338dc8211c60c7892f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33f4b08c5fa548b1a1c245e892929ef2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33f4b08c5fa548b1a1c245e892929ef2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -711,7 +1011,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24bc3124ba3a4a238904b840172d614c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24bc3124ba3a4a238904b840172d614c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/447baa426acc459d902c88047c9a3fe3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/447baa426acc459d902c88047c9a3fe3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -729,7 +1029,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eb6154d3dd2f427d9c436571fc0a1dba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eb6154d3dd2f427d9c436571fc0a1dba-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/454eabb7648c48bfb4968bf2768ecc8e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/454eabb7648c48bfb4968bf2768ecc8e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -747,7 +1047,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2f7af0d78cdc4d70b7ac6f475dfbe0a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2f7af0d78cdc4d70b7ac6f475dfbe0a9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46a870dd87484d1fb903a87502a5b021-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46a870dd87484d1fb903a87502a5b021-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -765,7 +1065,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6ffe0f2702474f73be07f53803ca5759-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6ffe0f2702474f73be07f53803ca5759-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e9387287411f4a6e8bbc99b6e5ad37ba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e9387287411f4a6e8bbc99b6e5ad37ba-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -783,7 +1083,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e1aad625d2b54202b604d9f347e1b4bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e1aad625d2b54202b604d9f347e1b4bd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/29094f56edbf47db9fd0e5614dafcdfa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/29094f56edbf47db9fd0e5614dafcdfa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -801,7 +1101,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f0649e8ae4f643eb83e979140d6bb46d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f0649e8ae4f643eb83e979140d6bb46d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7eb92409645a47999086f2d3a40cc43d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7eb92409645a47999086f2d3a40cc43d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -819,7 +1119,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/91060ff6ea4540eeb6b2834c9a009995-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/91060ff6ea4540eeb6b2834c9a009995-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d286792cac41457cae0a64103f48d3a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d286792cac41457cae0a64103f48d3a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -837,7 +1137,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/093dc17b2fb04372ba4533420ff0594e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/093dc17b2fb04372ba4533420ff0594e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/caa988ce4f674a6fb2151e86f01cdb60-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/caa988ce4f674a6fb2151e86f01cdb60-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -855,7 +1155,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc09a20cfab041d796734f2610a48e54-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc09a20cfab041d796734f2610a48e54-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e52dd94fd31148fb945284c553d4e54f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e52dd94fd31148fb945284c553d4e54f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -873,7 +1173,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82bbd88591e74e289e663c74f64d1988-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82bbd88591e74e289e663c74f64d1988-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a38869296964899954ca85935233a9a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a38869296964899954ca85935233a9a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -891,7 +1191,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc73651c43104d5e8b63e40518b9083d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc73651c43104d5e8b63e40518b9083d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0af47e01889b42b095a6090d56ba6031-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0af47e01889b42b095a6090d56ba6031-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -909,7 +1209,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8738e608fb2d4c778602b006146a4c19-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8738e608fb2d4c778602b006146a4c19-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3e84a765b03048df99b53021465073ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3e84a765b03048df99b53021465073ef-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -927,7 +1227,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8961c21122724911a7684585e4198664-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8961c21122724911a7684585e4198664-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1f633bb99204c75bd7861ed7ac0f295-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1f633bb99204c75bd7861ed7ac0f295-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -945,7 +1245,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e01b386885a8476782afb1bfbed515f7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e01b386885a8476782afb1bfbed515f7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32ec42a7a4be4f38bbce7be9d39cb408-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32ec42a7a4be4f38bbce7be9d39cb408-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -963,7 +1263,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13cd24a5e6a048a99589fc2f08769619-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13cd24a5e6a048a99589fc2f08769619-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/246904d3b97b46bebf0eef58cf401794-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/246904d3b97b46bebf0eef58cf401794-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -981,7 +1281,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d0cc4c80aad4504a4d342318dd337c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d0cc4c80aad4504a4d342318dd337c5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/831185c77453421a87f29beadbac4a0d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/831185c77453421a87f29beadbac4a0d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -999,7 +1299,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24a0f3dbbb0e4c58a78de683d36f119a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24a0f3dbbb0e4c58a78de683d36f119a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/632d41cfb79f4a46af6f8be7579802f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/632d41cfb79f4a46af6f8be7579802f3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1017,7 +1317,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/01c17eea00de479e9bd440451f6350d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/01c17eea00de479e9bd440451f6350d0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/850352474d0348bf954a329e9198c0a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/850352474d0348bf954a329e9198c0a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1035,7 +1335,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cfd9866600244317b559ac701a0636f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cfd9866600244317b559ac701a0636f1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dbead456f9b74007bb1f31b5355e8d80-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dbead456f9b74007bb1f31b5355e8d80-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1053,7 +1353,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75cfad87b3814b8491ec207848e787a3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75cfad87b3814b8491ec207848e787a3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef59a59d9b224b1a85fbff34f17b7579-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef59a59d9b224b1a85fbff34f17b7579-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1071,7 +1371,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/79dd3306359842c49f679815bb019277-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/79dd3306359842c49f679815bb019277-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ce968fa9af294a73ad70a458375a353d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ce968fa9af294a73ad70a458375a353d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1089,7 +1389,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42b883198506483ba5c7a0fb88adbbeb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42b883198506483ba5c7a0fb88adbbeb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae55920041714cffbc606dc625be21c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae55920041714cffbc606dc625be21c5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1107,7 +1407,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5035fd53b89c4c909d45d88d03c116d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5035fd53b89c4c909d45d88d03c116d8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cc56e08740640deb9833eeae6d89315-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cc56e08740640deb9833eeae6d89315-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1125,7 +1425,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13087a8cce044b63a3d1a4e5cc90c4ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13087a8cce044b63a3d1a4e5cc90c4ee-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a12fcd8c5174466fbadc7fa9d3c5b942-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a12fcd8c5174466fbadc7fa9d3c5b942-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1143,7 +1443,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0dfff785eda647e3892506ea9fcd430e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0dfff785eda647e3892506ea9fcd430e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aac21500d76d460d9a3c171deae30b8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aac21500d76d460d9a3c171deae30b8f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1161,7 +1461,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b18c0ad6e95d4db7a93dccd2a00bbe9a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b18c0ad6e95d4db7a93dccd2a00bbe9a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dd115bf38e8746558809e3e7803b7a05-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dd115bf38e8746558809e3e7803b7a05-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1179,7 +1479,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/984af2c59770459a8f0dd749ce928f5e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/984af2c59770459a8f0dd749ce928f5e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ba602d5e696476fb613b30f215ecbdf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ba602d5e696476fb613b30f215ecbdf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1197,7 +1497,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c9aa5ada51334d0199e0b5fb47f48ce5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c9aa5ada51334d0199e0b5fb47f48ce5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6194ab2488cf48dfb3d2c81974a609e0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6194ab2488cf48dfb3d2c81974a609e0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1215,7 +1515,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aa83b46c367242899385aa6b52690672-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aa83b46c367242899385aa6b52690672-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fba7e5bbd81947709969ecf2ea91de18-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fba7e5bbd81947709969ecf2ea91de18-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1233,7 +1533,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6043df15584344598f0dc02d0fb55e4c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6043df15584344598f0dc02d0fb55e4c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e33e338b4c0b44b6abaa4405fc9cab15-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e33e338b4c0b44b6abaa4405fc9cab15-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1251,7 +1551,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7b1a669b6d5e4b19b6f24f8930be30e8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7b1a669b6d5e4b19b6f24f8930be30e8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c8cd93a4eb644460974ee4b32ff7ba90-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c8cd93a4eb644460974ee4b32ff7ba90-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1269,7 +1569,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2f6ef3d11b684791a94838fbd67a6da3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2f6ef3d11b684791a94838fbd67a6da3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba4d8a911b85493e8aa09ee65d9ce9bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba4d8a911b85493e8aa09ee65d9ce9bb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1287,7 +1587,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/808c320dcc2c423d9f95291e364b05ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/808c320dcc2c423d9f95291e364b05ed-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/138767dfec404b4fa5b59d88da3e0623-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/138767dfec404b4fa5b59d88da3e0623-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1305,7 +1605,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f68a463e047a4f8ead48158201b121cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f68a463e047a4f8ead48158201b121cf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d21df1b84246418a8034d011e283c1dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d21df1b84246418a8034d011e283c1dd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1323,7 +1623,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1eb22572673d43408d7d67e7f471c935-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1eb22572673d43408d7d67e7f471c935-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d720d82925944487a26c7c631d1d51bf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d720d82925944487a26c7c631d1d51bf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1341,7 +1641,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cc0bac6a0ce474894242fdf3e07e071-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cc0bac6a0ce474894242fdf3e07e071-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a88535b9e91049eb973f725d5cfd340d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a88535b9e91049eb973f725d5cfd340d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1359,7 +1659,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/21302df6a3fe47ceb94684ca16bb5f5a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/21302df6a3fe47ceb94684ca16bb5f5a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d7e79c9f9d14d158e40a88197ba44f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d7e79c9f9d14d158e40a88197ba44f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1377,7 +1677,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/07dfe8f45eb04086b73821a0fbd65151-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/07dfe8f45eb04086b73821a0fbd65151-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e19a9ea3f2cb4669b67390d0cc79b619-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e19a9ea3f2cb4669b67390d0cc79b619-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1395,7 +1695,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0eacaf2159824c7a989fdc4a5f8ac146-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0eacaf2159824c7a989fdc4a5f8ac146-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ad8154cb93f42139f0fa40399724b72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ad8154cb93f42139f0fa40399724b72-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1413,7 +1713,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0f0046dd930643c7a47f101956ada6b4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0f0046dd930643c7a47f101956ada6b4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f808e38765142928fb7442e669cfe94-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f808e38765142928fb7442e669cfe94-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1431,7 +1731,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c63835bd60e24077a2a0a2de6798c581-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c63835bd60e24077a2a0a2de6798c581-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c74893b5bb8f4565a61e98f8ada763d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c74893b5bb8f4565a61e98f8ada763d2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1449,7 +1749,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0df81f23fd1649158bcac1bff8c69536-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0df81f23fd1649158bcac1bff8c69536-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/718868f9041d47329576c71b1f2cfd27-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/718868f9041d47329576c71b1f2cfd27-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1467,7 +1767,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1732543c8ebc4b95aadec8b5051fa585-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1732543c8ebc4b95aadec8b5051fa585-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1e46fa014f5d4d2d93ca33b7834b0ed4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1e46fa014f5d4d2d93ca33b7834b0ed4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1485,7 +1785,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/07b2f75537b145e4971a1a001c216792-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/07b2f75537b145e4971a1a001c216792-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/286f6735e07f48c5858d8e8bf6daf6d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/286f6735e07f48c5858d8e8bf6daf6d3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1503,7 +1803,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e34e4efbe9604010b7d5850af0c1167b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e34e4efbe9604010b7d5850af0c1167b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a9f5061fc60405f9f2fce2aede9b95b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a9f5061fc60405f9f2fce2aede9b95b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1521,7 +1821,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae91eee3d4ab4fefa21a638f3b46ee27-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae91eee3d4ab4fefa21a638f3b46ee27-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5bd7f3a870f347faa4dc023121325c90-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5bd7f3a870f347faa4dc023121325c90-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1539,7 +1839,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/62f1429ebbfd4b088854f6d8a2309df3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/62f1429ebbfd4b088854f6d8a2309df3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7bd276a83a454d7db64440c61c43e91d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7bd276a83a454d7db64440c61c43e91d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1557,7 +1857,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/39638febd7c84e0c83813b221458b7c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/39638febd7c84e0c83813b221458b7c0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3cc776c865d42d29a9965dda34fa5bf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3cc776c865d42d29a9965dda34fa5bf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1575,7 +1875,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7e9186d7ee084508baee0da46c2f92b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7e9186d7ee084508baee0da46c2f92b8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8dd8fabc763448cba8921c76a9fdc3d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8dd8fabc763448cba8921c76a9fdc3d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1593,7 +1893,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c43281931dab4f3f84fc70c96ee8c707-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c43281931dab4f3f84fc70c96ee8c707-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0461fe555aa94854abf1607572777c1e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0461fe555aa94854abf1607572777c1e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1611,7 +1911,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e13e50b7cff64492858ec983a2ef96b9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e13e50b7cff64492858ec983a2ef96b9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/20b04f5a16d14a068546215830201524-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/20b04f5a16d14a068546215830201524-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1629,7 +1929,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3fec1a5c6b6845c2820f3f53f21b96a6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3fec1a5c6b6845c2820f3f53f21b96a6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2611d4553daf48d1be34bbc916b2b0b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2611d4553daf48d1be34bbc916b2b0b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1647,7 +1947,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/52ba226798264b24893f13a0e833b86f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/52ba226798264b24893f13a0e833b86f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b891f20bc794689a67e9ad266c5034b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b891f20bc794689a67e9ad266c5034b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1665,7 +1965,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eeffee23b9ba4c7794b1d32b158460d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eeffee23b9ba4c7794b1d32b158460d0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b156a0f635364fe1b8efb27a1b96ad99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b156a0f635364fe1b8efb27a1b96ad99-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1683,7 +1983,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c850a37559024398a2f67a6ed7f3f4be-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c850a37559024398a2f67a6ed7f3f4be-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cb771b32949474ca695642bc6f02105-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cb771b32949474ca695642bc6f02105-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1701,7 +2001,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c43c07ecccd14f7cb230659fd0152e14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c43c07ecccd14f7cb230659fd0152e14-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c0b8879739c8473db3a5012920d673b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c0b8879739c8473db3a5012920d673b8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1719,7 +2019,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/680ba6b154e04cfa8160517b1c492072-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/680ba6b154e04cfa8160517b1c492072-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9eb67df0965449568a47045fac0807ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9eb67df0965449568a47045fac0807ed-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1737,7 +2037,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9abe20299eb14240aa7fe4008d47da65-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9abe20299eb14240aa7fe4008d47da65-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1fc7824bfb514f5e844985e4b75556b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1fc7824bfb514f5e844985e4b75556b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1755,7 +2055,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ebc391f43274b0793256d18ca2c5ce6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ebc391f43274b0793256d18ca2c5ce6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/926e05d862c9434ba3cf7a8fbacb8b5a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/926e05d862c9434ba3cf7a8fbacb8b5a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1773,7 +2073,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7a85c8fca86e4034886d937078d6fbc0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7a85c8fca86e4034886d937078d6fbc0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53201c7086c74ca19fcf45f6ff7e48ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53201c7086c74ca19fcf45f6ff7e48ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1791,7 +2091,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3ae016f0fb2b47ed9d2c78ac42869d07-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3ae016f0fb2b47ed9d2c78ac42869d07-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cfeb110b50f42fa8be465355b303d98-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cfeb110b50f42fa8be465355b303d98-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1809,7 +2109,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0fd54389ab4d45bdb7cb27a875ac52df-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0fd54389ab4d45bdb7cb27a875ac52df-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1b7fe9558faf43a6bb35ba1a16300d13-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1b7fe9558faf43a6bb35ba1a16300d13-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1827,7 +2127,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0182bda01c0c4042ba504f485d30cea7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0182bda01c0c4042ba504f485d30cea7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35fe18c4f6d14cdcb9cdde3ea063c16b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35fe18c4f6d14cdcb9cdde3ea063c16b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1845,7 +2145,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b6ef75fbbfe4472a9b53f8a08b8fa7b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b6ef75fbbfe4472a9b53f8a08b8fa7b5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d732fd5b4e2b473780d41c5d9e04f4e0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d732fd5b4e2b473780d41c5d9e04f4e0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1863,7 +2163,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d10b3b2830814a7d96572c3f5ec1448b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d10b3b2830814a7d96572c3f5ec1448b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/70b94206f87449a7b5ac69142d779ea0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/70b94206f87449a7b5ac69142d779ea0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1881,7 +2181,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/90236b8648864ce3bd43cc2f6206d28c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/90236b8648864ce3bd43cc2f6206d28c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eafd171f13394c5480a1a0c401a68034-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eafd171f13394c5480a1a0c401a68034-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1899,7 +2199,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e96f3dd7ce084c84ade46720f698c6f0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e96f3dd7ce084c84ade46720f698c6f0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c2ee98ae1224ec899b35f591e12d2ab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c2ee98ae1224ec899b35f591e12d2ab-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1917,7 +2217,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/504d6d4178b14f49ae13064b4c9eb29b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/504d6d4178b14f49ae13064b4c9eb29b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/92c4cfb45632442592b47cf28c3916f7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/92c4cfb45632442592b47cf28c3916f7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1935,7 +2235,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ce6519073644a95996231b8782d685d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ce6519073644a95996231b8782d685d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e710ef2f43cb446e94fde4629eae934c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e710ef2f43cb446e94fde4629eae934c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1953,7 +2253,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ce624b80509f4deaafe6de05f6275fa6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ce624b80509f4deaafe6de05f6275fa6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d83593b7bd8442a1be6e5d04a1ae6091-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d83593b7bd8442a1be6e5d04a1ae6091-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1971,7 +2271,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/65a68eb08d4c43b6b6d6153910439138-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/65a68eb08d4c43b6b6d6153910439138-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cd00cb79f1a442b6a960a9845a16765f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cd00cb79f1a442b6a960a9845a16765f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1989,7 +2289,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a78b2d4121f4d4584162f77217d19a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a78b2d4121f4d4584162f77217d19a9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1842d082ee7f4ec3bb5fb7f552d8d1d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1842d082ee7f4ec3bb5fb7f552d8d1d3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2007,7 +2307,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/93b76286657c44f7b93c94d93e965fe6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/93b76286657c44f7b93c94d93e965fe6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9ae412e50d4a4c5d9e0c57a7334d5bf0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9ae412e50d4a4c5d9e0c57a7334d5bf0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2025,7 +2325,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2f915561feb24c8fb87e450aebfa4ad2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2f915561feb24c8fb87e450aebfa4ad2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2be5fa5156e54bce83be67c929ab097e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2be5fa5156e54bce83be67c929ab097e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2043,7 +2343,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/367ae75d32644f92a11ab8502924d1eb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/367ae75d32644f92a11ab8502924d1eb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/805239464bb5465083b60ced55e5db24-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/805239464bb5465083b60ced55e5db24-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2061,7 +2361,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cc181c63d6da49c9b9dd760510e93074-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cc181c63d6da49c9b9dd760510e93074-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02b68e3885ee41daba88d370bec298a7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02b68e3885ee41daba88d370bec298a7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2079,7 +2379,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c21b6f9f75944bc6b4b38e2d833653bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c21b6f9f75944bc6b4b38e2d833653bc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f7cb73a6b1ec4231a308357ed0ebf9ad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f7cb73a6b1ec4231a308357ed0ebf9ad-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2097,7 +2397,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ce7f136368d6432f93013ca965596e13-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ce7f136368d6432f93013ca965596e13-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3e4262288ed74e4bb3791195b4e2dbc2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3e4262288ed74e4bb3791195b4e2dbc2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2115,7 +2415,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2aa0bf4ec9dc420fb1df47460c66ea03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2aa0bf4ec9dc420fb1df47460c66ea03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35d55a9f7b6c4dfa90027d14c6753644-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35d55a9f7b6c4dfa90027d14c6753644-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2133,7 +2433,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/013db806b7a6473f8282ac2a955a4495-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/013db806b7a6473f8282ac2a955a4495-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8c73363217f04e15a17cfde9a81e3425-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8c73363217f04e15a17cfde9a81e3425-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2151,7 +2451,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/544c490b32464323b416c58ed64cea8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/544c490b32464323b416c58ed64cea8f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d4c40102921846778e094ad25952994d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d4c40102921846778e094ad25952994d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2169,7 +2469,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d8ec7b6e6604e1a8f2b9fa4d097874d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d8ec7b6e6604e1a8f2b9fa4d097874d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e264b62fa2d948288bdb1c9cae5103a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e264b62fa2d948288bdb1c9cae5103a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2187,7 +2487,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b6a49df27bac41e69b79d5aa6b232a2f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b6a49df27bac41e69b79d5aa6b232a2f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ed5255b4ecc24128a33f2fa36af28afc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ed5255b4ecc24128a33f2fa36af28afc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2205,7 +2505,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/21b84bbc058a4ebc8fc5d2ebe79c6d17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/21b84bbc058a4ebc8fc5d2ebe79c6d17-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c08bd64828204425a02e8b05aa763e76-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c08bd64828204425a02e8b05aa763e76-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2223,7 +2523,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f92ef25afb234b7ea3972187722c49ae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f92ef25afb234b7ea3972187722c49ae-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7f58d5af75b74c3089d9311e3b3babee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7f58d5af75b74c3089d9311e3b3babee-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2241,7 +2541,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/07556f0581a1414b9a187eeae3724a80-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/07556f0581a1414b9a187eeae3724a80-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a09f7b4c3e384f40ae00c4611a5d15a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a09f7b4c3e384f40ae00c4611a5d15a4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2259,7 +2559,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2846a3fce3c74d57a0a846a2c4b62f9b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2846a3fce3c74d57a0a846a2c4b62f9b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b4b47bfa55514b54a4a711755800709e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b4b47bfa55514b54a4a711755800709e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2277,7 +2577,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ab821309e7b34111835918c5c96709f6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ab821309e7b34111835918c5c96709f6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ffba51b2e1364d2892e451a8f6a041a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ffba51b2e1364d2892e451a8f6a041a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2295,7 +2595,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/60e419733a1242729bafb59d85f9a54e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/60e419733a1242729bafb59d85f9a54e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35591768dbfe412ab601344dd0ce5eea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35591768dbfe412ab601344dd0ce5eea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2313,7 +2613,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e84bc8e50ae5498aad42cc4f196c100d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e84bc8e50ae5498aad42cc4f196c100d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1ac121e291f042afa86b7ba68344f6a6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1ac121e291f042afa86b7ba68344f6a6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2331,7 +2631,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f6b05f453cf4716a974f780a25c3ad4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f6b05f453cf4716a974f780a25c3ad4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1e17638eede4ee4a0b858f9a9eac28d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1e17638eede4ee4a0b858f9a9eac28d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2349,7 +2649,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/392e975ae24f4d11bfa50a27675c1797-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/392e975ae24f4d11bfa50a27675c1797-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c651d5ff3033464b9ce21f3834b5f8d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c651d5ff3033464b9ce21f3834b5f8d6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2367,7 +2667,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/26d4a17f3d80464cbf56975ef1dbddf2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/26d4a17f3d80464cbf56975ef1dbddf2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b630d10c8ea643b6a36df6923a19b8ad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b630d10c8ea643b6a36df6923a19b8ad-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2385,7 +2685,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b85a0e57e71d483682806e386388a612-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b85a0e57e71d483682806e386388a612-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/26b966a98db34a238ec2150c272061dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/26b966a98db34a238ec2150c272061dd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2403,7 +2703,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c8c544f454204b0498d64e061fff9143-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c8c544f454204b0498d64e061fff9143-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0953fdf265e140899b7ed42791ce097a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0953fdf265e140899b7ed42791ce097a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2421,7 +2721,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/37c3c6214e9d41f197a2d277eca031b0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/37c3c6214e9d41f197a2d277eca031b0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9dc377c2315849bd9ce1f9c966621c27-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9dc377c2315849bd9ce1f9c966621c27-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2439,7 +2739,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/31d4433360b94f85b2baaad57c508c6d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/31d4433360b94f85b2baaad57c508c6d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b13993bc1e346a0b7c84ddbef0b2e6b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b13993bc1e346a0b7c84ddbef0b2e6b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2470,10 +2770,10 @@
"dataset = {'p': 126, 'n': 131}\n",
"folding = {'n_folds': 2, 'n_repeats': 1}\n",
"scores = {'acc': 0.573, 'sens': 0.768, 'bacc': 0.862}\n",
- "result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_1_dataset_unknown_folds_mos(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
@@ -2490,7 +2790,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8bd57a7ec351427781381b9085fdaa45-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8bd57a7ec351427781381b9085fdaa45-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7308d608a2e24513a7e38a4de373cd19-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7308d608a2e24513a7e38a4de373cd19-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2510,7 +2810,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/60682dd4d246479faec8d469548f1026-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/60682dd4d246479faec8d469548f1026-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b2f234badd72473ca7df71d568fe87e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b2f234badd72473ca7df71d568fe87e7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2553,13 +2853,13 @@
"Time (Wallclock seconds): 0.00\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.01\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9bd9fc5161174da2b6f2834705873e98-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9bd9fc5161174da2b6f2834705873e98-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a4e7620bf12447958fdf547a6c859382-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a4e7620bf12447958fdf547a6c859382-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2582,7 +2882,7 @@
"Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 6 (TwoMirCuts) - 1 row cuts average 4.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0001I Search completed - best objective 1e+50, took 0 iterations and 0 nodes (0.01 seconds)\n",
+ "Cbc0001I Search completed - best objective 1e+50, took 0 iterations and 0 nodes (0.00 seconds)\n",
"Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
"Cuts at root node changed objective from 0 to 0\n",
"Probing was tried 1 times and created 1 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
@@ -2600,16 +2900,16 @@
"Enumerated nodes: 0\n",
"Total iterations: 0\n",
"Time (CPU seconds): 0.00\n",
- "Time (Wallclock seconds): 0.01\n",
+ "Time (Wallclock seconds): 0.00\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.01\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a7b90a9391a40668190dbdcd5584003-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a7b90a9391a40668190dbdcd5584003-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a32ff5f1fe95460eb9be0e495266153c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a32ff5f1fe95460eb9be0e495266153c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2629,7 +2929,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b0bddf9fe1dd4d77b9c7929dbd4d5de5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b0bddf9fe1dd4d77b9c7929dbd4d5de5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02d606a9cce84cb5aebbf3e2120bf3bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02d606a9cce84cb5aebbf3e2120bf3bd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2650,7 +2950,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ed5a522c08ce4f88a414b8a7f8ef2dd8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ed5a522c08ce4f88a414b8a7f8ef2dd8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/86e07e044162468099e360bad3d20cc5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/86e07e044162468099e360bad3d20cc5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2692,13 +2992,13 @@
"Time (Wallclock seconds): 0.00\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0b32952ed1b84ccabcef7adcab7700dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0b32952ed1b84ccabcef7adcab7700dd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7e80acb7c9ef4bcc931d10fd4a774a77-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7e80acb7c9ef4bcc931d10fd4a774a77-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2750,7 +3050,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e82e5f3dc70846e8a6a2da2a26b79143-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e82e5f3dc70846e8a6a2da2a26b79143-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/767543b38aa74b3e98838a00aacc57a3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/767543b38aa74b3e98838a00aacc57a3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2798,7 +3098,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0baf4c2ce91447479220f2319fac7e43-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0baf4c2ce91447479220f2319fac7e43-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/88973059ff87419a90436b5afdd6be5a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/88973059ff87419a90436b5afdd6be5a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2818,7 +3118,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02794b967d054b83b2691ade923d5c7d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02794b967d054b83b2691ade923d5c7d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/152a889c4695466f951e3dd3edd281cb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/152a889c4695466f951e3dd3edd281cb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2868,7 +3168,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1951d72e710f4abf991934ccce268ce8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1951d72e710f4abf991934ccce268ce8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd09b04e846b4f9989a2cc496163aa84-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd09b04e846b4f9989a2cc496163aa84-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2916,7 +3216,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e91fb559cd3c488f8f6491be7d5cf8bf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e91fb559cd3c488f8f6491be7d5cf8bf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2945c317e4224c6aa25d6c1fcb4488bf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2945c317e4224c6aa25d6c1fcb4488bf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2934,7 +3234,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/48b9f112394f40f2a9fc17caab9a565f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/48b9f112394f40f2a9fc17caab9a565f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/017fbd58307449f49ab018d139385d79-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/017fbd58307449f49ab018d139385d79-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2984,7 +3284,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8206232ff1bf4e088e907197c8d3c53c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8206232ff1bf4e088e907197c8d3c53c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00fc9acd8779430a8da1b112e629fba5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00fc9acd8779430a8da1b112e629fba5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3032,7 +3332,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13b3ad6c105a45fb9d14daed33e8a0d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13b3ad6c105a45fb9d14daed33e8a0d3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d625f8f24f704ba698c5967f163bda91-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d625f8f24f704ba698c5967f163bda91-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3050,7 +3350,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2cb5fae2092f4423ab0e2cb30607675e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2cb5fae2092f4423ab0e2cb30607675e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68976850fc5d41dda23dd873ca4d02a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68976850fc5d41dda23dd873ca4d02a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3068,7 +3368,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a2c2964936b43b1bcb1398b92ec60d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a2c2964936b43b1bcb1398b92ec60d9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d080e5151522428ea471189674cac12c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d080e5151522428ea471189674cac12c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3119,7 +3419,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/16c3d049e25e4770b5723fb111184d04-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/16c3d049e25e4770b5723fb111184d04-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/778777bdf19244648f23a431f0cdf7bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/778777bdf19244648f23a431f0cdf7bc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3137,7 +3437,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/03584539c0054f8fb29129ff8dec60eb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/03584539c0054f8fb29129ff8dec60eb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6dd2e63a9ab944ecbebbfd689b8a74a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6dd2e63a9ab944ecbebbfd689b8a74a4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3155,7 +3455,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d414147e110f43d4ae415ac1c2642797-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d414147e110f43d4ae415ac1c2642797-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87782fd603f74f7fa891ce30e14c4978-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87782fd603f74f7fa891ce30e14c4978-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3173,7 +3473,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/40961c77c61c4ad5bac097c3ec93fff0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/40961c77c61c4ad5bac097c3ec93fff0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d69d2bb61aa47d1a98cbc216d8810e4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d69d2bb61aa47d1a98cbc216d8810e4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3191,7 +3491,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/56d96ad8aba94c0abc9f95ba2aca88ac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/56d96ad8aba94c0abc9f95ba2aca88ac-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/03cfe0210b054240b67df173e75fbe18-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/03cfe0210b054240b67df173e75fbe18-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3209,7 +3509,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/54e1cd96f6b74e4c9aa0b9264c637ec5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/54e1cd96f6b74e4c9aa0b9264c637ec5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1070826a367a4e83819b426cc8044fab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1070826a367a4e83819b426cc8044fab-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3227,7 +3527,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d93b056563f84e2c9d17a978e6ae60cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d93b056563f84e2c9d17a978e6ae60cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b577fef9e2364f03b8576b718d0b4689-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b577fef9e2364f03b8576b718d0b4689-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3245,7 +3545,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a7564d244ad4428c89246816126b3824-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a7564d244ad4428c89246816126b3824-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7dc0d5249c75437d923e4a2a92fd0b4a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7dc0d5249c75437d923e4a2a92fd0b4a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3263,7 +3563,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d9bbe8cb403348788e2599085e90472d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d9bbe8cb403348788e2599085e90472d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b614d43e7e864c6ea044f167ed904d2a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b614d43e7e864c6ea044f167ed904d2a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3303,7 +3603,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5380ae8c7e124957b272563ded388c73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5380ae8c7e124957b272563ded388c73-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7c36f04f689048f69cf730fa995d4afe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7c36f04f689048f69cf730fa995d4afe-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3321,7 +3621,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5d56ba0f181c41a98231005a2126837d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5d56ba0f181c41a98231005a2126837d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3a554264f0a400d848c2f5429284f1b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3a554264f0a400d848c2f5429284f1b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3339,7 +3639,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64f4b8f7bcfa4d6c876633c2d1fd4739-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64f4b8f7bcfa4d6c876633c2d1fd4739-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f64877e35f047a7aa96e417461d4b87-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f64877e35f047a7aa96e417461d4b87-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3357,7 +3657,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d5ea48f39954348ab0da3b5c8549e0f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d5ea48f39954348ab0da3b5c8549e0f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ce5f368acad64c6591ae45ff703d7dac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ce5f368acad64c6591ae45ff703d7dac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3375,7 +3675,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/855057c2dd4143a9bf84b954da996aa5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/855057c2dd4143a9bf84b954da996aa5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3e69cd03cf5e477295df144120b9acf2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3e69cd03cf5e477295df144120b9acf2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3425,7 +3725,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e32841982e5f4c90883e939f1e059cf6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e32841982e5f4c90883e939f1e059cf6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02b993bed6de440988d20c287aed1a05-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02b993bed6de440988d20c287aed1a05-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3443,7 +3743,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3f0dae216dcc43d28c1a32874fa5fb60-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3f0dae216dcc43d28c1a32874fa5fb60-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7114d25b724e4119b4be70027af0819a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7114d25b724e4119b4be70027af0819a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3461,7 +3761,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/caac704f07144e78b744dc11e19a7bbc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/caac704f07144e78b744dc11e19a7bbc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a94ff9e0d50c4c2eae30d2769ae01ebc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a94ff9e0d50c4c2eae30d2769ae01ebc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3479,7 +3779,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a940db3927594bf486212e0da85121af-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a940db3927594bf486212e0da85121af-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cba4e3bdad6d46dbafe9500a14cebcc7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cba4e3bdad6d46dbafe9500a14cebcc7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3497,7 +3797,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/47931b2904584424ae69b83716d67852-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/47931b2904584424ae69b83716d67852-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a2bdafffa4084699b2798a27910af1ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a2bdafffa4084699b2798a27910af1ed-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3515,7 +3815,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d147eff0665942b79df9c3121488dd58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d147eff0665942b79df9c3121488dd58-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4842888d7cc941dfbef5942af753db72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4842888d7cc941dfbef5942af753db72-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3565,7 +3865,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/afab073e74364b709785d55f8cc85ef2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/afab073e74364b709785d55f8cc85ef2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/991cd811940b47838a53bcfa4d122e18-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/991cd811940b47838a53bcfa4d122e18-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3612,7 +3912,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4d1580f3e1f14fb68b39dcaaa10d975e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4d1580f3e1f14fb68b39dcaaa10d975e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/96f1fc1a491f46ae850b7873874ecf04-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/96f1fc1a491f46ae850b7873874ecf04-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3630,7 +3930,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5742a4d3bb294952948bdf9b67c696e3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5742a4d3bb294952948bdf9b67c696e3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5e5693c8262349fab36fb810a13da7ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5e5693c8262349fab36fb810a13da7ed-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3648,7 +3948,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4e6e6fcf58224a609f1d5232a75d7291-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4e6e6fcf58224a609f1d5232a75d7291-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8f3f34ad9654852aba8604b739b97f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8f3f34ad9654852aba8604b739b97f9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3666,7 +3966,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/133275acbad24af4bc67be39520310e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/133275acbad24af4bc67be39520310e7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a878aa822df94dc89060ba0f935163ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a878aa822df94dc89060ba0f935163ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3684,7 +3984,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d846ccba30c9473998a76de6f860bc5d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d846ccba30c9473998a76de6f860bc5d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f1c229ce0444493c959e6773f0453588-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f1c229ce0444493c959e6773f0453588-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3734,7 +4034,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0b373fa6865546c2a55b7c1b47aee258-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0b373fa6865546c2a55b7c1b47aee258-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4649d2a81f3f4a8aa1168b18462de4ba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4649d2a81f3f4a8aa1168b18462de4ba-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3781,7 +4081,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c093cd4bf0b4959987df860f194de59-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c093cd4bf0b4959987df860f194de59-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9f31348825684bf2aad767a5ba84f650-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9f31348825684bf2aad767a5ba84f650-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3799,7 +4099,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33ed4e818e334eddbb23b0bfd9df718f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33ed4e818e334eddbb23b0bfd9df718f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bfcf200b0e2c41e48aed42aabb12d95c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bfcf200b0e2c41e48aed42aabb12d95c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3817,7 +4117,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/832087e4b4be4322b3f7a329e2a02f24-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/832087e4b4be4322b3f7a329e2a02f24-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fdef367866174dfabc31798238895fe5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fdef367866174dfabc31798238895fe5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3835,7 +4135,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8191b4cde7a742eeb276aa079827b5bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8191b4cde7a742eeb276aa079827b5bc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7e482d142c714373a13fadcdfa91d600-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7e482d142c714373a13fadcdfa91d600-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3853,7 +4153,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f0da9b1201c43c19b00533c14efbac2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f0da9b1201c43c19b00533c14efbac2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fbb83858200440708168d69bd4f74b37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fbb83858200440708168d69bd4f74b37-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3871,7 +4171,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e85c18e559df41bc82d73355d3cc48fb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e85c18e559df41bc82d73355d3cc48fb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8aab9cdf3424afe9881a0201031bea9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8aab9cdf3424afe9881a0201031bea9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3892,7 +4192,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/84731b0d664946799dbe06c868a34896-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/84731b0d664946799dbe06c868a34896-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9b859e9950914fe8b07a669189da7d44-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9b859e9950914fe8b07a669189da7d44-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3939,7 +4239,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/added25eabc84acbbfe74c7617b4b70c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/added25eabc84acbbfe74c7617b4b70c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/17658929775a448b9c47d3403d18ffcd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/17658929775a448b9c47d3403d18ffcd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3957,7 +4257,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a825d5cf3e8d41e1b21f5eac9aec30f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a825d5cf3e8d41e1b21f5eac9aec30f3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c548fd8e8f1c4622a80c51138c3c4d40-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c548fd8e8f1c4622a80c51138c3c4d40-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3975,7 +4275,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7d88f31d06ae442693f18cd8737ab890-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7d88f31d06ae442693f18cd8737ab890-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/45113bddba874844be13ff66be585142-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/45113bddba874844be13ff66be585142-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3993,7 +4293,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/af84be08be96433fac25410a27c85be9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/af84be08be96433fac25410a27c85be9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7603510ff59b4670a1e4f66c27986196-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7603510ff59b4670a1e4f66c27986196-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4011,7 +4311,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae2a3cc3e2204252a267dcbff4ae9a63-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae2a3cc3e2204252a267dcbff4ae9a63-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7aa2e395cff945a8982226c57b764565-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7aa2e395cff945a8982226c57b764565-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4029,7 +4329,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a931624ced404164a376b8719802bc8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a931624ced404164a376b8719802bc8f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9104dcf3b46546e782640570f6af57ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9104dcf3b46546e782640570f6af57ee-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4079,7 +4379,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/01253ba7cafe4b1d94398fd55116f69f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/01253ba7cafe4b1d94398fd55116f69f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f81ddbc41f5a4a51adc667ed6050fc81-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f81ddbc41f5a4a51adc667ed6050fc81-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4128,7 +4428,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6dae961590f24364b366892515ba5129-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6dae961590f24364b366892515ba5129-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c36f52ceced340f8a27ab549af3660c1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c36f52ceced340f8a27ab549af3660c1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4175,7 +4475,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7983f84fced842ab8646041d18f485cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7983f84fced842ab8646041d18f485cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4a6863f65eb84fae9d25b8e1063eaf42-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4a6863f65eb84fae9d25b8e1063eaf42-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4193,7 +4493,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a263190443eb4c789b8c02fa1458bf6d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a263190443eb4c789b8c02fa1458bf6d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8897f86f1464336abcdcd737fc3d000-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8897f86f1464336abcdcd737fc3d000-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4211,7 +4511,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9ff6802f5b264a278deaa9b492b2b10f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9ff6802f5b264a278deaa9b492b2b10f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bce0017a3c7a4efcaa8bb53e256c0a3c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bce0017a3c7a4efcaa8bb53e256c0a3c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4229,7 +4529,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f4fca5148f5c4488b823f12d5ab8e775-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f4fca5148f5c4488b823f12d5ab8e775-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f21ac3bb788499bbe7bb7689ddd5176-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f21ac3bb788499bbe7bb7689ddd5176-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4247,7 +4547,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/352454b145974964857cf86d7412a46d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/352454b145974964857cf86d7412a46d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33d4814bc3ec4f4882f42549d40b2040-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33d4814bc3ec4f4882f42549d40b2040-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4265,7 +4565,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/59a56df06d6c48088c1c66bef4a1e0c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/59a56df06d6c48088c1c66bef4a1e0c9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c101c5396cd4cd59f450b932608ba7e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c101c5396cd4cd59f450b932608ba7e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4314,7 +4614,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7147a16c74d64989a139d630759edff1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7147a16c74d64989a139d630759edff1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aa4b54dce2b14ed0b2e6e545330738cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aa4b54dce2b14ed0b2e6e545330738cd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4361,7 +4661,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ddc4e9e9223e43ef9797b844f26e31f8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ddc4e9e9223e43ef9797b844f26e31f8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9171ed848919435fbf59041692e6f4c2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9171ed848919435fbf59041692e6f4c2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4409,7 +4709,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/351cd7c203b14c16b18a2a2e095f785e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/351cd7c203b14c16b18a2a2e095f785e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/944410fadcfe4bcd9d3ba307bc9893c1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/944410fadcfe4bcd9d3ba307bc9893c1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4457,7 +4757,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/317e73badf734e6fbf44e78db5bdc357-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/317e73badf734e6fbf44e78db5bdc357-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db493403a38e4ec08ef68147a243555f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db493403a38e4ec08ef68147a243555f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4504,7 +4804,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8c68e5f31cca48939a78cdf98732f1cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8c68e5f31cca48939a78cdf98732f1cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e26e18591cb47239c02746364317ad8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e26e18591cb47239c02746364317ad8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4524,7 +4824,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/05d2d7ab59d74b02a14273beca377101-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/05d2d7ab59d74b02a14273beca377101-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7c006fbfc51c4a3c9ae0cda8afb6a003-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7c006fbfc51c4a3c9ae0cda8afb6a003-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4542,7 +4842,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0797fe93ebe5490881dfffc5f7703b06-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0797fe93ebe5490881dfffc5f7703b06-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ec648b6fd6d94effb0d67c9d93e94185-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ec648b6fd6d94effb0d67c9d93e94185-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4560,7 +4860,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b4c3f4899bfd4606b7626c8cebd3f4d4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b4c3f4899bfd4606b7626c8cebd3f4d4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/94653ab43c124540b95c907713751390-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/94653ab43c124540b95c907713751390-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4618,7 +4918,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f1cff5d858ba4652bb0d0dda1d09f812-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f1cff5d858ba4652bb0d0dda1d09f812-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89060da6e6c141c6a09b803f58fee526-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89060da6e6c141c6a09b803f58fee526-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4667,7 +4967,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5113aac5b0ab429483b21f20dad4cffe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5113aac5b0ab429483b21f20dad4cffe-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/172165f6355d40d182451f12d5e7cfae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/172165f6355d40d182451f12d5e7cfae-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4717,7 +5017,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1fb889090d848e09c9b15e509b7cb40-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1fb889090d848e09c9b15e509b7cb40-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b31a71f57fa4acda34da56770371903-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b31a71f57fa4acda34da56770371903-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4738,7 +5038,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5dedebdfd24d4fd286cf5d01f9b33bf0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5dedebdfd24d4fd286cf5d01f9b33bf0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a5d5e34746ee4d7ca5e67746db7a2820-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a5d5e34746ee4d7ca5e67746db7a2820-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4789,7 +5089,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2074734af94940e1a14c2aeebdd7a05e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2074734af94940e1a14c2aeebdd7a05e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a10454603a844fcc8d4df0bdffab908f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a10454603a844fcc8d4df0bdffab908f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4839,7 +5139,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb7a03386ddc4304b8394e362a093b25-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb7a03386ddc4304b8394e362a093b25-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3f04bc80623c4364a2b4d07ca6126440-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3f04bc80623c4364a2b4d07ca6126440-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4889,7 +5189,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/583b6e1352b14b41acade255f1a46837-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/583b6e1352b14b41acade255f1a46837-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2503cbee5c5c4df7a4f1eb5024df416e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2503cbee5c5c4df7a4f1eb5024df416e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4929,7 +5229,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/65860b0476974dec9cba63237b5cda0c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/65860b0476974dec9cba63237b5cda0c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e33e41611d34c6c8cb24a7ca57e21b3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e33e41611d34c6c8cb24a7ca57e21b3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4960,10 +5260,10 @@
"dataset = {'p': 19, 'n': 97}\n",
"folding = {'n_folds': 3, 'n_repeats': 1}\n",
"scores = {'acc': 0.9, 'spec': 0.9, 'sens': 0.6}\n",
- "result = check_1_dataset_unknown_folds_mos_scores(dataset=dataset,\n",
- " folding=folding,\n",
- " scores=scores,\n",
- " eps=1e-4)\n",
+ "result = check_1_dataset_unknown_folds_mos(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
"result['inconsistency']"
]
},
diff --git a/notebooks/illustration/04_check_n_datasets_som_kfold_som.ipynb b/notebooks/illustration/binary/04_check_n_datasets_som_kfold_som.ipynb
similarity index 69%
rename from notebooks/illustration/04_check_n_datasets_som_kfold_som.ipynb
rename to notebooks/illustration/binary/04_check_n_datasets_som_kfold_som.ipynb
index 4615f9d..f08dad4 100644
--- a/notebooks/illustration/04_check_n_datasets_som_kfold_som.ipynb
+++ b/notebooks/illustration/binary/04_check_n_datasets_som_kfold_som.ipynb
@@ -7,7 +7,7 @@
"outputs": [],
"source": [
"from mlscorecheck.aggregated import generate_experiment, Experiment\n",
- "from mlscorecheck.check import check_n_datasets_som_kfold_som_scores"
+ "from mlscorecheck.check.binary import check_n_datasets_som_kfold_som"
]
},
{
@@ -19,14 +19,14 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:38:38,707:INFO:testing acc, feasible tptn pairs: 1099\n",
- "2023-10-26 23:38:38,743:INFO:testing sens, feasible tptn pairs: 1097\n",
- "2023-10-26 23:38:38,745:INFO:testing spec, feasible tptn pairs: 2\n",
- "2023-10-26 23:38:38,746:INFO:testing f1p, feasible tptn pairs: 2\n",
- "2023-10-26 23:38:38,746:INFO:testing fm, feasible tptn pairs: 2\n",
- "2023-10-26 23:38:38,747:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:38:38,749:INFO:final number of intervals: 2\n",
- "2023-10-26 23:38:38,751:INFO:final number of pairs: 4\n"
+ "2023-11-03 06:35:58,805:INFO:testing acc, feasible tptn pairs: 1099\n",
+ "2023-11-03 06:35:58,858:INFO:testing sens, feasible tptn pairs: 1097\n",
+ "2023-11-03 06:35:58,862:INFO:testing spec, feasible tptn pairs: 2\n",
+ "2023-11-03 06:35:58,863:INFO:testing f1p, feasible tptn pairs: 2\n",
+ "2023-11-03 06:35:58,864:INFO:testing fm, feasible tptn pairs: 2\n",
+ "2023-11-03 06:35:58,866:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:35:58,867:INFO:final number of intervals: 2\n",
+ "2023-11-03 06:35:58,869:INFO:final number of pairs: 4\n"
]
},
{
@@ -48,9 +48,9 @@
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.631, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}\n",
"\n",
- "result = check_n_datasets_som_kfold_som_scores(scores=scores,\n",
- " evaluations=evaluations,\n",
- " eps=1e-3)\n",
+ "result = check_n_datasets_som_kfold_som(scores=scores,\n",
+ " evaluations=evaluations,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
@@ -63,11 +63,11 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:38:38,768:INFO:testing acc, feasible tptn pairs: 1099\n",
- "2023-10-26 23:38:38,800:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:38:38,800:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:38:38,801:INFO:final number of intervals: 0\n",
- "2023-10-26 23:38:38,802:INFO:final number of pairs: 0\n"
+ "2023-11-03 06:35:58,928:INFO:testing acc, feasible tptn pairs: 1099\n",
+ "2023-11-03 06:35:59,020:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 06:35:59,023:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:35:59,027:INFO:final number of intervals: 0\n",
+ "2023-11-03 06:35:59,028:INFO:final number of pairs: 0\n"
]
},
{
@@ -89,7 +89,7 @@
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.731, 'sens': 0.341, 'spec': 0.802, 'f1p': 0.406, 'fm': 0.414}\n",
"\n",
- "result = check_n_datasets_som_kfold_som_scores(scores=scores,\n",
+ "result = check_n_datasets_som_kfold_som(scores=scores,\n",
" evaluations=evaluations,\n",
" eps=1e-4/2)\n",
"result['inconsistency']"
diff --git a/notebooks/illustration/05_check_n_datasets_mos_kfold_som.ipynb b/notebooks/illustration/binary/05_check_n_datasets_mos_kfold_som.ipynb
similarity index 80%
rename from notebooks/illustration/05_check_n_datasets_mos_kfold_som.ipynb
rename to notebooks/illustration/binary/05_check_n_datasets_mos_kfold_som.ipynb
index bbf0acf..da040bf 100644
--- a/notebooks/illustration/05_check_n_datasets_mos_kfold_som.ipynb
+++ b/notebooks/illustration/binary/05_check_n_datasets_mos_kfold_som.ipynb
@@ -6,7 +6,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.check import check_n_datasets_mos_kfold_som_scores\n",
+ "from mlscorecheck.check.binary import check_n_datasets_mos_kfold_som\n",
"from mlscorecheck.aggregated import Experiment"
]
},
@@ -19,10 +19,10 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:38:48,611:INFO:qynvx: adding lower bound 0.0 for acc\n",
- "2023-10-26 23:38:48,613:INFO:qynvx: adding upper bound 0.5 for acc\n",
- "2023-10-26 23:38:48,614:INFO:common_datasets.winequality-white-3_vs_7_ssp: adding lower bound 0.0 for acc\n",
- "2023-10-26 23:38:48,615:INFO:common_datasets.winequality-white-3_vs_7_ssp: adding upper bound 0.5 for acc\n"
+ "2023-11-03 07:28:07,181:INFO:fgryy: adding lower bound 0.0 for acc\n",
+ "2023-11-03 07:28:07,183:INFO:fgryy: adding upper bound 0.5 for acc\n",
+ "2023-11-03 07:28:07,184:INFO:common_datasets.winequality-white-3_vs_7_qiy: adding lower bound 0.0 for acc\n",
+ "2023-11-03 07:28:07,185:INFO:common_datasets.winequality-white-3_vs_7_qiy: adding upper bound 0.5 for acc\n"
]
},
{
@@ -33,7 +33,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e04b00c1cdae4237acefc166ed4fcde3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e04b00c1cdae4237acefc166ed4fcde3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc7cc1689f8943d4b7c2b4ee8e72ef0a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc7cc1689f8943d4b7c2b4ee8e72ef0a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 17 COLUMNS\n",
@@ -98,10 +98,10 @@
" 'folding': {'n_folds': 5, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}}\n",
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.312, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}\n",
- "result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,\n",
- " dataset_score_bounds={'acc': (0.0, 0.5)},\n",
- " eps=1e-4,\n",
- " scores=scores)\n",
+ "result = check_n_datasets_mos_kfold_som(evaluations=evaluations,\n",
+ " dataset_score_bounds={'acc': (0.0, 0.5)},\n",
+ " eps=1e-4,\n",
+ " scores=scores)\n",
"result['inconsistency']\n"
]
},
@@ -114,10 +114,10 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:38:48,639:INFO:bhkki: adding lower bound 0.5 for acc\n",
- "2023-10-26 23:38:48,641:INFO:bhkki: adding upper bound 1.0 for acc\n",
- "2023-10-26 23:38:48,642:INFO:common_datasets.winequality-white-3_vs_7_jws: adding lower bound 0.5 for acc\n",
- "2023-10-26 23:38:48,644:INFO:common_datasets.winequality-white-3_vs_7_jws: adding upper bound 1.0 for acc\n"
+ "2023-11-03 07:28:07,220:INFO:jpqbm: adding lower bound 0.5 for acc\n",
+ "2023-11-03 07:28:07,222:INFO:jpqbm: adding upper bound 1.0 for acc\n",
+ "2023-11-03 07:28:07,223:INFO:common_datasets.winequality-white-3_vs_7_kpi: adding lower bound 0.5 for acc\n",
+ "2023-11-03 07:28:07,225:INFO:common_datasets.winequality-white-3_vs_7_kpi: adding upper bound 1.0 for acc\n"
]
},
{
@@ -128,7 +128,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8226e670837e4df2a461d2fe01b49243-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8226e670837e4df2a461d2fe01b49243-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/436d49cf8ae14bbf82c80ab3daeacf86-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/436d49cf8ae14bbf82c80ab3daeacf86-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 17 COLUMNS\n",
@@ -162,10 +162,10 @@
" 'folding': {'n_folds': 3, 'n_repeats': 3, 'strategy': 'stratified_sklearn'}}\n",
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.412, 'sens': 0.45, 'spec': 0.312, 'bacc': 0.381}\n",
- "result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,\n",
- " dataset_score_bounds={'acc': (0.5, 1.0)},\n",
- " eps=1e-4,\n",
- " scores=scores)\n",
+ "result = check_n_datasets_mos_kfold_som(evaluations=evaluations,\n",
+ " dataset_score_bounds={'acc': (0.5, 1.0)},\n",
+ " eps=1e-4,\n",
+ " scores=scores)\n",
"result['inconsistency']\n"
]
},
@@ -199,7 +199,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ab27edd3c0904e13b2652d1eb97993af-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ab27edd3c0904e13b2652d1eb97993af-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3f430724d2a34971b4aa601c87567ea6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3f430724d2a34971b4aa601c87567ea6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -227,9 +227,9 @@
}
],
"source": [
- "result = check_n_datasets_mos_kfold_som_scores(evaluations=evaluations,\n",
- " eps=1e-4,\n",
- " scores=scores)\n",
+ "result = check_n_datasets_mos_kfold_som(evaluations=evaluations,\n",
+ " eps=1e-4,\n",
+ " scores=scores)\n",
"result['inconsistency']"
]
}
diff --git a/notebooks/illustration/06_check_n_datasets_mos_known_folds_mos.ipynb b/notebooks/illustration/binary/06_check_n_datasets_mos_known_folds_mos.ipynb
similarity index 79%
rename from notebooks/illustration/06_check_n_datasets_mos_known_folds_mos.ipynb
rename to notebooks/illustration/binary/06_check_n_datasets_mos_known_folds_mos.ipynb
index af6adb3..3ba2187 100644
--- a/notebooks/illustration/06_check_n_datasets_mos_known_folds_mos.ipynb
+++ b/notebooks/illustration/binary/06_check_n_datasets_mos_known_folds_mos.ipynb
@@ -2,17 +2,17 @@
"cells": [
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.check import check_n_datasets_mos_known_folds_mos_scores\n",
+ "from mlscorecheck.check.binary import check_n_datasets_mos_known_folds_mos\n",
"from mlscorecheck.aggregated import Experiment"
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 6,
"metadata": {},
"outputs": [
{
@@ -23,7 +23,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/97d3b2d322384729aee6f84dbc67d28b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/97d3b2d322384729aee6f84dbc67d28b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b443a865af7d44119a3e1bd681731ade-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b443a865af7d44119a3e1bd681731ade-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -57,7 +57,7 @@
"Time (Wallclock seconds): 0.00\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
"\n"
]
},
@@ -67,7 +67,7 @@
"False"
]
},
- "execution_count": 2,
+ "execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
@@ -80,9 +80,9 @@
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.61, 'sens': 0.709, 'spec': 0.461, 'bacc': 0.585}\n",
"\n",
- "result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
@@ -99,7 +99,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f129d378f794e338ca025f7783bc836-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f129d378f794e338ca025f7783bc836-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/adb9092f188a4abc80efed2e5428d4c4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/adb9092f188a4abc80efed2e5428d4c4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -134,9 +134,9 @@
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.71, 'sens': 0.709, 'spec': 0.461}\n",
"\n",
- "result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
@@ -149,10 +149,10 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:38:59,710:INFO:common_datasets.ecoli_0_2_3_4_vs_5_tpe: adding lower bound 0.8 for sens\n",
- "2023-10-26 23:38:59,714:INFO:common_datasets.ecoli_0_2_3_4_vs_5_tpe: adding upper bound 1.0 for sens\n",
- "2023-10-26 23:38:59,715:INFO:iynoj: adding lower bound 0.8 for sens\n",
- "2023-10-26 23:38:59,717:INFO:iynoj: adding upper bound 1.0 for sens\n"
+ "2023-11-03 07:00:11,341:INFO:common_datasets.ecoli_0_2_3_4_vs_5_wkx: adding lower bound 0.8 for sens\n",
+ "2023-11-03 07:00:11,343:INFO:common_datasets.ecoli_0_2_3_4_vs_5_wkx: adding upper bound 1.0 for sens\n",
+ "2023-11-03 07:00:11,344:INFO:rxnah: adding lower bound 0.8 for sens\n",
+ "2023-11-03 07:00:11,344:INFO:rxnah: adding upper bound 1.0 for sens\n"
]
},
{
@@ -163,7 +163,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/51210992e9234859a67a481860a7ca63-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/51210992e9234859a67a481860a7ca63-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5985bf8316574586b9b8139f6dce455d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5985bf8316574586b9b8139f6dce455d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 17 COLUMNS\n",
@@ -198,10 +198,10 @@
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.532, 'sens': 0.417, 'spec': 0.622, 'bacc': 0.519}\n",
"\n",
- "result = check_n_datasets_mos_known_folds_mos_scores(evaluations=evaluations,\n",
- " dataset_score_bounds={'sens': (0.8, 1.0)},\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_n_datasets_mos_known_folds_mos(evaluations=evaluations,\n",
+ " dataset_score_bounds={'sens': (0.8, 1.0)},\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
}
diff --git a/notebooks/illustration/07_check_n_datasets_mos_unknown_folds_mos.ipynb b/notebooks/illustration/binary/07_check_n_datasets_mos_unknown_folds_mos.ipynb
similarity index 83%
rename from notebooks/illustration/07_check_n_datasets_mos_unknown_folds_mos.ipynb
rename to notebooks/illustration/binary/07_check_n_datasets_mos_unknown_folds_mos.ipynb
index 8548602..b6ecfe7 100644
--- a/notebooks/illustration/07_check_n_datasets_mos_unknown_folds_mos.ipynb
+++ b/notebooks/illustration/binary/07_check_n_datasets_mos_unknown_folds_mos.ipynb
@@ -2,19 +2,20 @@
"cells": [
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
- "from mlscorecheck.check import check_n_datasets_mos_unknown_folds_mos_scores, estimate_n_experiments\n",
+ "from mlscorecheck.check.binary import (check_n_datasets_mos_unknown_folds_mos,\n",
+ " estimate_n_experiments)\n",
"from mlscorecheck.aggregated import generate_experiment, experiment_kfolds_generator"
]
},
{
"cell_type": "code",
- "execution_count": 12,
+ "execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
@@ -22,7 +23,7 @@
" rounding_decimals: int,\n",
" score_subset: list) -> (dict, dict):\n",
" \"\"\"\n",
- " Generate one random test case\n",
+ " Generate one random test case (a small one for illustration)\n",
"\n",
" Args:\n",
" random_seed (int): the random seed to use\n",
@@ -68,7 +69,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
@@ -79,7 +80,7 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": 9,
"metadata": {},
"outputs": [
{
@@ -90,7 +91,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d653504ba0ce4cb080ccc1652a531bfa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d653504ba0ce4cb080ccc1652a531bfa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/55c6c485ccb24157bd86782b3b667ce3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/55c6c485ccb24157bd86782b3b667ce3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -104,8 +105,8 @@
"Cgl0004I processed model has 4 rows, 10 columns (10 integer (2 of which binary)) and 30 elements\n",
"Cbc0031I 3 added rows had average density of 7.3333333\n",
"Cbc0013I At root node, 19 cuts changed objective from 0 to 0 in 66 passes\n",
- "Cbc0014I Cut generator 0 (Probing) - 28 row cuts average 2.4 elements, 1 column cuts (1 active) in 0.001 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 1 (Gomory) - 356 row cuts average 9.0 elements, 0 column cuts (0 active) in 0.002 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 0 (Probing) - 28 row cuts average 2.4 elements, 1 column cuts (1 active) in 0.002 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 356 row cuts average 9.0 elements, 0 column cuts (0 active) in 0.003 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 31 row cuts average 5.5 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
@@ -114,8 +115,8 @@
"Cbc0001I Search completed - best objective 1e+50, took 395 iterations and 0 nodes (0.02 seconds)\n",
"Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
"Cuts at root node changed objective from 0 to 0\n",
- "Probing was tried 66 times and created 29 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
- "Gomory was tried 66 times and created 356 cuts of which 0 were active after adding rounds of cuts (0.002 seconds)\n",
+ "Probing was tried 66 times and created 29 cuts of which 0 were active after adding rounds of cuts (0.002 seconds)\n",
+ "Gomory was tried 66 times and created 356 cuts of which 0 were active after adding rounds of cuts (0.003 seconds)\n",
"Knapsack was tried 66 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
"Clique was tried 66 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
"MixedIntegerRounding2 was tried 66 times and created 31 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
@@ -138,7 +139,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5ef017fd67042efb47ee433c64609d4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5ef017fd67042efb47ee433c64609d4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fed100c604e043f5b0b35f96f1ee18f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fed100c604e043f5b0b35f96f1ee18f2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -158,7 +159,7 @@
"Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 85 row cuts average 8.5 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is 1\n",
- "Cbc0014I Cut generator 5 (FlowCover) - 1 row cuts average 3.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 5 (FlowCover) - 1 row cuts average 3.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 6 (TwoMirCuts) - 83 row cuts average 6.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
"Cbc0001I Search completed - best objective 1e+50, took 256 iterations and 0 nodes (0.02 seconds)\n",
"Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
@@ -168,7 +169,7 @@
"Knapsack was tried 53 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
"Clique was tried 53 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
"MixedIntegerRounding2 was tried 53 times and created 85 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
- "FlowCover was tried 53 times and created 1 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 53 times and created 1 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
"TwoMirCuts was tried 53 times and created 83 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
"ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
"\n",
@@ -177,17 +178,17 @@
"No feasible solution found\n",
"Enumerated nodes: 0\n",
"Total iterations: 256\n",
- "Time (CPU seconds): 0.01\n",
+ "Time (CPU seconds): 0.02\n",
"Time (Wallclock seconds): 0.02\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.02\n",
+ "Total time (CPU seconds): 0.02 (Wallclock seconds): 0.02\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e4b29e4a9473452589b0a9115e623527-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e4b29e4a9473452589b0a9115e623527-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e82f39d49d314549ad6087b25b84346d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e82f39d49d314549ad6087b25b84346d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -232,7 +233,7 @@
"Time (Wallclock seconds): 0.03\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.03 (Wallclock seconds): 0.04\n",
+ "Total time (CPU seconds): 0.03 (Wallclock seconds): 0.03\n",
"\n"
]
},
@@ -242,7 +243,7 @@
"False"
]
},
- "execution_count": 14,
+ "execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
@@ -255,15 +256,15 @@
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.343}\n",
"\n",
- "result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
},
{
"cell_type": "code",
- "execution_count": 15,
+ "execution_count": 5,
"metadata": {},
"outputs": [
{
@@ -274,7 +275,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b57e9c77b7c44317981e7b2f1217c021-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b57e9c77b7c44317981e7b2f1217c021-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/23093f12b01e482b915cb03c1ee3d857-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/23093f12b01e482b915cb03c1ee3d857-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -292,7 +293,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/15598fafca034066803c919870c78b9f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/15598fafca034066803c919870c78b9f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bdd54772e22c4ceb9b39ce594dbd1ae8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bdd54772e22c4ceb9b39ce594dbd1ae8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -310,7 +311,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/62e7070fdd3c471e8eb08bffe960e896-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/62e7070fdd3c471e8eb08bffe960e896-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db0bbbb5619d4f55ad479bdaffd76d88-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db0bbbb5619d4f55ad479bdaffd76d88-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -328,7 +329,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc2779b6254644beb772058bd12bf6fd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc2779b6254644beb772058bd12bf6fd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/079658630ecd4616a2b0683082df280b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/079658630ecd4616a2b0683082df280b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -346,7 +347,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/34abc58e86e94ae2a58ea1b666721b4e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/34abc58e86e94ae2a58ea1b666721b4e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae22778f2e744357ace74d50d2a9d9dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae22778f2e744357ace74d50d2a9d9dd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -364,7 +365,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e139d893b8d443eb4a931f2500bcc2d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e139d893b8d443eb4a931f2500bcc2d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57aa49d6eae144ba8626e875ea20ce57-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57aa49d6eae144ba8626e875ea20ce57-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -382,7 +383,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/198202d992684e26a8b6d73358fbc907-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/198202d992684e26a8b6d73358fbc907-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a02a3e7efec45dc8b07c76ed0b10d40-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a02a3e7efec45dc8b07c76ed0b10d40-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -400,7 +401,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a3447325704b4bf2a0be6cd23db2a1e6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a3447325704b4bf2a0be6cd23db2a1e6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/519dbef4e720491b89f1d016cff48d57-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/519dbef4e720491b89f1d016cff48d57-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -418,7 +419,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d81dfbbc5e34da7a430d702715201c3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d81dfbbc5e34da7a430d702715201c3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4f9affb54776412bad2ef176c4367b2e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4f9affb54776412bad2ef176c4367b2e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -436,7 +437,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8843a76ed9341c7b6bc02ff10480009-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8843a76ed9341c7b6bc02ff10480009-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8c1db037bb7c4d4fb89ad2d83efd4d31-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8c1db037bb7c4d4fb89ad2d83efd4d31-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -454,7 +455,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/97c2c9a1a73f4e9185e5c20d05e500f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/97c2c9a1a73f4e9185e5c20d05e500f9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cf35336d489e4ef98ebcbf4a72876421-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cf35336d489e4ef98ebcbf4a72876421-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -472,7 +473,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cafdae8c00e5458c900355369b622292-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cafdae8c00e5458c900355369b622292-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46f58515ff6643439dbb589c0f77a3a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46f58515ff6643439dbb589c0f77a3a9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -490,7 +491,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b0e18bf6b2584758ab7eccb48651d684-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b0e18bf6b2584758ab7eccb48651d684-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/afc3bf4f745846fd978dc32443f76f2a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/afc3bf4f745846fd978dc32443f76f2a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -508,7 +509,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/26efa7c4db3b4a88bdd4f764becbec11-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/26efa7c4db3b4a88bdd4f764becbec11-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e3412658131d41659b256704637e32b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e3412658131d41659b256704637e32b8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -526,7 +527,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2059520b794141e69b372e5a18a4ebcc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2059520b794141e69b372e5a18a4ebcc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eaf04ac4b7c54baaa71be6118a7ef69d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eaf04ac4b7c54baaa71be6118a7ef69d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -544,7 +545,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b18fa88dda5640d39eaa2603dd166575-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b18fa88dda5640d39eaa2603dd166575-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c70ebd30e3f34658903af7ffdd600aa3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c70ebd30e3f34658903af7ffdd600aa3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -562,7 +563,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/59bec70ea2594e1aa6472c57fb906c84-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/59bec70ea2594e1aa6472c57fb906c84-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8cb6a9d03fe343d0a1eb6ef7ecd6f866-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8cb6a9d03fe343d0a1eb6ef7ecd6f866-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -580,7 +581,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ca535e076384159b4c754a44f09fabb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ca535e076384159b4c754a44f09fabb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/466beb8a16424366b6580e5561520e3e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/466beb8a16424366b6580e5561520e3e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -598,7 +599,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e9110f66c8745dd9b734e7854ecdd66-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e9110f66c8745dd9b734e7854ecdd66-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a1eae64cbfeb4876bf62bf6a9ddb763b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a1eae64cbfeb4876bf62bf6a9ddb763b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -616,7 +617,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1bcb344d4f7944e18c4edd688a83aa5f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1bcb344d4f7944e18c4edd688a83aa5f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/28216f90e66947638f4740255da561ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/28216f90e66947638f4740255da561ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -634,7 +635,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bb5a131af0f146bb9bff1fd8a9a6e1f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bb5a131af0f146bb9bff1fd8a9a6e1f5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/393dd328cbab4ad99be5cd38000793d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/393dd328cbab4ad99be5cd38000793d6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -652,7 +653,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/34345363aabd4c33b05726f187c08530-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/34345363aabd4c33b05726f187c08530-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4ecb9c8b45bd42079c359977ab0a8fcd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4ecb9c8b45bd42079c359977ab0a8fcd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -670,7 +671,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9e7df62dd454e1d9ec389f5218e6961-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9e7df62dd454e1d9ec389f5218e6961-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db905d5bbb234d57887e56ad405283e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db905d5bbb234d57887e56ad405283e9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -688,7 +689,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c02a337bc76f4e168d016fcd700d3d15-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c02a337bc76f4e168d016fcd700d3d15-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/541c91b54bda451c96be05a0349ee18e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/541c91b54bda451c96be05a0349ee18e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -706,7 +707,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b2e61d716e0749fbb0e3279772205931-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b2e61d716e0749fbb0e3279772205931-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/375d9f6e107b48ce93c7b7d88b45d877-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/375d9f6e107b48ce93c7b7d88b45d877-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -724,7 +725,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ac24836ece4a447b950ff1f34b07be4a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ac24836ece4a447b950ff1f34b07be4a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a280a2df1234928966c8d7cb7a23e73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a280a2df1234928966c8d7cb7a23e73-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -742,7 +743,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a6e82ecb6d16452ab0d12ceb8d91aae6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a6e82ecb6d16452ab0d12ceb8d91aae6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b6489e1ac6204ab9a102e3bf7eeead6b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b6489e1ac6204ab9a102e3bf7eeead6b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -760,7 +761,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f125a350ccd5407c972824ed1c417b8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f125a350ccd5407c972824ed1c417b8f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68dd3562279641bb9661de2a5141a561-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68dd3562279641bb9661de2a5141a561-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -778,7 +779,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32674d6ce68047caac91c895324b7fd6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32674d6ce68047caac91c895324b7fd6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1bf38c3ef7fb4855a0f038390eea9831-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1bf38c3ef7fb4855a0f038390eea9831-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -796,7 +797,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/094d25c6e3884108ac42ea4b198acf75-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/094d25c6e3884108ac42ea4b198acf75-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d4ce0e8ba84741a7a2ae24ae0b73115e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d4ce0e8ba84741a7a2ae24ae0b73115e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -814,7 +815,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8273d7b54b7540f7bf8e6d988d26120c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8273d7b54b7540f7bf8e6d988d26120c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b92fe8ff557f42dfbf75cf82a3a06775-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b92fe8ff557f42dfbf75cf82a3a06775-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -832,7 +833,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/52e074a7eba241dd87f8428be3bb3311-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/52e074a7eba241dd87f8428be3bb3311-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0e760e6a0f6542d3898f5f2d3f7a8065-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0e760e6a0f6542d3898f5f2d3f7a8065-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -850,7 +851,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c59f0965d5e47999fb53458d198cd58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c59f0965d5e47999fb53458d198cd58-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4b4a5940c0de4046a8d471e3ae547891-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4b4a5940c0de4046a8d471e3ae547891-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -868,7 +869,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4d19b817483c4bf78f667217b13c3319-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4d19b817483c4bf78f667217b13c3319-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3fe183081c1480a8df2e8f35f47d866-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3fe183081c1480a8df2e8f35f47d866-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -886,7 +887,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c2519a8d1e7f49488b07bf8f2e59a84b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c2519a8d1e7f49488b07bf8f2e59a84b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb94fa6895724dd78f52f7cd7475cc42-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb94fa6895724dd78f52f7cd7475cc42-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -904,7 +905,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57e6c619d4904f64bf19a0c14b5e58fc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57e6c619d4904f64bf19a0c14b5e58fc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1fe0a56ff16549d5a6a5ac052d0f7d85-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1fe0a56ff16549d5a6a5ac052d0f7d85-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -922,7 +923,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7d88a5da00d64290b26a200adca94040-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7d88a5da00d64290b26a200adca94040-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/40bb20e0c08c4553a6c3d754168c9012-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/40bb20e0c08c4553a6c3d754168c9012-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -940,7 +941,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1a022ca1a2ab4af0a062ed963c9659c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1a022ca1a2ab4af0a062ed963c9659c0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cee70028a77a45d7b6ee6a6ce91c91a6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cee70028a77a45d7b6ee6a6ce91c91a6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -958,7 +959,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a2247d21885f40879bfceb9ed4ccd4f0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a2247d21885f40879bfceb9ed4ccd4f0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/310f015955384a5082e924861d53acec-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/310f015955384a5082e924861d53acec-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -976,7 +977,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b511407b856f4bc5a4cb759aceecaf16-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b511407b856f4bc5a4cb759aceecaf16-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f94cea2531654e7ca71a6ffbb850475c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f94cea2531654e7ca71a6ffbb850475c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -994,7 +995,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02217a9df1ac49e3aee81afe7da2bfdb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02217a9df1ac49e3aee81afe7da2bfdb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d2594ead766044b98493fec26f038f4e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d2594ead766044b98493fec26f038f4e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1012,7 +1013,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f7ecab44b3b9475fb16de6a4f0c56840-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f7ecab44b3b9475fb16de6a4f0c56840-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aa17961335dc4cc28a4433a3476452cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aa17961335dc4cc28a4433a3476452cf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1030,7 +1031,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cdbfa26ea1354c04bd83308ff6a66b1f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cdbfa26ea1354c04bd83308ff6a66b1f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/412119882007463fb71e6195b214ad88-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/412119882007463fb71e6195b214ad88-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1048,7 +1049,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/abc58a10ac864ff282d3ff59f8c899ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/abc58a10ac864ff282d3ff59f8c899ef-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9e60440836ee4a19a4e8289229d084d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9e60440836ee4a19a4e8289229d084d9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1066,7 +1067,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f5ceb2c455c4605933be1d1f2089f16-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f5ceb2c455c4605933be1d1f2089f16-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e83cdc426c0844f889a2e445530e950f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e83cdc426c0844f889a2e445530e950f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1084,7 +1085,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e53a3e500cd0482b97bf0436c8c3ce5f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e53a3e500cd0482b97bf0436c8c3ce5f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4562f3a1b9ac419b8a8d2946787db8b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4562f3a1b9ac419b8a8d2946787db8b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1102,7 +1103,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f18735bd9f8345fe9b448169cebf9c89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f18735bd9f8345fe9b448169cebf9c89-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/60a3d7709a484fd990ae25b47e2b0105-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/60a3d7709a484fd990ae25b47e2b0105-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1115,18 +1116,12 @@
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
"Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
- "\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
+ "\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/74ba6b3661e94eeead35589e53b3f54b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/74ba6b3661e94eeead35589e53b3f54b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f45b9a020bb04ddd91207b1c675880a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f45b9a020bb04ddd91207b1c675880a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1144,7 +1139,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/27fb1c91f55e4b8c8f6fad7b4a586bc5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/27fb1c91f55e4b8c8f6fad7b4a586bc5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/91a2b71cc19840eb8dba9e12d419fa76-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/91a2b71cc19840eb8dba9e12d419fa76-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1162,7 +1157,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fa66366485764fab950eba0818ff1544-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fa66366485764fab950eba0818ff1544-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2f51f291f2fb4f548f59934760809631-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2f51f291f2fb4f548f59934760809631-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1180,7 +1175,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d295e745d0cb4c7caa0ba0b386294478-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d295e745d0cb4c7caa0ba0b386294478-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b0c3c1288f3743689776661e0ab9a6bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b0c3c1288f3743689776661e0ab9a6bd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1198,7 +1193,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/025ba3da5cca4117b48c00df72f0147f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/025ba3da5cca4117b48c00df72f0147f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/01abf9cb5b4f4645974c82012bba82a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/01abf9cb5b4f4645974c82012bba82a9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1216,7 +1211,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bbbd42bfc47046fc97ac8e98c8b187e8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bbbd42bfc47046fc97ac8e98c8b187e8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fe5b994445c148cd9cb76a8ae92546c3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fe5b994445c148cd9cb76a8ae92546c3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1234,7 +1229,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ad7e9d79a6944b51827c56b8291212bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ad7e9d79a6944b51827c56b8291212bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f59bf623fcd4da5b59faa367e46c775-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f59bf623fcd4da5b59faa367e46c775-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1252,7 +1247,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db432cfe49ee45f8a362ffbc01bb9625-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db432cfe49ee45f8a362ffbc01bb9625-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9eb3cc2799e344e4bce90bf2239f49f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9eb3cc2799e344e4bce90bf2239f49f9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1270,7 +1265,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7172143035c646e586ae138c4660e7f0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7172143035c646e586ae138c4660e7f0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1dcf94b9c8004b3f99abf8934612e4b0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1dcf94b9c8004b3f99abf8934612e4b0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1288,7 +1283,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/126b65a895d8463db1a7d0e075ec7961-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/126b65a895d8463db1a7d0e075ec7961-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a94dc53ee3d4424f829088846f63f666-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a94dc53ee3d4424f829088846f63f666-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1306,7 +1301,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2620bdef85df4223955f5683c5a96ad8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2620bdef85df4223955f5683c5a96ad8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/58eb0804cad04daa9fe49967fe97eb79-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/58eb0804cad04daa9fe49967fe97eb79-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1324,7 +1319,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8768e555565646bc8fbce6acadbaa892-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8768e555565646bc8fbce6acadbaa892-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3dde636948f4bf5b228a54291243f8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3dde636948f4bf5b228a54291243f8c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1342,7 +1337,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f7e6ff58f40c485fac1b5006f785e192-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f7e6ff58f40c485fac1b5006f785e192-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/95dd9c10c7c84e2b82442da5274a9aaf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/95dd9c10c7c84e2b82442da5274a9aaf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1360,7 +1355,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/069d71669ce74cad9a89107f0ec25e0c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/069d71669ce74cad9a89107f0ec25e0c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cce3178ffb024d4b80fc0c3d6d478e09-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cce3178ffb024d4b80fc0c3d6d478e09-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1378,7 +1373,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc2d3f9ccb6c4852be29851147602a5c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc2d3f9ccb6c4852be29851147602a5c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0aeade5564f5404287bf2078b0732cb9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0aeade5564f5404287bf2078b0732cb9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1396,7 +1391,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24d97df2890e44e48265d929e630bb8a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24d97df2890e44e48265d929e630bb8a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d29bfdb4ce40425c84a4eaef31bed7d5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d29bfdb4ce40425c84a4eaef31bed7d5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1414,7 +1409,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53dc00251ab443eea1dcc8c11d0531f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53dc00251ab443eea1dcc8c11d0531f9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3d614537cada4eb697d0735cb111e4a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3d614537cada4eb697d0735cb111e4a4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1432,7 +1427,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c3ab5ba471224d248fd68aa935f471c3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c3ab5ba471224d248fd68aa935f471c3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/342fb309275a4e02934731c3c23bc72f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/342fb309275a4e02934731c3c23bc72f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1450,7 +1445,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a17e94e8950c4cfc84199c3a871ff350-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a17e94e8950c4cfc84199c3a871ff350-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/44cdd0a08fba402ab38499af0cbc140e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/44cdd0a08fba402ab38499af0cbc140e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1468,7 +1463,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e03de38a22674c319f7e4ba8d145c642-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e03de38a22674c319f7e4ba8d145c642-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b036c6f2a203437a9d106c7bb9164df4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b036c6f2a203437a9d106c7bb9164df4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1486,7 +1481,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/efbee9c2490f461bb06c0a7cff47827a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/efbee9c2490f461bb06c0a7cff47827a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68edbc3b857b4eb1a873e48a3e8bf9a7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68edbc3b857b4eb1a873e48a3e8bf9a7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1504,7 +1499,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b0a10642db394ca285a9b77cef284723-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b0a10642db394ca285a9b77cef284723-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/219740b819ac4a0494b04fb573ec3abf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/219740b819ac4a0494b04fb573ec3abf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1522,7 +1517,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ebaa9a48a2804c718b4c4ad323ed91c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ebaa9a48a2804c718b4c4ad323ed91c0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f731cc3f1e6344f1b8b1262d54fb19fb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f731cc3f1e6344f1b8b1262d54fb19fb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1540,7 +1535,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a511bd02d5f497893972a573d04de4e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a511bd02d5f497893972a573d04de4e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/58c3bcbe687649e3add14f5318d08cca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/58c3bcbe687649e3add14f5318d08cca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1558,7 +1553,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c3b12961eea4ab3bd73f5435cbf540a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c3b12961eea4ab3bd73f5435cbf540a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/08fc6a2101954edcb946e9a93b22de96-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/08fc6a2101954edcb946e9a93b22de96-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1576,7 +1571,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/684499b789ce475a807cff216b95d144-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/684499b789ce475a807cff216b95d144-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a0d49202e3354438aed88d9d5cd751a2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a0d49202e3354438aed88d9d5cd751a2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1594,7 +1589,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/22d779641b284968b4646bacd7cdcf72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/22d779641b284968b4646bacd7cdcf72-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bb09c23563294e4d9d0f278d986b498b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bb09c23563294e4d9d0f278d986b498b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1612,7 +1607,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/55393829b85d4a81934a8fc754be7f14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/55393829b85d4a81934a8fc754be7f14-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0070b91292d74f909758c60f93c9a698-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0070b91292d74f909758c60f93c9a698-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1630,7 +1625,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/21f599ab0ed3400bad94c412fd8c0d20-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/21f599ab0ed3400bad94c412fd8c0d20-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9f1cc7c8f17e49cc97e7c557cecafab4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9f1cc7c8f17e49cc97e7c557cecafab4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1648,7 +1643,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1983452c55c48558c814b2268219aba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1983452c55c48558c814b2268219aba-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9fbac9fec9c2441db9f196ceb9978be9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9fbac9fec9c2441db9f196ceb9978be9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1666,7 +1661,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/12ddc35f4dc149ffa37a9c068ad7ad4f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/12ddc35f4dc149ffa37a9c068ad7ad4f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2f8ed6583da04f05b2e7e598f13352d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2f8ed6583da04f05b2e7e598f13352d2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1684,7 +1679,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/530e720cd4bd45f39f45d87d0da25551-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/530e720cd4bd45f39f45d87d0da25551-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/407984eba2a748249628d402b65637d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/407984eba2a748249628d402b65637d6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1702,7 +1697,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f46d6afb7f0a4e3cb86bca8337d0bce2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f46d6afb7f0a4e3cb86bca8337d0bce2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f54caace76744ab69b87711fe62b8337-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f54caace76744ab69b87711fe62b8337-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1720,7 +1715,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b406bd1b7964263998681dc4d5204d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b406bd1b7964263998681dc4d5204d0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f119458f27064763b66cf31134391c1a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f119458f27064763b66cf31134391c1a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1738,7 +1733,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/17750a6ca6d14832b9ca4401a8c9e767-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/17750a6ca6d14832b9ca4401a8c9e767-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/30b7d7027b054332880b198590f00ec9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/30b7d7027b054332880b198590f00ec9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1756,7 +1751,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc19f1a9bcac49a981d6b424a14f95cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc19f1a9bcac49a981d6b424a14f95cf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/256f8566f84e4f5ba33d08baf743a232-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/256f8566f84e4f5ba33d08baf743a232-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1774,7 +1769,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae07714a834b4769a109dcbef741db1c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae07714a834b4769a109dcbef741db1c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d4e1ae7285c416a9ce6c1093ab80e8a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d4e1ae7285c416a9ce6c1093ab80e8a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1792,7 +1787,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/49d007b8b9de47949689a7e11e960674-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/49d007b8b9de47949689a7e11e960674-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a537bd71063b41899fafc500e26f91fb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a537bd71063b41899fafc500e26f91fb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1810,7 +1805,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd4b1906605d441f9348b80fe9b75788-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd4b1906605d441f9348b80fe9b75788-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0f94ffe323b943ed9756dcc64a2176da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0f94ffe323b943ed9756dcc64a2176da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1828,7 +1823,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e643cea499cc438b886705c0fcd93e77-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e643cea499cc438b886705c0fcd93e77-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b72be2e0a834d21a28895b37446176e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b72be2e0a834d21a28895b37446176e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1846,7 +1841,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e73e41bf74b243efaddc52b3989ec1d5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e73e41bf74b243efaddc52b3989ec1d5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/796e90be2c7c4f97b5d07deb6492f29b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/796e90be2c7c4f97b5d07deb6492f29b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1864,7 +1859,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e4384c97e7054751bce65be5ec02898e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e4384c97e7054751bce65be5ec02898e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1360c5305fb24b89bce685c28cd89bb6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1360c5305fb24b89bce685c28cd89bb6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1882,7 +1877,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/93fdff1e0e0a46db9e3790fa96e68c39-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/93fdff1e0e0a46db9e3790fa96e68c39-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1e89ea13d0624e0ebbb76d08eeb4615b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1e89ea13d0624e0ebbb76d08eeb4615b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1900,7 +1895,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f573087e6d9a473b9c8197d8eb959b6c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f573087e6d9a473b9c8197d8eb959b6c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/11808b9815e345d5a990e3e3c41ecb50-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/11808b9815e345d5a990e3e3c41ecb50-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1918,7 +1913,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8ef046b0f30470980e89b4de16cca03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8ef046b0f30470980e89b4de16cca03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cef72d14eb664acdb80d3ddc2c53e274-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cef72d14eb664acdb80d3ddc2c53e274-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1936,7 +1931,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1101178d4e649bd970f72445964f375-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1101178d4e649bd970f72445964f375-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/736137c1b70846b6ac02d6d3b0676769-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/736137c1b70846b6ac02d6d3b0676769-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1954,7 +1949,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8fc9475a80c148389df01cc8ea8815a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8fc9475a80c148389df01cc8ea8815a9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c94f7ae874324a12bb1b2c428fdd1463-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c94f7ae874324a12bb1b2c428fdd1463-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1972,7 +1967,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dd48d70be4ea493199f4650caed6bc41-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dd48d70be4ea493199f4650caed6bc41-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/232d1bcc207e4e59b80e7998fa4c6d52-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/232d1bcc207e4e59b80e7998fa4c6d52-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -1990,7 +1985,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b016bcb92f9c46d78e01df78eaa28bdf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b016bcb92f9c46d78e01df78eaa28bdf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a7c1141c4820496c87e5116dffcfae86-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a7c1141c4820496c87e5116dffcfae86-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2008,7 +2003,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f28c5361bc8c4e21bd63776bbe401a0f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f28c5361bc8c4e21bd63776bbe401a0f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42d53183ecb94ffcba49894a697c665d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42d53183ecb94ffcba49894a697c665d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2026,7 +2021,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0b4c73192cd94a6fa8ad755a59061a03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0b4c73192cd94a6fa8ad755a59061a03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5db009ce416c4de4bf118f41d8bb8b9e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5db009ce416c4de4bf118f41d8bb8b9e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2044,7 +2039,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6517b7323f9a4f1b956176b5aa40042c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6517b7323f9a4f1b956176b5aa40042c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78ebe7e2aee340c2a4ef85ed5f5bdf66-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78ebe7e2aee340c2a4ef85ed5f5bdf66-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2062,7 +2057,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a99abe43060040aa8e8f6644f10ffd8a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a99abe43060040aa8e8f6644f10ffd8a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6bba530d4b7a4b088a83cef4367a8c6d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6bba530d4b7a4b088a83cef4367a8c6d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2080,7 +2075,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dee60752bbff49e08e0be4c32d7b64b2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dee60752bbff49e08e0be4c32d7b64b2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fa7f67a4a7054435a02b03cb8c6d46b1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fa7f67a4a7054435a02b03cb8c6d46b1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2098,7 +2093,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5890444ea61b4f05b7f2d36f9044d33a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5890444ea61b4f05b7f2d36f9044d33a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/686cd97c810641ada35a3239d354926e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/686cd97c810641ada35a3239d354926e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2116,7 +2111,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/52e74e6ecf5748f58d9be8ab609ae8dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/52e74e6ecf5748f58d9be8ab609ae8dd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6abcc104f7f046b38d37e17d54facec5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6abcc104f7f046b38d37e17d54facec5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2134,7 +2129,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc526f6be79d47d0b6a710638fa86b30-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc526f6be79d47d0b6a710638fa86b30-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b838c45c6fa14b1693266d40a86e68ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b838c45c6fa14b1693266d40a86e68ef-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2152,7 +2147,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9641f169d12540fdb2e6d0f10e888fae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9641f169d12540fdb2e6d0f10e888fae-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7127900fa0444dc881610d773e1f1353-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7127900fa0444dc881610d773e1f1353-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2170,7 +2165,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/38049d794d7c433687cb4982f51b5e73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/38049d794d7c433687cb4982f51b5e73-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4cbad13ccffe45edb6244ce85dfe3029-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4cbad13ccffe45edb6244ce85dfe3029-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2188,7 +2183,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a0cca23b1db941b782784b827763a5d5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a0cca23b1db941b782784b827763a5d5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4b8ef45427174ff093b784ecbe757a92-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4b8ef45427174ff093b784ecbe757a92-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2206,7 +2201,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/034391e073fe4a0a87ee2b5a67bf4ab2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/034391e073fe4a0a87ee2b5a67bf4ab2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0c91ef81da704db2ac5fbc6d3fa5a8e8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0c91ef81da704db2ac5fbc6d3fa5a8e8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2224,7 +2219,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a4f7f7f386af41ff8cf567f0dcda907c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a4f7f7f386af41ff8cf567f0dcda907c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2bcefe2b908649f68a604582467239a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2bcefe2b908649f68a604582467239a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2242,7 +2237,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e194d3510cec4f969a0b3bd5a7b3a21a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e194d3510cec4f969a0b3bd5a7b3a21a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/748d20472727493194212955601d0590-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/748d20472727493194212955601d0590-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2260,7 +2255,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f55ccc2ce8ae4579964049ec1a81f146-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f55ccc2ce8ae4579964049ec1a81f146-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/84b9cba56871416bbc92e5bbc0ba485f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/84b9cba56871416bbc92e5bbc0ba485f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2278,7 +2273,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/83291c780aa14b6485b6ed665a64285c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/83291c780aa14b6485b6ed665a64285c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/34691c960dd140a98561bb9bd66e41ac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/34691c960dd140a98561bb9bd66e41ac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2296,7 +2291,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42b6fff605fb46c99049457f45768c80-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42b6fff605fb46c99049457f45768c80-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c9929b0d775044b3ab345d37879460a5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c9929b0d775044b3ab345d37879460a5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2314,7 +2309,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4c80fbd1fe864041ad4b2abf623af24b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4c80fbd1fe864041ad4b2abf623af24b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32d6ab55ca514f7993eb93504e331d3e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32d6ab55ca514f7993eb93504e331d3e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2332,7 +2327,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/06af18e2d7734931ac4e5e1b4cd5b249-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/06af18e2d7734931ac4e5e1b4cd5b249-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cff9a8d04cf84aa98dee5ccda881b13b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cff9a8d04cf84aa98dee5ccda881b13b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2350,7 +2345,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a504f47634094bedaae30daa56539801-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a504f47634094bedaae30daa56539801-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba8f00963edb4e498b36c98bf850f3c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba8f00963edb4e498b36c98bf850f3c5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2368,7 +2363,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/441fb098fb6b41a2a7c146964e489464-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/441fb098fb6b41a2a7c146964e489464-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/65233d66b3b342d9a75c2b1b963d237b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/65233d66b3b342d9a75c2b1b963d237b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2386,7 +2381,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3879e2074bcf4ab1999abf3b251f8b6e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3879e2074bcf4ab1999abf3b251f8b6e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b9b21d43abfb40dfa4af482525fa20fa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b9b21d43abfb40dfa4af482525fa20fa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2404,7 +2399,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/972d4f589fda4057a2175bfe887da8c2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/972d4f589fda4057a2175bfe887da8c2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d262a763e21f4b3495ec7414fa5e4849-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d262a763e21f4b3495ec7414fa5e4849-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2422,7 +2417,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/765a9d0300ac438c88696329979ca511-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/765a9d0300ac438c88696329979ca511-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef890e60e5ce4829b56133004b6df4ad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef890e60e5ce4829b56133004b6df4ad-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2440,7 +2435,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53cfc1e8d94344e7abba55c4d7531035-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53cfc1e8d94344e7abba55c4d7531035-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53a647c67c98495da29231b05913d6a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53a647c67c98495da29231b05913d6a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2458,7 +2453,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a78f4d034b1d46c5b332f6b52d4ae4a2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a78f4d034b1d46c5b332f6b52d4ae4a2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a7c5dacc8154fa7a6b47bf5e5a311b0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a7c5dacc8154fa7a6b47bf5e5a311b0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2476,7 +2471,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae7cd408342a4219978f96504f7e8f54-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae7cd408342a4219978f96504f7e8f54-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4680b192a9234eb5abeddef936e82d11-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4680b192a9234eb5abeddef936e82d11-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2494,7 +2489,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b785fc1dfc6a4113b2973b725d28cd8a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b785fc1dfc6a4113b2973b725d28cd8a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/25003bb0871a43aaa251a7080669730c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/25003bb0871a43aaa251a7080669730c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2512,7 +2507,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/10eb4d48123a45ebb01a05ba07f45c84-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/10eb4d48123a45ebb01a05ba07f45c84-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e15a59c081c4ff2a7690a7a45cf9578-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e15a59c081c4ff2a7690a7a45cf9578-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2530,7 +2525,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b600bda52e3948caa02c8c7ba8ed9fee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b600bda52e3948caa02c8c7ba8ed9fee-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8aed1c1fa05740988e30291f7ae747dc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8aed1c1fa05740988e30291f7ae747dc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2548,7 +2543,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/06445bafbb5045c8bfbd709aaa761b30-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/06445bafbb5045c8bfbd709aaa761b30-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fef51fbea2bb4cafba45c3aefa2648a0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fef51fbea2bb4cafba45c3aefa2648a0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2566,7 +2561,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/228cb5e5895243ee828b817302cf33b4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/228cb5e5895243ee828b817302cf33b4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/042fe4afb8cb4392b9feb5d62b72910f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/042fe4afb8cb4392b9feb5d62b72910f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2584,7 +2579,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f06b8b9a7f240459879863e3cb22701-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f06b8b9a7f240459879863e3cb22701-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/63929188fd574e279fcf480ed035db76-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/63929188fd574e279fcf480ed035db76-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2602,7 +2597,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dcd8479162de42f0b37b20b37914c4f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dcd8479162de42f0b37b20b37914c4f3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/18add36462ac4727b78c06a748fa001c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/18add36462ac4727b78c06a748fa001c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2620,7 +2615,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1100ede1c59b400f8add267988d30586-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1100ede1c59b400f8add267988d30586-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3ad0bb63304e43a4b5abb02697c3cb10-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3ad0bb63304e43a4b5abb02697c3cb10-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2638,7 +2633,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/653ecf819919416588f059d28276f02d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/653ecf819919416588f059d28276f02d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1dd7ff6925cb4987a68a96bb894a3daf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1dd7ff6925cb4987a68a96bb894a3daf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2656,7 +2651,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f32aa0adcad949a696dbb76c57c7ea06-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f32aa0adcad949a696dbb76c57c7ea06-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/443dad444dac4c7fb4e10b16b75f9d77-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/443dad444dac4c7fb4e10b16b75f9d77-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2674,7 +2669,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87a52eb638944df1be25e0d582db5129-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87a52eb638944df1be25e0d582db5129-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/604afdfa1ecf4c1389351814ac5c6ef1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/604afdfa1ecf4c1389351814ac5c6ef1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2692,7 +2687,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1316bde0493048f8a7fae38bc2ccfd1f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1316bde0493048f8a7fae38bc2ccfd1f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/186d303d0e9e40999bd0f097336873a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/186d303d0e9e40999bd0f097336873a4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2710,7 +2705,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4a1f9e9caa65459d83701032e650013b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4a1f9e9caa65459d83701032e650013b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/928d4c1458f84dca90b63cad79704dd6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/928d4c1458f84dca90b63cad79704dd6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2728,7 +2723,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/585259875e9f46c7ba677538ba7da3cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/585259875e9f46c7ba677538ba7da3cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a83c8d730f1443c3a1ed15c05cf63f61-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a83c8d730f1443c3a1ed15c05cf63f61-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2746,7 +2741,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d369d413df204897b87109d8ed4cbb3e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d369d413df204897b87109d8ed4cbb3e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2df9f35cb0f44a73bd84e944cf7ee42e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2df9f35cb0f44a73bd84e944cf7ee42e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2764,7 +2759,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a441a6faa8b446478ec3b3b578d0b00b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a441a6faa8b446478ec3b3b578d0b00b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/71bdb4191a5b40d08214fa68c823779b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/71bdb4191a5b40d08214fa68c823779b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2782,7 +2777,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a1c49c14f1a74edc838793a9c79627c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a1c49c14f1a74edc838793a9c79627c0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cf1133d958454f91b102f8ac4d460d45-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cf1133d958454f91b102f8ac4d460d45-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2800,7 +2795,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/df7541621abe4f55b232da5e07be0d80-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/df7541621abe4f55b232da5e07be0d80-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2ced2f18874c4fb999eadc85add9e593-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2ced2f18874c4fb999eadc85add9e593-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2818,7 +2813,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/144bb6b9170c4ef284baa2240cd2bc07-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/144bb6b9170c4ef284baa2240cd2bc07-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/18dd10093ffc496487e8497cd82b0569-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/18dd10093ffc496487e8497cd82b0569-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2836,7 +2831,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6fe7c9999579448981eb8bae9864d058-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6fe7c9999579448981eb8bae9864d058-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/40eacf6832c647f9a5c434ddd116bbcf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/40eacf6832c647f9a5c434ddd116bbcf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2854,7 +2849,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fea1206933f2435fabb71343aad4a526-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fea1206933f2435fabb71343aad4a526-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ed242c4c9b8e46e28a4a3f273d8449ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ed242c4c9b8e46e28a4a3f273d8449ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2872,7 +2867,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3af2484543e14cfbb9978d63c4cd6bcc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3af2484543e14cfbb9978d63c4cd6bcc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aa3a7359555042598cc62488f83e7f2c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aa3a7359555042598cc62488f83e7f2c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2890,7 +2885,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/34f46d4b06094809a7d1c2b17cfefd08-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/34f46d4b06094809a7d1c2b17cfefd08-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/358588484a3d4d32ae4d788c1b2ba6a3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/358588484a3d4d32ae4d788c1b2ba6a3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2908,7 +2903,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d85213160fa4b468508e537dd60635e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d85213160fa4b468508e537dd60635e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0aafcf320e7f44e1903d04fc937663b4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0aafcf320e7f44e1903d04fc937663b4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2926,7 +2921,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d45a6a843d354e81be652897d7431a77-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d45a6a843d354e81be652897d7431a77-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/995755b5a4a54a80a75a184d5b2bfe22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/995755b5a4a54a80a75a184d5b2bfe22-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2944,7 +2939,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e558e6ce838c40b2b74704c9cfa8f41c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e558e6ce838c40b2b74704c9cfa8f41c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/729b1f2e203b480483b7b24925411fa4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/729b1f2e203b480483b7b24925411fa4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2962,7 +2957,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d681550294c546f68c19b7ef76eb3102-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d681550294c546f68c19b7ef76eb3102-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5bd6817ceafb469a96dbb0a8eae715f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5bd6817ceafb469a96dbb0a8eae715f3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2980,7 +2975,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82310f4d2afb445b91a6d3c9a7a759bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82310f4d2afb445b91a6d3c9a7a759bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/254bcb4b40404789853175466be0a699-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/254bcb4b40404789853175466be0a699-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -2998,7 +2993,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fb3560ee6c754017b71a87fbb34b2fb0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fb3560ee6c754017b71a87fbb34b2fb0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c46d7afd763479eae8f8c7a6608373d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c46d7afd763479eae8f8c7a6608373d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3016,7 +3011,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/abaf21ba5ac14c86a2b742be9a1f696b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/abaf21ba5ac14c86a2b742be9a1f696b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b365971578054e2a9c2bd69e82139aea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b365971578054e2a9c2bd69e82139aea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3034,7 +3029,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6de0af156f1c433395c8890ce9abd0ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6de0af156f1c433395c8890ce9abd0ee-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de703bc0df0546c1a14c09dfe4a6bac8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de703bc0df0546c1a14c09dfe4a6bac8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3052,7 +3047,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/83c6dd419c4a4267a7808b47f1b3ace5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/83c6dd419c4a4267a7808b47f1b3ace5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ff5eeeebfc414ffc89f965d6906bf3b3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ff5eeeebfc414ffc89f965d6906bf3b3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3070,7 +3065,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7c275c5be8cb45639c503188f98e9701-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7c275c5be8cb45639c503188f98e9701-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f043a20dd4124380b0916f13cbb22609-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f043a20dd4124380b0916f13cbb22609-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3088,7 +3083,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0aa0e61422f34bdaa8c446d9f90e14c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0aa0e61422f34bdaa8c446d9f90e14c0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e59b2d937de14ce89be0031f5b19018d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e59b2d937de14ce89be0031f5b19018d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3106,7 +3101,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5c38c9222fcf4c09b1d53fd512167f5c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5c38c9222fcf4c09b1d53fd512167f5c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/92aa79139b104b5186b7d29fb89036ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/92aa79139b104b5186b7d29fb89036ca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3124,7 +3119,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef2dec909c8440ccab38f6c742b2a43d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef2dec909c8440ccab38f6c742b2a43d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/50340f087b63477faee0b655229602b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/50340f087b63477faee0b655229602b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3142,7 +3137,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/19d395a9823e4c45af014487528caa69-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/19d395a9823e4c45af014487528caa69-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1622cc620b054dcd806ac540e2890a06-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1622cc620b054dcd806ac540e2890a06-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3160,7 +3155,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/70ae168ccd1043e3b08e12222315874a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/70ae168ccd1043e3b08e12222315874a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89c1935522b74cba8d23f650d354c7c2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89c1935522b74cba8d23f650d354c7c2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3178,7 +3173,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aa3c9bdc6dd34c6183392d1d4d63c244-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aa3c9bdc6dd34c6183392d1d4d63c244-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3256ea093dd4926acda1cfa0dc56a47-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3256ea093dd4926acda1cfa0dc56a47-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3196,7 +3191,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a1daad923aae43bbb82d2c912aa525dc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a1daad923aae43bbb82d2c912aa525dc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a630930b25b0414c985765758bd8c103-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a630930b25b0414c985765758bd8c103-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3214,7 +3209,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/30471b11c64b42699d3255acfb8cdf3a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/30471b11c64b42699d3255acfb8cdf3a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/22efe8b051fc4ab3a1e4ccaecc34c2e2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/22efe8b051fc4ab3a1e4ccaecc34c2e2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3232,7 +3227,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46a01bdf1e1147f4a5cd0ab38d45e654-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46a01bdf1e1147f4a5cd0ab38d45e654-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3224e16279ae483185d387b65efebcd6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3224e16279ae483185d387b65efebcd6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3250,7 +3245,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f2aead694ad4f7faabde10e24147664-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f2aead694ad4f7faabde10e24147664-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/97f837c4b8f841f989174941eb17b8e6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/97f837c4b8f841f989174941eb17b8e6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3268,7 +3263,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d621e83f35614887a574cb4f1376fa9c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d621e83f35614887a574cb4f1376fa9c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de637fc224c24decaa2df8dd95784c9b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de637fc224c24decaa2df8dd95784c9b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3286,7 +3281,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/855bf9ed5207426996c2a42a9f514899-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/855bf9ed5207426996c2a42a9f514899-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5965ef1740e844418d183eea06aad7a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5965ef1740e844418d183eea06aad7a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3304,7 +3299,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82595de1472547489078801726df5995-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82595de1472547489078801726df5995-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75c396a6b95c48199733e7a78b3e0192-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75c396a6b95c48199733e7a78b3e0192-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3322,7 +3317,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a7ebf7b6df394b2cb00b3ea7f9c45f3d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a7ebf7b6df394b2cb00b3ea7f9c45f3d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/72064fd02aad43318ea12118e1205d88-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/72064fd02aad43318ea12118e1205d88-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3340,7 +3335,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3173e4e0dd8e4aff835fdfa38da5ab31-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3173e4e0dd8e4aff835fdfa38da5ab31-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/728c4fdb4bc441b2b551589101afa47c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/728c4fdb4bc441b2b551589101afa47c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3358,7 +3353,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/962716319e0c47fe8ba6af76523c6ccd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/962716319e0c47fe8ba6af76523c6ccd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/931ce4eae9334ce0b3883eb8808c21e6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/931ce4eae9334ce0b3883eb8808c21e6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3376,7 +3371,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a47b249ac2504891bbf04961ea526651-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a47b249ac2504891bbf04961ea526651-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f36a4179c9d041f692e0f135bd88d39b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f36a4179c9d041f692e0f135bd88d39b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3394,7 +3389,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/917520b23ad44947a511a92a25d5da9b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/917520b23ad44947a511a92a25d5da9b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5a0c4294efe34a2b8699ff23a93bc11a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5a0c4294efe34a2b8699ff23a93bc11a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3412,7 +3407,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc896d52cebd43e286b2e9e25c3c96db-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc896d52cebd43e286b2e9e25c3c96db-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00ef97875057410a9c750c2b8047de52-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00ef97875057410a9c750c2b8047de52-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3430,7 +3425,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7df5a1b6133043429018996190685743-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7df5a1b6133043429018996190685743-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/da0ed6224ac3495f83c9b940e4170bc8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/da0ed6224ac3495f83c9b940e4170bc8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3448,7 +3443,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/69d9c8d2dc69424ca51e9ce549396a64-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/69d9c8d2dc69424ca51e9ce549396a64-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a288a0b6260483b8930890247d862de-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a288a0b6260483b8930890247d862de-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3466,7 +3461,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0257208f7a114cfa8501d8945e582466-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0257208f7a114cfa8501d8945e582466-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a355761e6bb64f24aaa912aa706e5625-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a355761e6bb64f24aaa912aa706e5625-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3484,7 +3479,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35aac521f41a4c47972ae8131051d493-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35aac521f41a4c47972ae8131051d493-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/efd0ac88b9b2448693d82b12873b6c92-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/efd0ac88b9b2448693d82b12873b6c92-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3502,7 +3497,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9574091bd8a457c9eb024038235c03a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9574091bd8a457c9eb024038235c03a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/05e5a86321ad4d93a7d21203a40a1dc9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/05e5a86321ad4d93a7d21203a40a1dc9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3520,7 +3515,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5d0b20f0bfbf41d6a84a9eb6ba4e1637-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5d0b20f0bfbf41d6a84a9eb6ba4e1637-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/21172e9535b24da4a2e62bde00e3dbc0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/21172e9535b24da4a2e62bde00e3dbc0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3538,7 +3533,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e847cacc664b42f0b483bca08881eabc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e847cacc664b42f0b483bca08881eabc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7456504c67fd4a499cec64d0357743d5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7456504c67fd4a499cec64d0357743d5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3556,7 +3551,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/90eb9bf2613b4e498d301941cd22b29a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/90eb9bf2613b4e498d301941cd22b29a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78a8939823e24f838cbb026ad8a92b32-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78a8939823e24f838cbb026ad8a92b32-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3574,7 +3569,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/54f0d9885c64414688ebb2bbede8ef4a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/54f0d9885c64414688ebb2bbede8ef4a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aeb34cef206a4ebb8aa01bccde718b62-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aeb34cef206a4ebb8aa01bccde718b62-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3592,7 +3587,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82506c9d3c024f93aeb592722d1e03f6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82506c9d3c024f93aeb592722d1e03f6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee8653ae368f4a2f836aa87ec7825dac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee8653ae368f4a2f836aa87ec7825dac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3610,7 +3605,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f40ba1e942c43e1b5831bf909d4fa33-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f40ba1e942c43e1b5831bf909d4fa33-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9b4cfa0f669a438d859aba5b4127c15c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9b4cfa0f669a438d859aba5b4127c15c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3628,7 +3623,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ffd61efeeaaa43288d064aaf9fa606ad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ffd61efeeaaa43288d064aaf9fa606ad-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/29b9ba792b044f5f939984f6f8527126-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/29b9ba792b044f5f939984f6f8527126-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3646,7 +3641,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/af07fd65fc6641e4b7f3e42c42edfbfc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/af07fd65fc6641e4b7f3e42c42edfbfc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b12e7617086e495a8d8bf0da01831489-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b12e7617086e495a8d8bf0da01831489-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3664,7 +3659,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d6f25ff3363946e7bda74e20318bbded-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d6f25ff3363946e7bda74e20318bbded-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fca30a506b8f438c83f089818beb88ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fca30a506b8f438c83f089818beb88ef-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3682,7 +3677,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2733c86d3a5845e7b19187bf204bfbc4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2733c86d3a5845e7b19187bf204bfbc4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e0a3b86e417d443aba367785b7131893-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e0a3b86e417d443aba367785b7131893-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3700,7 +3695,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87c0f58464434adf8534980b977b41d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87c0f58464434adf8534980b977b41d8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c004aaf95b3049e0b477eb9dec2193a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c004aaf95b3049e0b477eb9dec2193a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3718,7 +3713,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f200ec6efa24a6b9814227cdc97e4d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f200ec6efa24a6b9814227cdc97e4d2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6ac022511ea04cbeb7521aa7c62e19b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6ac022511ea04cbeb7521aa7c62e19b8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3736,7 +3731,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/806c159953f94e3a850bf0efe7aa2074-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/806c159953f94e3a850bf0efe7aa2074-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2ee90a9823ea495c9cee6f636d675f51-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2ee90a9823ea495c9cee6f636d675f51-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3754,7 +3749,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32f0ae07b38a40a1b7d145dc068a27cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32f0ae07b38a40a1b7d145dc068a27cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24811cc97e794bbaac8767dae5f40614-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24811cc97e794bbaac8767dae5f40614-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3772,7 +3767,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35902bdc6d494f95b8e55bb6f8204642-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35902bdc6d494f95b8e55bb6f8204642-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f67d3a5f1ffc42cda146ce31df587999-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f67d3a5f1ffc42cda146ce31df587999-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3790,7 +3785,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d835a878fc56437093041424cd056b4f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d835a878fc56437093041424cd056b4f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3d18cacd7b3d4d6d909a75adb02993bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3d18cacd7b3d4d6d909a75adb02993bc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3808,7 +3803,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/200a88dbf45843fb82d71a922a6d7cd9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/200a88dbf45843fb82d71a922a6d7cd9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/60d3c67ccbf3468eb872428e34c63994-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/60d3c67ccbf3468eb872428e34c63994-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3826,7 +3821,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9b7c468531e040b1aaa08e77a996b2bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9b7c468531e040b1aaa08e77a996b2bc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba2ee711a29a4bf19ddd3912fd38c68c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba2ee711a29a4bf19ddd3912fd38c68c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3844,7 +3839,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba00bcc4ce924923baf99be48f9a368f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba00bcc4ce924923baf99be48f9a368f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2077fc266d1d4137ae3c4087576cf5e0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2077fc266d1d4137ae3c4087576cf5e0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3862,7 +3857,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8dd35fdd1c9491eaa9519ab6d610298-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8dd35fdd1c9491eaa9519ab6d610298-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9bc2cde6bb814315875c66509330fdbf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9bc2cde6bb814315875c66509330fdbf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3880,7 +3875,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9dbef1e389954e27a17d28a92105c7e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9dbef1e389954e27a17d28a92105c7e7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42f72a8c0e7f44b28b28d600a8f000bf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42f72a8c0e7f44b28b28d600a8f000bf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3898,7 +3893,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/851fbad53d7d41c5b46a21d93d03027f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/851fbad53d7d41c5b46a21d93d03027f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/80eaceb2c2b9414dbea5660ee171dd00-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/80eaceb2c2b9414dbea5660ee171dd00-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3916,7 +3911,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e885becb7f5e4d01aa50711a2cfa36ad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e885becb7f5e4d01aa50711a2cfa36ad-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0b34cc8458584489ab84ce5d80735114-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0b34cc8458584489ab84ce5d80735114-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3934,7 +3929,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7256341224f84589b129a5cb96fb466c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7256341224f84589b129a5cb96fb466c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/049ac5663c9547f8a906633723a100b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/049ac5663c9547f8a906633723a100b8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3952,7 +3947,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d39e515e977e4bab8a49bc42b82e2c47-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d39e515e977e4bab8a49bc42b82e2c47-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b50c82a33882466c8f9a788f34cecc1a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b50c82a33882466c8f9a788f34cecc1a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3970,7 +3965,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/483fdcb7eeef492caa509c5d5e39c128-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/483fdcb7eeef492caa509c5d5e39c128-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/05daee1beee24652948b4d088c3ea36d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/05daee1beee24652948b4d088c3ea36d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -3988,7 +3983,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d0f48e6238094de4885a51143243ab38-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d0f48e6238094de4885a51143243ab38-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c096a37883ab4d9d84180d02d22560d4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c096a37883ab4d9d84180d02d22560d4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4006,7 +4001,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/059b62ecc27540f49b1e230cd53e27e8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/059b62ecc27540f49b1e230cd53e27e8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8dd1e96ed25342808490884feec9d3bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8dd1e96ed25342808490884feec9d3bb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4024,7 +4019,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b9374e777c147a2b92bfcc4db9101cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b9374e777c147a2b92bfcc4db9101cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ca047eea6c2940ba81f5cd5abe17b796-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ca047eea6c2940ba81f5cd5abe17b796-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4042,7 +4037,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1a36e6283d98468cb1b93533319eed3c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1a36e6283d98468cb1b93533319eed3c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/092a9c06412148808ff70f3db85ec616-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/092a9c06412148808ff70f3db85ec616-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4060,7 +4055,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a416ada5f9814a82a37a60270f8912f6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a416ada5f9814a82a37a60270f8912f6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e26f18f9f964218ae8178fe98fe3a1f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e26f18f9f964218ae8178fe98fe3a1f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4078,7 +4073,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b9613eaccbe481397ff32c6ce188b04-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b9613eaccbe481397ff32c6ce188b04-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/36949a3c370a432ea80e67febfa7010c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/36949a3c370a432ea80e67febfa7010c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4096,7 +4091,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c3c53f46cba34217af9be7408a6e4844-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c3c53f46cba34217af9be7408a6e4844-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b73265ad42cc41179bca8cd75da5c879-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b73265ad42cc41179bca8cd75da5c879-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4114,7 +4109,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a9fd006a03544f0b4e3ff5aff1e6a8a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a9fd006a03544f0b4e3ff5aff1e6a8a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/61c2c4de9a014a9ba2a1d9263d293e42-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/61c2c4de9a014a9ba2a1d9263d293e42-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4132,7 +4127,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9c1581ab3cf3411ea5b3675a940dc793-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9c1581ab3cf3411ea5b3675a940dc793-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/098f7ffde10f44678ddd07a0bbd2baf6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/098f7ffde10f44678ddd07a0bbd2baf6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4150,7 +4145,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/526d9b4e3a044322942c360ac78381d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/526d9b4e3a044322942c360ac78381d2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1987752f64242fc9eb235be105cdd7d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1987752f64242fc9eb235be105cdd7d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4168,7 +4163,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c0115a962fc4f4eab24f07a7ffc408f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c0115a962fc4f4eab24f07a7ffc408f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/119bc32222544801a2f035e2f696297d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/119bc32222544801a2f035e2f696297d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4186,7 +4181,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6755f77549cb4dad88f2db945fb11cdf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6755f77549cb4dad88f2db945fb11cdf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53a92788bedd4732b653c84d5d644d3b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53a92788bedd4732b653c84d5d644d3b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4204,7 +4199,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9959f42f8f4d4048874d4ff7a4d5a4d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9959f42f8f4d4048874d4ff7a4d5a4d9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ab60c19bd7e4502ad853888dcff1cfc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ab60c19bd7e4502ad853888dcff1cfc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4222,7 +4217,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/026749ab48844d4697964aa9b3029fa8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/026749ab48844d4697964aa9b3029fa8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2efc9251a41f471aad85cdcf223fa2c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2efc9251a41f471aad85cdcf223fa2c0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4240,7 +4235,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/adb66a7b410b4ec4a30235fdb7a0f25d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/adb66a7b410b4ec4a30235fdb7a0f25d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1ff1765bdf8a4bba9030b7bfa9f17634-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1ff1765bdf8a4bba9030b7bfa9f17634-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4258,7 +4253,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e5c00e811388461ca611878dbe393a5f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e5c00e811388461ca611878dbe393a5f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0c3db0ffdfc1485a8d942e19cbbc86da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0c3db0ffdfc1485a8d942e19cbbc86da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4276,7 +4271,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/291dd24466d849a0b0273366ab822d8a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/291dd24466d849a0b0273366ab822d8a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/47852c16636a4283ab22681e4c7b9e8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/47852c16636a4283ab22681e4c7b9e8c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4294,7 +4289,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cbf89e2922d4928bfb633ca3052d012-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cbf89e2922d4928bfb633ca3052d012-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/74fa212e2c8b477896bd07177c5dbb37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/74fa212e2c8b477896bd07177c5dbb37-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4312,7 +4307,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/31eb9636e1b24376b42dbf115a189b29-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/31eb9636e1b24376b42dbf115a189b29-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6afa7eca9d774de0992066896c48b591-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6afa7eca9d774de0992066896c48b591-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4330,7 +4325,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/885a3dc60320480aa597aee5fcb188c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/885a3dc60320480aa597aee5fcb188c9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/622adb9de59c4e009c5c07f9899fea82-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/622adb9de59c4e009c5c07f9899fea82-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4348,7 +4343,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6ff7da4c13bf4ab2a2649e07ca27bbbc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6ff7da4c13bf4ab2a2649e07ca27bbbc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e18c2fd4924644c9910970f9e8693e48-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e18c2fd4924644c9910970f9e8693e48-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4366,7 +4361,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0fb8f94de41e4402a1c111ba20f0be43-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0fb8f94de41e4402a1c111ba20f0be43-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e19f4173519a486b808be66aa41f6192-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e19f4173519a486b808be66aa41f6192-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4384,7 +4379,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3b0602308d7645ab8220f6465bea2788-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3b0602308d7645ab8220f6465bea2788-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3d871a0a96804d8293e014a9e0e05b7a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3d871a0a96804d8293e014a9e0e05b7a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4402,7 +4397,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f877266409249bf993678111947b50d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f877266409249bf993678111947b50d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a049d6044934ca9be992c3ab540878f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a049d6044934ca9be992c3ab540878f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4420,7 +4415,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d7e2a104513e4e5c969d7cb09e87ada6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d7e2a104513e4e5c969d7cb09e87ada6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/858fa5d9229a4c84bba76005bfdc5c40-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/858fa5d9229a4c84bba76005bfdc5c40-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4438,7 +4433,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/38dd90cd592047a2b3c460fa9d6c2b20-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/38dd90cd592047a2b3c460fa9d6c2b20-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/30057e83dec447978c84ac3f7a9dc891-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/30057e83dec447978c84ac3f7a9dc891-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4456,7 +4451,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3f2420fb4074c82851ba2ff006ca625-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3f2420fb4074c82851ba2ff006ca625-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2fd9fe9a64964886beac0fb26e10ea72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2fd9fe9a64964886beac0fb26e10ea72-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4474,7 +4469,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4bc9718bf2544045a3659e7cb0f46784-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4bc9718bf2544045a3659e7cb0f46784-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/48dea273f76c463287ed0aac28b5497d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/48dea273f76c463287ed0aac28b5497d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4492,7 +4487,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/be9daff3027e41df89c0bc4984a84ee8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/be9daff3027e41df89c0bc4984a84ee8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8dbf83d576747369c37a12a6f15ca0b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8dbf83d576747369c37a12a6f15ca0b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4510,7 +4505,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/39662dcda8bb4a8d89d94e06a602746c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/39662dcda8bb4a8d89d94e06a602746c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c96cf16fb5a439bae07194c4ba6c112-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c96cf16fb5a439bae07194c4ba6c112-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4528,7 +4523,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/abc635ce179a4edb9360ed8402102c42-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/abc635ce179a4edb9360ed8402102c42-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/be5357f9329e4ea79159d5bc79e17490-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/be5357f9329e4ea79159d5bc79e17490-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4546,7 +4541,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3fecb6073d6546298bea43deb93783d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3fecb6073d6546298bea43deb93783d0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/41ff9c52570f4c509f1d20725f56c718-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/41ff9c52570f4c509f1d20725f56c718-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4564,7 +4559,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3fe9c2df798e46bcbb190b506cbba40c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3fe9c2df798e46bcbb190b506cbba40c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/88e82063a16742a0a55e84294b976728-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/88e82063a16742a0a55e84294b976728-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4582,7 +4577,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/612d8425859842b7858678264372f8dc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/612d8425859842b7858678264372f8dc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/137d1e26404f4478bec14d2343b377bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/137d1e26404f4478bec14d2343b377bc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4600,7 +4595,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5aefbd721ca4a32ab8d649edd445a4d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5aefbd721ca4a32ab8d649edd445a4d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd595d6d46f141189e9984402b25ff16-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd595d6d46f141189e9984402b25ff16-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4618,7 +4613,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7e7caf5d89bf46fbb90f5c4ffb6fffa6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7e7caf5d89bf46fbb90f5c4ffb6fffa6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d3aba3c6e68f4495baf5944c346612ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d3aba3c6e68f4495baf5944c346612ee-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4636,7 +4631,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c568655b3b5a45fd98d2a9b510a44f9e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c568655b3b5a45fd98d2a9b510a44f9e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9fa068d9b404c0fa76ffbef3e8a23b1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9fa068d9b404c0fa76ffbef3e8a23b1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4654,7 +4649,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75777628d1a34f0ba3c76aa6ab20d3b7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75777628d1a34f0ba3c76aa6ab20d3b7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d57b64a79088470887642f1e9837b0d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d57b64a79088470887642f1e9837b0d8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4672,7 +4667,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/024d4b8b5ba84455b2f1e764a910dbfa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/024d4b8b5ba84455b2f1e764a910dbfa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/43c2d70e708a40e0b553e83af142d4a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/43c2d70e708a40e0b553e83af142d4a9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4690,7 +4685,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3073578c3e574b29868b038f1e23611e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3073578c3e574b29868b038f1e23611e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0deb44471dd644ef80891618fc3387c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0deb44471dd644ef80891618fc3387c5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4708,7 +4703,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82af853f7f944206944081c5e30d7859-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82af853f7f944206944081c5e30d7859-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f7f13d0845a4da3b849ed6eea9cfa4d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f7f13d0845a4da3b849ed6eea9cfa4d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4726,7 +4721,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b79656a090e446a382a97c94bb3675d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b79656a090e446a382a97c94bb3675d6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a0d4ea8b892746908ccbc389c456d0bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a0d4ea8b892746908ccbc389c456d0bc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4744,7 +4739,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b43480c97e59477985396f6a8e35b5b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b43480c97e59477985396f6a8e35b5b5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/98da76dbaee248d48303260facd62129-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/98da76dbaee248d48303260facd62129-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4762,7 +4757,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7f4927a3adfa40f9be225dc5a5310f0c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7f4927a3adfa40f9be225dc5a5310f0c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b91b4f964fca455d88294bc08ce364c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b91b4f964fca455d88294bc08ce364c5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4780,7 +4775,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5a1dcc9f6cc4b45b888a44832092067-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5a1dcc9f6cc4b45b888a44832092067-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/978142e6422c4d81ba9fda235e6161da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/978142e6422c4d81ba9fda235e6161da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4798,7 +4793,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/19afcbc5c6814ca982ebc9677252669b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/19afcbc5c6814ca982ebc9677252669b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2aa7569069374c95ab0e5dc88a0fd55f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2aa7569069374c95ab0e5dc88a0fd55f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4816,7 +4811,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a31b64b5b600449bb958f23076833351-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a31b64b5b600449bb958f23076833351-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8c8aeaeadf8e4086ac3cbd46a5e908f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8c8aeaeadf8e4086ac3cbd46a5e908f5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4834,7 +4829,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/54f8983590614b3c97c99b09e66a7ff2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/54f8983590614b3c97c99b09e66a7ff2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ac4ac3aae2fe41ee8b9165a02cfd1bb0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ac4ac3aae2fe41ee8b9165a02cfd1bb0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4852,7 +4847,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/79f45ffb6f9443d3bbbd3186d1c9dc2b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/79f45ffb6f9443d3bbbd3186d1c9dc2b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fb7967f312414c4aba5f7288bfac22c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fb7967f312414c4aba5f7288bfac22c6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4870,7 +4865,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ca0c81b0a0ca43f2b6fdd8d384ea23d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ca0c81b0a0ca43f2b6fdd8d384ea23d6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/94b08f30ea264e6c8ff70c549ca2214b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/94b08f30ea264e6c8ff70c549ca2214b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4888,7 +4883,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a2667b8d93cf48d0880ed7bd6e2638d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a2667b8d93cf48d0880ed7bd6e2638d3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/584189d5b28548b3899b54cddba365be-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/584189d5b28548b3899b54cddba365be-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4906,7 +4901,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9297a43f34fd44cabe4648b7e665f920-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9297a43f34fd44cabe4648b7e665f920-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2a4403f7858141349a5d4d0b8c18332f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2a4403f7858141349a5d4d0b8c18332f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4924,7 +4919,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/888320cac328430aa90911350460ff71-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/888320cac328430aa90911350460ff71-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d82fc569e61a4bb29c418216b8788169-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d82fc569e61a4bb29c418216b8788169-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4942,7 +4937,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4212d4a725c845558b14938d53785808-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4212d4a725c845558b14938d53785808-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea3f9c62d357441abc82739d680498ae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea3f9c62d357441abc82739d680498ae-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4960,7 +4955,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8dce42133ee425b875e2b861625b434-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8dce42133ee425b875e2b861625b434-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9e837aa7902d435498c8dc0ed5b71c89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9e837aa7902d435498c8dc0ed5b71c89-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4978,7 +4973,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/21c5318977f54a13a14e2cb293c53304-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/21c5318977f54a13a14e2cb293c53304-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/498fb4909f0f440296f22486f864d104-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/498fb4909f0f440296f22486f864d104-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -4996,7 +4991,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c54029f0706047278fb1a28839b6ff01-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c54029f0706047278fb1a28839b6ff01-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8d0435d63cf54ce7acc137986c031a4e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8d0435d63cf54ce7acc137986c031a4e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5014,7 +5009,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/660d5ebd238142758e66ada0f7a465af-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/660d5ebd238142758e66ada0f7a465af-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dec42736b4184375a3d2a73cb449b279-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dec42736b4184375a3d2a73cb449b279-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5032,7 +5027,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5480d371191e450680fc57858a1c6518-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5480d371191e450680fc57858a1c6518-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ee287fa47464c2793d3eae9fec9a8ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ee287fa47464c2793d3eae9fec9a8ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5050,7 +5045,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b972593e01874be8a616a1c944cd5135-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b972593e01874be8a616a1c944cd5135-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7e5ceb64dab7439e886ebbe6c01a1da9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7e5ceb64dab7439e886ebbe6c01a1da9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5068,7 +5063,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/701163d839214e25ab5aaf83f8658618-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/701163d839214e25ab5aaf83f8658618-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8361071634145058187631d9a0e1b46-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8361071634145058187631d9a0e1b46-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5086,7 +5081,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/220c6e4856f54e819f04ff8c9e941451-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/220c6e4856f54e819f04ff8c9e941451-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/67ae0da1641045f2881a61b2393e4bd2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/67ae0da1641045f2881a61b2393e4bd2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5104,7 +5099,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/680f09558d834b01a2edd7e42bcd46ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/680f09558d834b01a2edd7e42bcd46ea-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7d2eab1fa9a14aaab354e3dd191d8464-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7d2eab1fa9a14aaab354e3dd191d8464-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5122,7 +5117,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/903ee0b7441144f0890360a2fc953ac3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/903ee0b7441144f0890360a2fc953ac3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f359724e139f44af98ed50288bf5340e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f359724e139f44af98ed50288bf5340e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5140,7 +5135,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bb3d2887a50741adbe36fdda87001d1a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bb3d2887a50741adbe36fdda87001d1a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cd5dfd3518e04c039a43f95531c5f935-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cd5dfd3518e04c039a43f95531c5f935-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5158,7 +5153,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d07a93494f4446999c938f219c169372-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d07a93494f4446999c938f219c169372-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c16f5b5e6d004a03a88097669d15e5f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c16f5b5e6d004a03a88097669d15e5f2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5176,7 +5171,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f27289d31e36449c90ac0c1e871863c8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f27289d31e36449c90ac0c1e871863c8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/67261e32a27a42158ae65ca9d886edf1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/67261e32a27a42158ae65ca9d886edf1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5194,7 +5189,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dbee2bebb3c3454caedd60d76c999ad0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dbee2bebb3c3454caedd60d76c999ad0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4c50bfbb2aa544ec80ce28fc4b36086f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4c50bfbb2aa544ec80ce28fc4b36086f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5212,7 +5207,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/976ba1099f564031af4e5be7d191963e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/976ba1099f564031af4e5be7d191963e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f92acc27fd1c4e2eab2f56ebafcd3ed3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f92acc27fd1c4e2eab2f56ebafcd3ed3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5230,7 +5225,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c300759ae90c4b40b2602c5564682d3b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c300759ae90c4b40b2602c5564682d3b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e5354bd044ac4704b131e19da5c82fa7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e5354bd044ac4704b131e19da5c82fa7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5248,7 +5243,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e9ffb29d16124223887c8ce59548c628-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e9ffb29d16124223887c8ce59548c628-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/617c41b60b084c70813ccf47f49b3fc7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/617c41b60b084c70813ccf47f49b3fc7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5266,7 +5261,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b4cf9ac259d344a7bff9c02794297397-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b4cf9ac259d344a7bff9c02794297397-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c3c29bbbab24d558f30c7e8a28e91d5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c3c29bbbab24d558f30c7e8a28e91d5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5284,7 +5279,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2a7170f400a14080a28f9eefa9246af2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2a7170f400a14080a28f9eefa9246af2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/991641434e3f4a1b8cad96438ea13713-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/991641434e3f4a1b8cad96438ea13713-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5302,7 +5297,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57e03f7c35be4d35a2de78d35b6055c1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57e03f7c35be4d35a2de78d35b6055c1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/718cdd4957034796ab384764b6a75d72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/718cdd4957034796ab384764b6a75d72-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 13 COLUMNS\n",
@@ -5324,7 +5319,7 @@
"True"
]
},
- "execution_count": 15,
+ "execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
@@ -5337,9 +5332,9 @@
"evaluations = [evaluation0, evaluation1]\n",
"scores = {'acc': 0.357, 'sens': 0.323, 'spec': 0.362, 'bacc': 0.9}\n",
"\n",
- "result = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=evaluations,\n",
- " scores=scores,\n",
- " eps=1e-3)\n",
+ "result = check_n_datasets_mos_unknown_folds_mos(evaluations=evaluations,\n",
+ " scores=scores,\n",
+ " eps=1e-3)\n",
"result['inconsistency']"
]
}
diff --git a/notebooks/illustration/binary/08_check_n_testsets_mos_no_kfold.ipynb b/notebooks/illustration/binary/08_check_n_testsets_mos_no_kfold.ipynb
new file mode 100644
index 0000000..16ec3b0
--- /dev/null
+++ b/notebooks/illustration/binary/08_check_n_testsets_mos_no_kfold.ipynb
@@ -0,0 +1,184 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.binary import check_n_testsets_mos_no_kfold"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "testsets = [{'p': 349, 'n': 50},\n",
+ " {'p': 478, 'n': 323},\n",
+ " {'p': 324, 'n': 83},\n",
+ " {'p': 123, 'n': 145}]\n",
+ "\n",
+ "scores = {'acc': 0.6441, 'sens': 0.6706, 'spec': 0.3796, 'bacc': 0.5251}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d999fb087e246d89c9911715e4446bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d999fb087e246d89c9911715e4446bd-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 13 COLUMNS\n",
+ "At line 79 RHS\n",
+ "At line 88 BOUNDS\n",
+ "At line 98 ENDATA\n",
+ "Problem MODEL has 8 rows, 9 columns and 48 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0004I processed model has 4 rows, 8 columns (8 integer (0 of which binary)) and 24 elements\n",
+ "Cbc0031I 2 added rows had average density of 8\n",
+ "Cbc0013I At root node, 2 cuts changed objective from 0 to 0 in 28 passes\n",
+ "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 51 row cuts average 7.5 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 2 row cuts average 3.5 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 6 (TwoMirCuts) - 6 row cuts average 3.3 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.01 seconds)\n",
+ "Cbc0016I Integer solution of 0 found by strong branching after 206 iterations and 52 nodes (0.02 seconds)\n",
+ "Cbc0001I Search completed - best objective 0, took 206 iterations and 52 nodes (0.02 seconds)\n",
+ "Cbc0032I Strong branching done 206 times (291 iterations), fathomed 14 nodes and fixed 31 variables\n",
+ "Cbc0035I Maximum depth 28, 0 variables fixed on reduced cost\n",
+ "Cuts at root node changed objective from 0 to 0\n",
+ "Probing was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Gomory was tried 28 times and created 51 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Knapsack was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Clique was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 28 times and created 2 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 28 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 28 times and created 6 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "\n",
+ "Result - Optimal solution found\n",
+ "\n",
+ "Objective value: 0.00000000\n",
+ "Enumerated nodes: 52\n",
+ "Total iterations: 206\n",
+ "Time (CPU seconds): 0.02\n",
+ "Time (Wallclock seconds): 0.02\n",
+ "\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.02 (Wallclock seconds): 0.02\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_n_testsets_mos_no_kfold(testsets=testsets,\n",
+ " scores=scores,\n",
+ " eps=0.0001)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['sens'] = 0.6756"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/51b08105771f4a8a9f27f05b5edf0673-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/51b08105771f4a8a9f27f05b5edf0673-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 13 COLUMNS\n",
+ "At line 79 RHS\n",
+ "At line 88 BOUNDS\n",
+ "At line 98 ENDATA\n",
+ "Problem MODEL has 8 rows, 9 columns and 48 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Problem is infeasible - 0.00 seconds\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_n_testsets_mos_no_kfold(testsets=testsets,\n",
+ " scores=scores,\n",
+ " eps=0.0001)\n",
+ "result['inconsistency']"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/09_check_n_testsets_som_no_kfold.ipynb b/notebooks/illustration/binary/09_check_n_testsets_som_no_kfold.ipynb
similarity index 65%
rename from notebooks/illustration/09_check_n_testsets_som_no_kfold.ipynb
rename to notebooks/illustration/binary/09_check_n_testsets_som_no_kfold.ipynb
index 580a9aa..a585353 100644
--- a/notebooks/illustration/09_check_n_testsets_som_no_kfold.ipynb
+++ b/notebooks/illustration/binary/09_check_n_testsets_som_no_kfold.ipynb
@@ -6,7 +6,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.check import check_n_testsets_som_no_kfold_scores"
+ "from mlscorecheck.check.binary import check_n_testsets_som_no_kfold"
]
},
{
@@ -28,28 +28,28 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:41:34,660:INFO:checking the scores {'acc': 0.4719, 'npv': 0.6253, 'f1p': 0.3091}\n",
- "2023-10-26 23:41:34,661:INFO:evaluating the tp and tn solution for acc and npv\n",
- "2023-10-26 23:41:34,662:INFO:intervals before: (0, 517), (0, 1049)\n",
- "2023-10-26 23:41:34,663:INFO:the tp solutions: (183.3735345619046, 186.76921301883434)\n",
- "2023-10-26 23:41:34,664:INFO:the tn solutions: (552.9864593956394, 554.8661474746565)\n",
- "2023-10-26 23:41:34,666:INFO:intervals after: [(184, 186)], [(553, 554)]\n",
- "2023-10-26 23:41:34,667:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:41:34,669:INFO:intervals before: [(184, 186)], [(553, 554)]\n",
- "2023-10-26 23:41:34,669:INFO:the tp solutions: (184.87168671166216, 185.11986556349612)\n",
- "2023-10-26 23:41:34,670:INFO:the tn solutions: (553.4808574181667, 554.5185801488539)\n",
- "2023-10-26 23:41:34,671:INFO:intervals after: [(185, 185)], [(554, 554)]\n",
- "2023-10-26 23:41:34,672:INFO:evaluating the tp and tn solution for npv and f1p\n",
- "2023-10-26 23:41:34,673:INFO:intervals before: [(185, 185)], [(554, 554)]\n",
- "2023-10-26 23:41:34,674:INFO:the tp solutions: (184.64422257239337, 185.32173809718321)\n",
- "2023-10-26 23:41:34,675:INFO:the tn solutions: (553.2944463587239, 554.8488214174102)\n",
- "2023-10-26 23:41:34,676:INFO:intervals after: [(185, 185)], [(554, 554)]\n",
- "2023-10-26 23:41:34,676:INFO:testing acc, feasible tptn pairs: 1\n",
- "2023-10-26 23:41:34,677:INFO:testing npv, feasible tptn pairs: 1\n",
- "2023-10-26 23:41:34,678:INFO:testing f1p, feasible tptn pairs: 1\n",
- "2023-10-26 23:41:34,681:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:41:34,682:INFO:final number of intervals: 1\n",
- "2023-10-26 23:41:34,684:INFO:final number of pairs: 1\n"
+ "2023-11-03 06:39:34,967:INFO:checking the scores {'acc': 0.4719, 'npv': 0.6253, 'f1p': 0.3091}\n",
+ "2023-11-03 06:39:34,968:INFO:evaluating the tp and tn solution for acc and npv\n",
+ "2023-11-03 06:39:34,970:INFO:intervals before: (0, 517), (0, 1049)\n",
+ "2023-11-03 06:39:34,971:INFO:the tp solutions: (183.3735345619046, 186.76921301883434)\n",
+ "2023-11-03 06:39:34,972:INFO:the tn solutions: (552.9864593956394, 554.8661474746565)\n",
+ "2023-11-03 06:39:34,973:INFO:intervals after: [(184, 186)], [(553, 554)]\n",
+ "2023-11-03 06:39:34,974:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 06:39:34,975:INFO:intervals before: [(184, 186)], [(553, 554)]\n",
+ "2023-11-03 06:39:34,978:INFO:the tp solutions: (184.87168671166216, 185.11986556349612)\n",
+ "2023-11-03 06:39:34,979:INFO:the tn solutions: (553.4808574181667, 554.5185801488539)\n",
+ "2023-11-03 06:39:34,980:INFO:intervals after: [(185, 185)], [(554, 554)]\n",
+ "2023-11-03 06:39:34,981:INFO:evaluating the tp and tn solution for npv and f1p\n",
+ "2023-11-03 06:39:34,982:INFO:intervals before: [(185, 185)], [(554, 554)]\n",
+ "2023-11-03 06:39:34,983:INFO:the tp solutions: (184.64422257239337, 185.32173809718321)\n",
+ "2023-11-03 06:39:34,983:INFO:the tn solutions: (553.2944463587239, 554.8488214174102)\n",
+ "2023-11-03 06:39:34,984:INFO:intervals after: [(185, 185)], [(554, 554)]\n",
+ "2023-11-03 06:39:34,984:INFO:testing acc, feasible tptn pairs: 1\n",
+ "2023-11-03 06:39:34,985:INFO:testing npv, feasible tptn pairs: 1\n",
+ "2023-11-03 06:39:34,986:INFO:testing f1p, feasible tptn pairs: 1\n",
+ "2023-11-03 06:39:34,987:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:39:34,988:INFO:final number of intervals: 1\n",
+ "2023-11-03 06:39:34,988:INFO:final number of pairs: 1\n"
]
},
{
@@ -74,7 +74,7 @@
" 'n_valid_tptn_pairs': 1,\n",
" 'iterate_by': 'tp',\n",
" 'solve_for': 'tn',\n",
- " 'evidence': (185, [(554, 554)]),\n",
+ " 'evidence': {'tp': 185, 'tn': 554},\n",
" 'prefiltering_details': {'inconsistency': False,\n",
" 'tp': [(185, 185)],\n",
" 'tn': [(554, 554)],\n",
@@ -107,9 +107,9 @@
}
],
"source": [
- "check_n_testsets_som_no_kfold_scores(testsets=testsets,\n",
- " scores=scores,\n",
- " eps=0.0001)"
+ "check_n_testsets_som_no_kfold(testsets=testsets,\n",
+ " scores=scores,\n",
+ " eps=0.0001)"
]
},
{
@@ -130,27 +130,27 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:41:34,746:INFO:checking the scores {'acc': 0.4719, 'npv': 0.6263, 'f1p': 0.3091}\n",
- "2023-10-26 23:41:34,748:INFO:evaluating the tp and tn solution for acc and npv\n",
- "2023-10-26 23:41:34,750:INFO:intervals before: (0, 517), (0, 1049)\n",
- "2023-10-26 23:41:34,751:INFO:the tp solutions: (186.89044330890292, 190.26616505437528)\n",
- "2023-10-26 23:41:34,752:INFO:the tn solutions: (549.4891363675572, 551.3495419700156)\n",
- "2023-10-26 23:41:34,753:INFO:intervals after: [(187, 190)], [(550, 551)]\n",
- "2023-10-26 23:41:34,754:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:41:34,755:INFO:intervals before: [(187, 190)], [(550, 551)]\n",
- "2023-10-26 23:41:34,756:INFO:the tp solutions: (184.87168671166216, 185.11986556349612)\n",
- "2023-10-26 23:41:34,758:INFO:the tn solutions: (553.4808574181667, 554.5185801488539)\n",
- "2023-10-26 23:41:34,760:INFO:intervals after: [], []\n",
- "2023-10-26 23:41:34,761:INFO:evaluating the tp and tn solution for npv and f1p\n",
- "2023-10-26 23:41:34,763:INFO:intervals before: [], []\n",
- "2023-10-26 23:41:34,764:INFO:the tp solutions: (184.01870812695688, 184.69747317618967)\n",
- "2023-10-26 23:41:34,766:INFO:the tn solutions: (556.7064405556058, 558.2736510014568)\n",
- "2023-10-26 23:41:34,766:INFO:intervals after: [], []\n",
- "2023-10-26 23:41:34,768:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:41:34,769:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:41:34,770:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:41:34,771:INFO:final number of intervals: 0\n",
- "2023-10-26 23:41:34,771:INFO:final number of pairs: 0\n"
+ "2023-11-03 06:39:35,014:INFO:checking the scores {'acc': 0.4719, 'npv': 0.6263, 'f1p': 0.3091}\n",
+ "2023-11-03 06:39:35,015:INFO:evaluating the tp and tn solution for acc and npv\n",
+ "2023-11-03 06:39:35,016:INFO:intervals before: (0, 517), (0, 1049)\n",
+ "2023-11-03 06:39:35,017:INFO:the tp solutions: (186.89044330890292, 190.26616505437528)\n",
+ "2023-11-03 06:39:35,017:INFO:the tn solutions: (549.4891363675572, 551.3495419700156)\n",
+ "2023-11-03 06:39:35,018:INFO:intervals after: [(187, 190)], [(550, 551)]\n",
+ "2023-11-03 06:39:35,019:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 06:39:35,020:INFO:intervals before: [(187, 190)], [(550, 551)]\n",
+ "2023-11-03 06:39:35,020:INFO:the tp solutions: (184.87168671166216, 185.11986556349612)\n",
+ "2023-11-03 06:39:35,021:INFO:the tn solutions: (553.4808574181667, 554.5185801488539)\n",
+ "2023-11-03 06:39:35,022:INFO:intervals after: [], []\n",
+ "2023-11-03 06:39:35,023:INFO:evaluating the tp and tn solution for npv and f1p\n",
+ "2023-11-03 06:39:35,024:INFO:intervals before: [], []\n",
+ "2023-11-03 06:39:35,025:INFO:the tp solutions: (184.01870812695688, 184.69747317618967)\n",
+ "2023-11-03 06:39:35,026:INFO:the tn solutions: (556.7064405556058, 558.2736510014568)\n",
+ "2023-11-03 06:39:35,026:INFO:intervals after: [], []\n",
+ "2023-11-03 06:39:35,027:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 06:39:35,028:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 06:39:35,029:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 06:39:35,029:INFO:final number of intervals: 0\n",
+ "2023-11-03 06:39:35,030:INFO:final number of pairs: 0\n"
]
},
{
@@ -198,9 +198,9 @@
}
],
"source": [
- "check_n_testsets_som_no_kfold_scores(testsets=testsets,\n",
- " scores=scores,\n",
- " eps=0.0001)"
+ "check_n_testsets_som_no_kfold(testsets=testsets,\n",
+ " scores=scores,\n",
+ " eps=0.0001)"
]
},
{
diff --git a/notebooks/illustration/20_retina_drive.ipynb b/notebooks/illustration/bundles/00_retina_drive.ipynb
similarity index 51%
rename from notebooks/illustration/20_retina_drive.ipynb
rename to notebooks/illustration/bundles/00_retina_drive.ipynb
index 8b52086..3766d7a 100644
--- a/notebooks/illustration/20_retina_drive.ipynb
+++ b/notebooks/illustration/bundles/00_retina_drive.ipynb
@@ -13,8 +13,8 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.retina import (check_drive_vessel_image,\n",
- " check_drive_vessel_aggregated)"
+ "from mlscorecheck.check.bundles.retina import (check_drive_vessel_image,\n",
+ " check_drive_vessel_aggregated)"
]
},
{
@@ -55,49 +55,49 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:05,250:INFO:checking the scores {'acc': 0.9494, 'sens': 0.745, 'spec': 0.9793}\n",
- "2023-10-26 23:42:05,251:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,253:INFO:intervals before: (0, 577649), (0, 3960494)\n",
- "2023-10-26 23:42:05,254:INFO:the tp solutions: (430289.58480199997, 430407.42519800004)\n",
- "2023-10-26 23:42:05,255:INFO:the tn solutions: (3877642.6484160004, 3878686.2699840004)\n",
- "2023-10-26 23:42:05,256:INFO:intervals after: [(430290, 430407)], [(3877643, 3878686)]\n",
- "2023-10-26 23:42:05,258:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,259:INFO:intervals before: [(430290, 430407)], [(3877643, 3878686)]\n",
- "2023-10-26 23:42:05,260:INFO:the tp solutions: (429134.3290260006, 430868.0509740007)\n",
- "2023-10-26 23:42:05,261:INFO:the tn solutions: (3878107.8038119995, 3878915.744588)\n",
- "2023-10-26 23:42:05,262:INFO:intervals after: [(430290, 430407)], [(3878108, 3878686)]\n",
- "2023-10-26 23:42:05,263:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,263:INFO:intervals before: [(430290, 430407)], [(3878108, 3878686)]\n",
- "2023-10-26 23:42:05,264:INFO:the tp solutions: (430289.58480199997, 430407.42519800004)\n",
- "2023-10-26 23:42:05,265:INFO:the tn solutions: (3878107.8038119995, 3878915.744588)\n",
- "2023-10-26 23:42:05,266:INFO:intervals after: [(430290, 430407)], [(3878108, 3878686)]\n",
- "2023-10-26 23:42:05,269:INFO:testing acc, feasible tptn pairs: 118\n",
- "2023-10-26 23:42:05,282:INFO:testing sens, feasible tptn pairs: 118\n",
- "2023-10-26 23:42:05,284:INFO:testing spec, feasible tptn pairs: 116\n",
- "2023-10-26 23:42:05,288:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,289:INFO:final number of intervals: 116\n",
- "2023-10-26 23:42:05,289:INFO:final number of pairs: 59334\n",
- "2023-10-26 23:42:05,308:INFO:checking the scores {'acc': 0.9494, 'sens': 0.745, 'spec': 0.9793}\n",
- "2023-10-26 23:42:05,309:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,310:INFO:intervals before: (0, 577945), (0, 6021255)\n",
- "2023-10-26 23:42:05,311:INFO:the tp solutions: (430510.07460999995, 430627.97539000004)\n",
- "2023-10-26 23:42:05,313:INFO:the tn solutions: (5833979.38621, 5835443.523790001)\n",
- "2023-10-26 23:42:05,315:INFO:intervals after: [(430511, 430627)], [(5833980, 5835443)]\n",
- "2023-10-26 23:42:05,316:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,316:INFO:intervals before: [(430511, 430627)], [(5833980, 5835443)]\n",
- "2023-10-26 23:42:05,318:INFO:the tp solutions: (367378.1720900005, 369952.7449100008)\n",
- "2023-10-26 23:42:05,319:INFO:the tn solutions: (5896000.85349, 5897229.18951)\n",
- "2023-10-26 23:42:05,319:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,320:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,321:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:05,322:INFO:the tp solutions: (430510.07460999995, 430627.97539000004)\n",
- "2023-10-26 23:42:05,323:INFO:the tn solutions: (5896000.85349, 5897229.18951)\n",
- "2023-10-26 23:42:05,324:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,325:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:05,326:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:05,327:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,328:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:05,329:INFO:final number of pairs: 0\n"
+ "2023-11-03 07:59:01,488:INFO:checking the scores {'acc': 0.9494, 'sens': 0.745, 'spec': 0.9793}\n",
+ "2023-11-03 07:59:01,489:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,490:INFO:intervals before: (0, 577649), (0, 3960494)\n",
+ "2023-11-03 07:59:01,491:INFO:the tp solutions: (430289.58480199997, 430407.42519800004)\n",
+ "2023-11-03 07:59:01,492:INFO:the tn solutions: (3877642.6484160004, 3878686.2699840004)\n",
+ "2023-11-03 07:59:01,493:INFO:intervals after: [(430290, 430407)], [(3877643, 3878686)]\n",
+ "2023-11-03 07:59:01,493:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,495:INFO:intervals before: [(430290, 430407)], [(3877643, 3878686)]\n",
+ "2023-11-03 07:59:01,497:INFO:the tp solutions: (429134.3290260006, 430868.0509740007)\n",
+ "2023-11-03 07:59:01,498:INFO:the tn solutions: (3878107.8038119995, 3878915.744588)\n",
+ "2023-11-03 07:59:01,499:INFO:intervals after: [(430290, 430407)], [(3878108, 3878686)]\n",
+ "2023-11-03 07:59:01,500:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,501:INFO:intervals before: [(430290, 430407)], [(3878108, 3878686)]\n",
+ "2023-11-03 07:59:01,502:INFO:the tp solutions: (430289.58480199997, 430407.42519800004)\n",
+ "2023-11-03 07:59:01,503:INFO:the tn solutions: (3878107.8038119995, 3878915.744588)\n",
+ "2023-11-03 07:59:01,504:INFO:intervals after: [(430290, 430407)], [(3878108, 3878686)]\n",
+ "2023-11-03 07:59:01,507:INFO:testing acc, feasible tptn pairs: 118\n",
+ "2023-11-03 07:59:01,517:INFO:testing sens, feasible tptn pairs: 118\n",
+ "2023-11-03 07:59:01,518:INFO:testing spec, feasible tptn pairs: 116\n",
+ "2023-11-03 07:59:01,522:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,522:INFO:final number of intervals: 116\n",
+ "2023-11-03 07:59:01,523:INFO:final number of pairs: 59334\n",
+ "2023-11-03 07:59:01,538:INFO:checking the scores {'acc': 0.9494, 'sens': 0.745, 'spec': 0.9793}\n",
+ "2023-11-03 07:59:01,539:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,540:INFO:intervals before: (0, 577945), (0, 6021255)\n",
+ "2023-11-03 07:59:01,543:INFO:the tp solutions: (430510.07460999995, 430627.97539000004)\n",
+ "2023-11-03 07:59:01,544:INFO:the tn solutions: (5833979.38621, 5835443.523790001)\n",
+ "2023-11-03 07:59:01,545:INFO:intervals after: [(430511, 430627)], [(5833980, 5835443)]\n",
+ "2023-11-03 07:59:01,546:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,548:INFO:intervals before: [(430511, 430627)], [(5833980, 5835443)]\n",
+ "2023-11-03 07:59:01,549:INFO:the tp solutions: (367378.1720900005, 369952.7449100008)\n",
+ "2023-11-03 07:59:01,550:INFO:the tn solutions: (5896000.85349, 5897229.18951)\n",
+ "2023-11-03 07:59:01,551:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,553:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,554:INFO:intervals before: [], []\n",
+ "2023-11-03 07:59:01,555:INFO:the tp solutions: (430510.07460999995, 430627.97539000004)\n",
+ "2023-11-03 07:59:01,557:INFO:the tn solutions: (5896000.85349, 5897229.18951)\n",
+ "2023-11-03 07:59:01,557:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,558:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 07:59:01,559:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 07:59:01,560:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,561:INFO:final number of intervals: 0\n",
+ "2023-11-03 07:59:01,562:INFO:final number of pairs: 0\n"
]
}
],
@@ -164,48 +164,48 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:05,390:INFO:checking the scores {'acc': 0.9898, 'sens': 0.8027, 'spec': 0.998}\n",
- "2023-10-26 23:42:05,392:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,393:INFO:intervals before: (0, 577649), (0, 3960494)\n",
- "2023-10-26 23:42:05,394:INFO:the tp solutions: (463619.93210199993, 463737.772498)\n",
- "2023-10-26 23:42:05,395:INFO:the tn solutions: (4027653.278316, 4028696.899884)\n",
- "2023-10-26 23:42:05,397:INFO:intervals after: [(463620, 463737)], []\n",
- "2023-10-26 23:42:05,398:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,399:INFO:intervals before: [(463620, 463737)], []\n",
- "2023-10-26 23:42:05,401:INFO:the tp solutions: (538414.068426, 540147.7903740001)\n",
- "2023-10-26 23:42:05,402:INFO:the tn solutions: (3952169.0416119997, 3952976.982388)\n",
- "2023-10-26 23:42:05,403:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,404:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,405:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:05,406:INFO:the tp solutions: (463619.93210199993, 463737.772498)\n",
- "2023-10-26 23:42:05,408:INFO:the tn solutions: (3952169.0416119997, 3952976.982388)\n",
- "2023-10-26 23:42:05,409:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,410:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:05,411:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:05,412:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,413:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:05,414:INFO:final number of pairs: 0\n",
- "2023-10-26 23:42:05,434:INFO:checking the scores {'acc': 0.9898, 'sens': 0.8027, 'spec': 0.998}\n",
- "2023-10-26 23:42:05,436:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,438:INFO:intervals before: (0, 577945), (0, 6021255)\n",
- "2023-10-26 23:42:05,439:INFO:the tp solutions: (463857.50110999995, 463975.40189000004)\n",
- "2023-10-26 23:42:05,441:INFO:the tn solutions: (6067239.63971, 6068703.777290001)\n",
- "2023-10-26 23:42:05,442:INFO:intervals after: [(463858, 463975)], []\n",
- "2023-10-26 23:42:05,444:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,445:INFO:intervals before: [(463858, 463975)], []\n",
- "2023-10-26 23:42:05,446:INFO:the tp solutions: (521388.38358999975, 523962.9564100001)\n",
- "2023-10-26 23:42:05,446:INFO:the tn solutions: (6008598.32199, 6009826.65801)\n",
- "2023-10-26 23:42:05,447:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,448:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,450:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:05,451:INFO:the tp solutions: (463857.50110999995, 463975.40189000004)\n",
- "2023-10-26 23:42:05,452:INFO:the tn solutions: (6008598.32199, 6009826.65801)\n",
- "2023-10-26 23:42:05,454:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,457:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:05,458:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:05,459:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,461:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:05,462:INFO:final number of pairs: 0\n"
+ "2023-11-03 07:59:01,624:INFO:checking the scores {'acc': 0.9898, 'sens': 0.8027, 'spec': 0.998}\n",
+ "2023-11-03 07:59:01,626:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,627:INFO:intervals before: (0, 577649), (0, 3960494)\n",
+ "2023-11-03 07:59:01,629:INFO:the tp solutions: (463619.93210199993, 463737.772498)\n",
+ "2023-11-03 07:59:01,630:INFO:the tn solutions: (4027653.278316, 4028696.899884)\n",
+ "2023-11-03 07:59:01,631:INFO:intervals after: [(463620, 463737)], []\n",
+ "2023-11-03 07:59:01,632:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,633:INFO:intervals before: [(463620, 463737)], []\n",
+ "2023-11-03 07:59:01,634:INFO:the tp solutions: (538414.068426, 540147.7903740001)\n",
+ "2023-11-03 07:59:01,635:INFO:the tn solutions: (3952169.0416119997, 3952976.982388)\n",
+ "2023-11-03 07:59:01,636:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,636:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,637:INFO:intervals before: [], []\n",
+ "2023-11-03 07:59:01,639:INFO:the tp solutions: (463619.93210199993, 463737.772498)\n",
+ "2023-11-03 07:59:01,640:INFO:the tn solutions: (3952169.0416119997, 3952976.982388)\n",
+ "2023-11-03 07:59:01,641:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,642:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 07:59:01,642:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 07:59:01,643:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,644:INFO:final number of intervals: 0\n",
+ "2023-11-03 07:59:01,645:INFO:final number of pairs: 0\n",
+ "2023-11-03 07:59:01,673:INFO:checking the scores {'acc': 0.9898, 'sens': 0.8027, 'spec': 0.998}\n",
+ "2023-11-03 07:59:01,675:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,677:INFO:intervals before: (0, 577945), (0, 6021255)\n",
+ "2023-11-03 07:59:01,679:INFO:the tp solutions: (463857.50110999995, 463975.40189000004)\n",
+ "2023-11-03 07:59:01,681:INFO:the tn solutions: (6067239.63971, 6068703.777290001)\n",
+ "2023-11-03 07:59:01,682:INFO:intervals after: [(463858, 463975)], []\n",
+ "2023-11-03 07:59:01,686:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,687:INFO:intervals before: [(463858, 463975)], []\n",
+ "2023-11-03 07:59:01,688:INFO:the tp solutions: (521388.38358999975, 523962.9564100001)\n",
+ "2023-11-03 07:59:01,689:INFO:the tn solutions: (6008598.32199, 6009826.65801)\n",
+ "2023-11-03 07:59:01,692:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,694:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,695:INFO:intervals before: [], []\n",
+ "2023-11-03 07:59:01,696:INFO:the tp solutions: (463857.50110999995, 463975.40189000004)\n",
+ "2023-11-03 07:59:01,697:INFO:the tn solutions: (6008598.32199, 6009826.65801)\n",
+ "2023-11-03 07:59:01,698:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,700:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 07:59:01,701:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 07:59:01,703:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,704:INFO:final number of intervals: 0\n",
+ "2023-11-03 07:59:01,705:INFO:final number of pairs: 0\n"
]
}
],
@@ -282,53 +282,53 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:05,527:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:05,529:INFO:calling the score check with scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}, uncertainty 0.0001, p 32886 and n 192841\n",
- "2023-10-26 23:42:05,531:INFO:checking the scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}\n",
- "2023-10-26 23:42:05,533:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,534:INFO:intervals before: (0, 32886), (0, 192841)\n",
- "2023-10-26 23:42:05,536:INFO:the tp solutions: (18666.027828, 18672.736572)\n",
- "2023-10-26 23:42:05,537:INFO:the tn solutions: (191749.521374, 191802.27842600003)\n",
- "2023-10-26 23:42:05,539:INFO:intervals after: [(18667, 18672)], [(191750, 191802)]\n",
- "2023-10-26 23:42:05,541:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,543:INFO:intervals before: [(18667, 18672)], [(191750, 191802)]\n",
- "2023-10-26 23:42:05,544:INFO:the tp solutions: (18641.497764, 18726.88563600005)\n",
- "2023-10-26 23:42:05,545:INFO:the tn solutions: (191741.42061799997, 191780.760182)\n",
- "2023-10-26 23:42:05,548:INFO:intervals after: [(18667, 18672)], [(191750, 191780)]\n",
- "2023-10-26 23:42:05,550:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,551:INFO:intervals before: [(18667, 18672)], [(191750, 191780)]\n",
- "2023-10-26 23:42:05,553:INFO:the tp solutions: (18666.027828, 18672.736572)\n",
- "2023-10-26 23:42:05,554:INFO:the tn solutions: (191741.42061799997, 191780.760182)\n",
- "2023-10-26 23:42:05,556:INFO:intervals after: [(18667, 18672)], [(191750, 191780)]\n",
- "2023-10-26 23:42:05,557:INFO:testing acc, feasible tptn pairs: 6\n",
- "2023-10-26 23:42:05,559:INFO:testing sens, feasible tptn pairs: 6\n",
- "2023-10-26 23:42:05,561:INFO:testing spec, feasible tptn pairs: 6\n",
- "2023-10-26 23:42:05,564:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,566:INFO:final number of intervals: 6\n",
- "2023-10-26 23:42:05,568:INFO:final number of pairs: 165\n",
- "2023-10-26 23:42:05,569:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:05,571:INFO:calling the score check with scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}, uncertainty 0.0001, p 32893 and n 297067\n",
- "2023-10-26 23:42:05,584:INFO:checking the scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}\n",
- "2023-10-26 23:42:05,587:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,589:INFO:intervals before: (0, 32893), (0, 297067)\n",
- "2023-10-26 23:42:05,590:INFO:the tp solutions: (18670.001013999998, 18676.711186)\n",
- "2023-10-26 23:42:05,592:INFO:the tn solutions: (288911.3408939999, 288985.36290600005)\n",
- "2023-10-26 23:42:05,593:INFO:intervals after: [(18671, 18676)], [(288912, 288985)]\n",
- "2023-10-26 23:42:05,595:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,596:INFO:intervals before: [(18671, 18676)], [(288912, 288985)]\n",
- "2023-10-26 23:42:05,597:INFO:the tp solutions: (12154.326445999963, 12282.239954000048)\n",
- "2023-10-26 23:42:05,599:INFO:the tn solutions: (295373.123966, 295433.725634)\n",
- "2023-10-26 23:42:05,602:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,604:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,606:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:05,607:INFO:the tp solutions: (18670.001013999998, 18676.711186)\n",
- "2023-10-26 23:42:05,609:INFO:the tn solutions: (295373.123966, 295433.725634)\n",
- "2023-10-26 23:42:05,610:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,611:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:05,612:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:05,613:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,615:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:05,616:INFO:final number of pairs: 0\n"
+ "2023-11-03 07:59:01,757:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 07:59:01,758:INFO:calling the score check with scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}, uncertainty 0.0001, p 32886 and n 192841\n",
+ "2023-11-03 07:59:01,761:INFO:checking the scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}\n",
+ "2023-11-03 07:59:01,762:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,764:INFO:intervals before: (0, 32886), (0, 192841)\n",
+ "2023-11-03 07:59:01,765:INFO:the tp solutions: (18666.027828, 18672.736572)\n",
+ "2023-11-03 07:59:01,767:INFO:the tn solutions: (191749.521374, 191802.27842600003)\n",
+ "2023-11-03 07:59:01,771:INFO:intervals after: [(18667, 18672)], [(191750, 191802)]\n",
+ "2023-11-03 07:59:01,772:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,773:INFO:intervals before: [(18667, 18672)], [(191750, 191802)]\n",
+ "2023-11-03 07:59:01,775:INFO:the tp solutions: (18641.497764, 18726.88563600005)\n",
+ "2023-11-03 07:59:01,777:INFO:the tn solutions: (191741.42061799997, 191780.760182)\n",
+ "2023-11-03 07:59:01,779:INFO:intervals after: [(18667, 18672)], [(191750, 191780)]\n",
+ "2023-11-03 07:59:01,780:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,782:INFO:intervals before: [(18667, 18672)], [(191750, 191780)]\n",
+ "2023-11-03 07:59:01,783:INFO:the tp solutions: (18666.027828, 18672.736572)\n",
+ "2023-11-03 07:59:01,784:INFO:the tn solutions: (191741.42061799997, 191780.760182)\n",
+ "2023-11-03 07:59:01,786:INFO:intervals after: [(18667, 18672)], [(191750, 191780)]\n",
+ "2023-11-03 07:59:01,787:INFO:testing acc, feasible tptn pairs: 6\n",
+ "2023-11-03 07:59:01,789:INFO:testing sens, feasible tptn pairs: 6\n",
+ "2023-11-03 07:59:01,790:INFO:testing spec, feasible tptn pairs: 6\n",
+ "2023-11-03 07:59:01,791:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,792:INFO:final number of intervals: 6\n",
+ "2023-11-03 07:59:01,794:INFO:final number of pairs: 165\n",
+ "2023-11-03 07:59:01,795:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 07:59:01,796:INFO:calling the score check with scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}, uncertainty 0.0001, p 32893 and n 297067\n",
+ "2023-11-03 07:59:01,797:INFO:checking the scores {'acc': 0.9323, 'sens': 0.5677, 'spec': 0.9944}\n",
+ "2023-11-03 07:59:01,798:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,799:INFO:intervals before: (0, 32893), (0, 297067)\n",
+ "2023-11-03 07:59:01,800:INFO:the tp solutions: (18670.001013999998, 18676.711186)\n",
+ "2023-11-03 07:59:01,801:INFO:the tn solutions: (288911.3408939999, 288985.36290600005)\n",
+ "2023-11-03 07:59:01,802:INFO:intervals after: [(18671, 18676)], [(288912, 288985)]\n",
+ "2023-11-03 07:59:01,803:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,805:INFO:intervals before: [(18671, 18676)], [(288912, 288985)]\n",
+ "2023-11-03 07:59:01,806:INFO:the tp solutions: (12154.326445999963, 12282.239954000048)\n",
+ "2023-11-03 07:59:01,815:INFO:the tn solutions: (295373.123966, 295433.725634)\n",
+ "2023-11-03 07:59:01,816:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,818:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,819:INFO:intervals before: [], []\n",
+ "2023-11-03 07:59:01,820:INFO:the tp solutions: (18670.001013999998, 18676.711186)\n",
+ "2023-11-03 07:59:01,821:INFO:the tn solutions: (295373.123966, 295433.725634)\n",
+ "2023-11-03 07:59:01,822:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,823:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 07:59:01,824:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 07:59:01,826:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,827:INFO:final number of intervals: 0\n",
+ "2023-11-03 07:59:01,828:INFO:final number of pairs: 0\n"
]
}
],
@@ -394,53 +394,53 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:05,688:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:05,691:INFO:calling the score check with scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}, uncertainty 0.0001, p 29412 and n 194965\n",
- "2023-10-26 23:42:05,694:INFO:checking the scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}\n",
- "2023-10-26 23:42:05,696:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,697:INFO:intervals before: (0, 29412), (0, 194965)\n",
- "2023-10-26 23:42:05,700:INFO:the tp solutions: (21779.527176, 21785.527224)\n",
- "2023-10-26 23:42:05,701:INFO:the tn solutions: (194333.950422, 194385.72337800002)\n",
- "2023-10-26 23:42:05,704:INFO:intervals after: [(21780, 21785)], [(194334, 194385)]\n",
- "2023-10-26 23:42:05,706:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,711:INFO:intervals before: [(21780, 21785)], [(194334, 194385)]\n",
- "2023-10-26 23:42:05,713:INFO:the tp solutions: (24078.562715999986, 24164.108484000026)\n",
- "2023-10-26 23:42:05,715:INFO:the tn solutions: (192001.14207, 192040.91493)\n",
- "2023-10-26 23:42:05,717:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,719:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,721:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:05,724:INFO:the tp solutions: (21779.527176, 21785.527224)\n",
- "2023-10-26 23:42:05,728:INFO:the tn solutions: (192001.14207, 192040.91493)\n",
- "2023-10-26 23:42:05,731:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:05,733:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:05,747:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:05,749:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,751:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:05,753:INFO:final number of pairs: 0\n",
- "2023-10-26 23:42:05,754:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:05,755:INFO:calling the score check with scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}, uncertainty 0.0001, p 29440 and n 300520\n",
- "2023-10-26 23:42:05,756:INFO:checking the scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}\n",
- "2023-10-26 23:42:05,758:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:05,759:INFO:intervals before: (0, 29440), (0, 300520)\n",
- "2023-10-26 23:42:05,761:INFO:the tp solutions: (21800.26112, 21806.266880000003)\n",
- "2023-10-26 23:42:05,766:INFO:the tn solutions: (296010.5452, 296083.86280000006)\n",
- "2023-10-26 23:42:05,768:INFO:intervals after: [(21801, 21806)], [(296011, 296083)]\n",
- "2023-10-26 23:42:05,770:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:05,772:INFO:intervals before: [(21801, 21806)], [(296011, 296083)]\n",
- "2023-10-26 23:42:05,773:INFO:the tp solutions: (21804.011040000012, 21932.62896000006)\n",
- "2023-10-26 23:42:05,774:INFO:the tn solutions: (295951.49496, 296012.80104)\n",
- "2023-10-26 23:42:05,776:INFO:intervals after: [(21805, 21806)], [(296011, 296012)]\n",
- "2023-10-26 23:42:05,778:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:05,780:INFO:intervals before: [(21805, 21806)], [(296011, 296012)]\n",
- "2023-10-26 23:42:05,783:INFO:the tp solutions: (21800.26112, 21806.266880000003)\n",
- "2023-10-26 23:42:05,785:INFO:the tn solutions: (295951.49496, 296012.80104)\n",
- "2023-10-26 23:42:05,798:INFO:intervals after: [(21805, 21806)], [(296011, 296012)]\n",
- "2023-10-26 23:42:05,802:INFO:testing acc, feasible tptn pairs: 2\n",
- "2023-10-26 23:42:05,804:INFO:testing sens, feasible tptn pairs: 1\n",
- "2023-10-26 23:42:05,806:INFO:testing spec, feasible tptn pairs: 1\n",
- "2023-10-26 23:42:05,807:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:05,809:INFO:final number of intervals: 1\n",
- "2023-10-26 23:42:05,810:INFO:final number of pairs: 1\n"
+ "2023-11-03 07:59:01,874:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 07:59:01,876:INFO:calling the score check with scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}, uncertainty 0.0001, p 29412 and n 194965\n",
+ "2023-11-03 07:59:01,877:INFO:checking the scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}\n",
+ "2023-11-03 07:59:01,878:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,880:INFO:intervals before: (0, 29412), (0, 194965)\n",
+ "2023-11-03 07:59:01,881:INFO:the tp solutions: (21779.527176, 21785.527224)\n",
+ "2023-11-03 07:59:01,882:INFO:the tn solutions: (194333.950422, 194385.72337800002)\n",
+ "2023-11-03 07:59:01,883:INFO:intervals after: [(21780, 21785)], [(194334, 194385)]\n",
+ "2023-11-03 07:59:01,886:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,887:INFO:intervals before: [(21780, 21785)], [(194334, 194385)]\n",
+ "2023-11-03 07:59:01,889:INFO:the tp solutions: (24078.562715999986, 24164.108484000026)\n",
+ "2023-11-03 07:59:01,895:INFO:the tn solutions: (192001.14207, 192040.91493)\n",
+ "2023-11-03 07:59:01,896:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,897:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,898:INFO:intervals before: [], []\n",
+ "2023-11-03 07:59:01,899:INFO:the tp solutions: (21779.527176, 21785.527224)\n",
+ "2023-11-03 07:59:01,901:INFO:the tn solutions: (192001.14207, 192040.91493)\n",
+ "2023-11-03 07:59:01,902:INFO:intervals after: [], []\n",
+ "2023-11-03 07:59:01,903:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 07:59:01,904:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 07:59:01,904:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,905:INFO:final number of intervals: 0\n",
+ "2023-11-03 07:59:01,906:INFO:final number of pairs: 0\n",
+ "2023-11-03 07:59:01,907:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 07:59:01,908:INFO:calling the score check with scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}, uncertainty 0.0001, p 29440 and n 300520\n",
+ "2023-11-03 07:59:01,909:INFO:checking the scores {'acc': 0.9633, 'sens': 0.7406, 'spec': 0.9849}\n",
+ "2023-11-03 07:59:01,911:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 07:59:01,918:INFO:intervals before: (0, 29440), (0, 300520)\n",
+ "2023-11-03 07:59:01,920:INFO:the tp solutions: (21800.26112, 21806.266880000003)\n",
+ "2023-11-03 07:59:01,921:INFO:the tn solutions: (296010.5452, 296083.86280000006)\n",
+ "2023-11-03 07:59:01,923:INFO:intervals after: [(21801, 21806)], [(296011, 296083)]\n",
+ "2023-11-03 07:59:01,924:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 07:59:01,925:INFO:intervals before: [(21801, 21806)], [(296011, 296083)]\n",
+ "2023-11-03 07:59:01,926:INFO:the tp solutions: (21804.011040000012, 21932.62896000006)\n",
+ "2023-11-03 07:59:01,927:INFO:the tn solutions: (295951.49496, 296012.80104)\n",
+ "2023-11-03 07:59:01,931:INFO:intervals after: [(21805, 21806)], [(296011, 296012)]\n",
+ "2023-11-03 07:59:01,933:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 07:59:01,935:INFO:intervals before: [(21805, 21806)], [(296011, 296012)]\n",
+ "2023-11-03 07:59:01,936:INFO:the tp solutions: (21800.26112, 21806.266880000003)\n",
+ "2023-11-03 07:59:01,939:INFO:the tn solutions: (295951.49496, 296012.80104)\n",
+ "2023-11-03 07:59:01,941:INFO:intervals after: [(21805, 21806)], [(296011, 296012)]\n",
+ "2023-11-03 07:59:01,942:INFO:testing acc, feasible tptn pairs: 2\n",
+ "2023-11-03 07:59:01,944:INFO:testing sens, feasible tptn pairs: 1\n",
+ "2023-11-03 07:59:01,945:INFO:testing spec, feasible tptn pairs: 1\n",
+ "2023-11-03 07:59:01,946:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 07:59:01,947:INFO:final number of intervals: 1\n",
+ "2023-11-03 07:59:01,948:INFO:final number of pairs: 1\n"
]
}
],
diff --git a/notebooks/illustration/21_retina_stare.ipynb b/notebooks/illustration/bundles/01_retina_stare.ipynb
similarity index 53%
rename from notebooks/illustration/21_retina_stare.ipynb
rename to notebooks/illustration/bundles/01_retina_stare.ipynb
index f7f301a..87e6bc7 100644
--- a/notebooks/illustration/21_retina_stare.ipynb
+++ b/notebooks/illustration/bundles/01_retina_stare.ipynb
@@ -13,8 +13,8 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.retina import (check_stare_vessel_image,\n",
- " check_stare_vessel_aggregated)"
+ "from mlscorecheck.check.bundles.retina import (check_stare_vessel_image,\n",
+ " check_stare_vessel_aggregated)"
]
},
{
@@ -43,42 +43,42 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:15,585:INFO:checking the scores {'acc': 0.4964, 'sens': 0.5793, 'spec': 0.4871, 'bacc': 0.5332}\n",
- "2023-10-26 23:42:15,587:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:15,588:INFO:intervals before: (0, 644053), (0, 7825947)\n",
- "2023-10-26 23:42:15,589:INFO:the tp solutions: (373034.209494, 373165.5963060001)\n",
- "2023-10-26 23:42:15,590:INFO:the tn solutions: (3830478.4636940006, 3832337.7305060006)\n",
- "2023-10-26 23:42:15,591:INFO:intervals after: [(373035, 373165)], [(3830479, 3832337)]\n",
- "2023-10-26 23:42:15,592:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:15,594:INFO:intervals before: [(373035, 373165)], [(3830479, 3832337)]\n",
- "2023-10-26 23:42:15,596:INFO:the tp solutions: (390827.0297060008, 394151.4028940005)\n",
- "2023-10-26 23:42:15,597:INFO:the tn solutions: (3811220.537106, 3812817.0302939997)\n",
- "2023-10-26 23:42:15,598:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:15,599:INFO:evaluating the tp and tn solution for acc and bacc\n",
- "2023-10-26 23:42:15,600:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:15,602:INFO:the tp solutions: (371140.476202914, 371581.7663071766)\n",
- "2023-10-26 23:42:15,603:INFO:the tn solutions: (3832062.293692824, 3834231.4637970864)\n",
- "2023-10-26 23:42:15,604:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:15,605:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:15,607:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:15,609:INFO:the tp solutions: (373034.209494, 373165.5963060001)\n",
- "2023-10-26 23:42:15,611:INFO:the tn solutions: (3811220.537106, 3812817.0302939997)\n",
- "2023-10-26 23:42:15,612:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:15,613:INFO:evaluating the tp and tn solution for sens and bacc\n",
- "2023-10-26 23:42:15,615:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:15,616:INFO:the tp solutions: (373034.209494, 373165.5963060001)\n",
- "2023-10-26 23:42:15,616:INFO:the tn solutions: (3809624.0439179987, 3814413.523482001)\n",
- "2023-10-26 23:42:15,618:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:15,619:INFO:evaluating the tp and tn solution for spec and bacc\n",
- "2023-10-26 23:42:15,621:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:15,622:INFO:the tp solutions: (372902.822682, 373296.98311800003)\n",
- "2023-10-26 23:42:15,624:INFO:the tn solutions: (3811220.537106, 3812817.0302939997)\n",
- "2023-10-26 23:42:15,625:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:15,628:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:15,629:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:15,631:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:15,632:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:15,633:INFO:final number of pairs: 0\n"
+ "2023-11-03 08:14:16,715:INFO:checking the scores {'acc': 0.4964, 'sens': 0.5793, 'spec': 0.4871, 'bacc': 0.5332}\n",
+ "2023-11-03 08:14:16,717:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:14:16,718:INFO:intervals before: (0, 644053), (0, 7825947)\n",
+ "2023-11-03 08:14:16,719:INFO:the tp solutions: (373034.209494, 373165.5963060001)\n",
+ "2023-11-03 08:14:16,720:INFO:the tn solutions: (3830478.4636940006, 3832337.7305060006)\n",
+ "2023-11-03 08:14:16,721:INFO:intervals after: [(373035, 373165)], [(3830479, 3832337)]\n",
+ "2023-11-03 08:14:16,722:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:14:16,723:INFO:intervals before: [(373035, 373165)], [(3830479, 3832337)]\n",
+ "2023-11-03 08:14:16,724:INFO:the tp solutions: (390827.0297060008, 394151.4028940005)\n",
+ "2023-11-03 08:14:16,725:INFO:the tn solutions: (3811220.537106, 3812817.0302939997)\n",
+ "2023-11-03 08:14:16,726:INFO:intervals after: [], []\n",
+ "2023-11-03 08:14:16,727:INFO:evaluating the tp and tn solution for acc and bacc\n",
+ "2023-11-03 08:14:16,728:INFO:intervals before: [], []\n",
+ "2023-11-03 08:14:16,729:INFO:the tp solutions: (371140.476202914, 371581.7663071766)\n",
+ "2023-11-03 08:14:16,730:INFO:the tn solutions: (3832062.293692824, 3834231.4637970864)\n",
+ "2023-11-03 08:14:16,731:INFO:intervals after: [], []\n",
+ "2023-11-03 08:14:16,734:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:14:16,736:INFO:intervals before: [], []\n",
+ "2023-11-03 08:14:16,737:INFO:the tp solutions: (373034.209494, 373165.5963060001)\n",
+ "2023-11-03 08:14:16,738:INFO:the tn solutions: (3811220.537106, 3812817.0302939997)\n",
+ "2023-11-03 08:14:16,739:INFO:intervals after: [], []\n",
+ "2023-11-03 08:14:16,740:INFO:evaluating the tp and tn solution for sens and bacc\n",
+ "2023-11-03 08:14:16,742:INFO:intervals before: [], []\n",
+ "2023-11-03 08:14:16,743:INFO:the tp solutions: (373034.209494, 373165.5963060001)\n",
+ "2023-11-03 08:14:16,744:INFO:the tn solutions: (3809624.0439179987, 3814413.523482001)\n",
+ "2023-11-03 08:14:16,745:INFO:intervals after: [], []\n",
+ "2023-11-03 08:14:16,746:INFO:evaluating the tp and tn solution for spec and bacc\n",
+ "2023-11-03 08:14:16,747:INFO:intervals before: [], []\n",
+ "2023-11-03 08:14:16,748:INFO:the tp solutions: (372902.822682, 373296.98311800003)\n",
+ "2023-11-03 08:14:16,749:INFO:the tn solutions: (3811220.537106, 3812817.0302939997)\n",
+ "2023-11-03 08:14:16,750:INFO:intervals after: [], []\n",
+ "2023-11-03 08:14:16,751:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:14:16,752:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:14:16,753:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:14:16,754:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:14:16,754:INFO:final number of pairs: 0\n"
]
},
{
@@ -135,30 +135,30 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:15,680:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:15,682:INFO:calling the score check with scores {'acc': 0.4699, 'npv': 0.8993, 'f1p': 0.134}, uncertainty 0.0001, p 37701 and n 385799\n",
- "2023-10-26 23:42:15,684:INFO:checking the scores {'acc': 0.4699, 'npv': 0.8993, 'f1p': 0.134}\n",
- "2023-10-26 23:42:15,685:INFO:evaluating the tp and tn solution for acc and npv\n",
- "2023-10-26 23:42:15,687:INFO:intervals before: (0, 37701), (0, 385799)\n",
- "2023-10-26 23:42:15,688:INFO:the tp solutions: (17224.19894829517, 17499.003859606033)\n",
- "2023-10-26 23:42:15,689:INFO:the tn solutions: (181525.47861639896, 181756.76992556825)\n",
- "2023-10-26 23:42:15,691:INFO:intervals after: [(17225, 17499)], [(181526, 181756)]\n",
- "2023-10-26 23:42:15,693:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:42:15,695:INFO:intervals before: [(17225, 17499)], [(181526, 181756)]\n",
- "2023-10-26 23:42:15,696:INFO:the tp solutions: (17350.128609790765, 17387.34610415661)\n",
- "2023-10-26 23:42:15,697:INFO:the tn solutions: (181522.6513884081, 181745.20449418173)\n",
- "2023-10-26 23:42:15,698:INFO:intervals after: [(17351, 17387)], [(181526, 181745)]\n",
- "2023-10-26 23:42:15,699:INFO:evaluating the tp and tn solution for npv and f1p\n",
- "2023-10-26 23:42:15,701:INFO:intervals before: [(17351, 17387)], [(181526, 181745)]\n",
- "2023-10-26 23:42:15,702:INFO:the tp solutions: (17195.366421230152, 17566.700828011573)\n",
- "2023-10-26 23:42:15,703:INFO:the tn solutions: (180009.9078568309, 182954.30415410912)\n",
- "2023-10-26 23:42:15,705:INFO:intervals after: [(17351, 17387)], [(181526, 181745)]\n",
- "2023-10-26 23:42:15,707:INFO:testing acc, feasible tptn pairs: 37\n",
- "2023-10-26 23:42:15,711:INFO:testing npv, feasible tptn pairs: 37\n",
- "2023-10-26 23:42:15,715:INFO:testing f1p, feasible tptn pairs: 37\n",
- "2023-10-26 23:42:15,719:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:15,720:INFO:final number of intervals: 37\n",
- "2023-10-26 23:42:15,721:INFO:final number of pairs: 2600\n"
+ "2023-11-03 08:14:16,800:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:14:16,802:INFO:calling the score check with scores {'acc': 0.4699, 'npv': 0.8993, 'f1p': 0.134}, uncertainty 0.0001, p 37701 and n 385799\n",
+ "2023-11-03 08:14:16,803:INFO:checking the scores {'acc': 0.4699, 'npv': 0.8993, 'f1p': 0.134}\n",
+ "2023-11-03 08:14:16,804:INFO:evaluating the tp and tn solution for acc and npv\n",
+ "2023-11-03 08:14:16,806:INFO:intervals before: (0, 37701), (0, 385799)\n",
+ "2023-11-03 08:14:16,807:INFO:the tp solutions: (17224.19894829517, 17499.003859606033)\n",
+ "2023-11-03 08:14:16,809:INFO:the tn solutions: (181525.47861639896, 181756.76992556825)\n",
+ "2023-11-03 08:14:16,810:INFO:intervals after: [(17225, 17499)], [(181526, 181756)]\n",
+ "2023-11-03 08:14:16,812:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 08:14:16,813:INFO:intervals before: [(17225, 17499)], [(181526, 181756)]\n",
+ "2023-11-03 08:14:16,815:INFO:the tp solutions: (17350.128609790765, 17387.34610415661)\n",
+ "2023-11-03 08:14:16,819:INFO:the tn solutions: (181522.6513884081, 181745.20449418173)\n",
+ "2023-11-03 08:14:16,820:INFO:intervals after: [(17351, 17387)], [(181526, 181745)]\n",
+ "2023-11-03 08:14:16,822:INFO:evaluating the tp and tn solution for npv and f1p\n",
+ "2023-11-03 08:14:16,823:INFO:intervals before: [(17351, 17387)], [(181526, 181745)]\n",
+ "2023-11-03 08:14:16,824:INFO:the tp solutions: (17195.366421230152, 17566.700828011573)\n",
+ "2023-11-03 08:14:16,825:INFO:the tn solutions: (180009.9078568309, 182954.30415410912)\n",
+ "2023-11-03 08:14:16,827:INFO:intervals after: [(17351, 17387)], [(181526, 181745)]\n",
+ "2023-11-03 08:14:16,831:INFO:testing acc, feasible tptn pairs: 37\n",
+ "2023-11-03 08:14:16,834:INFO:testing npv, feasible tptn pairs: 37\n",
+ "2023-11-03 08:14:16,839:INFO:testing f1p, feasible tptn pairs: 37\n",
+ "2023-11-03 08:14:16,843:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:14:16,846:INFO:final number of intervals: 37\n",
+ "2023-11-03 08:14:16,847:INFO:final number of pairs: 2600\n"
]
}
],
diff --git a/notebooks/illustration/22_retina_chasedb1.ipynb b/notebooks/illustration/bundles/02_retina_chasedb1.ipynb
similarity index 58%
rename from notebooks/illustration/22_retina_chasedb1.ipynb
rename to notebooks/illustration/bundles/02_retina_chasedb1.ipynb
index d258fc2..3379bf0 100644
--- a/notebooks/illustration/22_retina_chasedb1.ipynb
+++ b/notebooks/illustration/bundles/02_retina_chasedb1.ipynb
@@ -13,8 +13,8 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.retina import (check_chasedb1_vessel_image,\n",
- " check_chasedb1_vessel_aggregated)"
+ "from mlscorecheck.check.bundles.retina import (check_chasedb1_vessel_image,\n",
+ " check_chasedb1_vessel_aggregated)"
]
},
{
@@ -43,27 +43,27 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:26,586:INFO:checking the scores {'acc': 0.5063, 'sens': 0.4147, 'spec': 0.5126}\n",
- "2023-10-26 23:42:26,588:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:26,589:INFO:intervals before: (0, 1861974), (0, 24991146)\n",
- "2023-10-26 23:42:26,590:INFO:the tp solutions: (771970.6964520001, 772350.539148)\n",
- "2023-10-26 23:42:26,591:INFO:the tn solutions: (12820645.098612, 12826502.977788001)\n",
- "2023-10-26 23:42:26,592:INFO:intervals after: [(771971, 772350)], [(12820646, 12826502)]\n",
- "2023-10-26 23:42:26,593:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:26,594:INFO:intervals before: [(771971, 772350)], [(12820646, 12826502)]\n",
- "2023-10-26 23:42:26,594:INFO:the tp solutions: (779985.101267999, 790561.3315320034)\n",
- "2023-10-26 23:42:26,595:INFO:the tn solutions: (12807912.342707997, 12813010.536492)\n",
- "2023-10-26 23:42:26,596:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:26,597:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:26,598:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:26,600:INFO:the tp solutions: (771970.6964520001, 772350.539148)\n",
- "2023-10-26 23:42:26,601:INFO:the tn solutions: (12807912.342707997, 12813010.536492)\n",
- "2023-10-26 23:42:26,602:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:26,603:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:26,604:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:26,605:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:26,605:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:26,606:INFO:final number of pairs: 0\n"
+ "2023-11-03 08:31:57,349:INFO:checking the scores {'acc': 0.5063, 'sens': 0.4147, 'spec': 0.5126}\n",
+ "2023-11-03 08:31:57,350:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:31:57,351:INFO:intervals before: (0, 1861974), (0, 24991146)\n",
+ "2023-11-03 08:31:57,353:INFO:the tp solutions: (771970.6964520001, 772350.539148)\n",
+ "2023-11-03 08:31:57,354:INFO:the tn solutions: (12820645.098612, 12826502.977788001)\n",
+ "2023-11-03 08:31:57,355:INFO:intervals after: [(771971, 772350)], [(12820646, 12826502)]\n",
+ "2023-11-03 08:31:57,356:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:31:57,357:INFO:intervals before: [(771971, 772350)], [(12820646, 12826502)]\n",
+ "2023-11-03 08:31:57,357:INFO:the tp solutions: (779985.101267999, 790561.3315320034)\n",
+ "2023-11-03 08:31:57,358:INFO:the tn solutions: (12807912.342707997, 12813010.536492)\n",
+ "2023-11-03 08:31:57,358:INFO:intervals after: [], []\n",
+ "2023-11-03 08:31:57,359:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:31:57,360:INFO:intervals before: [], []\n",
+ "2023-11-03 08:31:57,360:INFO:the tp solutions: (771970.6964520001, 772350.539148)\n",
+ "2023-11-03 08:31:57,362:INFO:the tn solutions: (12807912.342707997, 12813010.536492)\n",
+ "2023-11-03 08:31:57,363:INFO:intervals after: [], []\n",
+ "2023-11-03 08:31:57,363:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:31:57,364:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:31:57,365:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:31:57,367:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:31:57,368:INFO:final number of pairs: 0\n"
]
},
{
@@ -120,30 +120,30 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:26,713:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:26,715:INFO:calling the score check with scores {'acc': 0.4457, 'sens': 0.0051, 'spec': 0.4706}, uncertainty 0.0001, p 51133 and n 907907\n",
- "2023-10-26 23:42:26,716:INFO:checking the scores {'acc': 0.4457, 'sens': 0.0051, 'spec': 0.4706}\n",
- "2023-10-26 23:42:26,718:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:26,719:INFO:intervals before: (0, 51133), (0, 907907)\n",
- "2023-10-26 23:42:26,720:INFO:the tp solutions: (255.56273399999998, 265.993866)\n",
- "2023-10-26 23:42:26,720:INFO:the tn solutions: (427080.312054, 427286.387346)\n",
- "2023-10-26 23:42:26,721:INFO:intervals after: [(256, 265)], [(427081, 427286)]\n",
- "2023-10-26 23:42:26,722:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:26,723:INFO:intervals before: [(256, 265)], [(427081, 427286)]\n",
- "2023-10-26 23:42:26,725:INFO:the tp solutions: (-7.334794000082184, 373.5223939999705)\n",
- "2023-10-26 23:42:26,726:INFO:the tn solutions: (427168.427686, 427353.64071400004)\n",
- "2023-10-26 23:42:26,729:INFO:intervals after: [(256, 265)], [(427169, 427286)]\n",
- "2023-10-26 23:42:26,730:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:26,731:INFO:intervals before: [(256, 265)], [(427169, 427286)]\n",
- "2023-10-26 23:42:26,732:INFO:the tp solutions: (255.56273399999998, 265.993866)\n",
- "2023-10-26 23:42:26,733:INFO:the tn solutions: (427168.427686, 427353.64071400004)\n",
- "2023-10-26 23:42:26,734:INFO:intervals after: [(256, 265)], [(427169, 427286)]\n",
- "2023-10-26 23:42:26,735:INFO:testing acc, feasible tptn pairs: 10\n",
- "2023-10-26 23:42:26,736:INFO:testing sens, feasible tptn pairs: 10\n",
- "2023-10-26 23:42:26,737:INFO:testing spec, feasible tptn pairs: 10\n",
- "2023-10-26 23:42:26,739:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:26,742:INFO:final number of intervals: 10\n",
- "2023-10-26 23:42:26,743:INFO:final number of pairs: 1105\n"
+ "2023-11-03 08:31:57,403:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:31:57,405:INFO:calling the score check with scores {'acc': 0.4457, 'sens': 0.0051, 'spec': 0.4706}, uncertainty 0.0001, p 51133 and n 907907\n",
+ "2023-11-03 08:31:57,406:INFO:checking the scores {'acc': 0.4457, 'sens': 0.0051, 'spec': 0.4706}\n",
+ "2023-11-03 08:31:57,407:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:31:57,408:INFO:intervals before: (0, 51133), (0, 907907)\n",
+ "2023-11-03 08:31:57,409:INFO:the tp solutions: (255.56273399999998, 265.993866)\n",
+ "2023-11-03 08:31:57,410:INFO:the tn solutions: (427080.312054, 427286.387346)\n",
+ "2023-11-03 08:31:57,411:INFO:intervals after: [(256, 265)], [(427081, 427286)]\n",
+ "2023-11-03 08:31:57,411:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:31:57,412:INFO:intervals before: [(256, 265)], [(427081, 427286)]\n",
+ "2023-11-03 08:31:57,413:INFO:the tp solutions: (-7.334794000082184, 373.5223939999705)\n",
+ "2023-11-03 08:31:57,415:INFO:the tn solutions: (427168.427686, 427353.64071400004)\n",
+ "2023-11-03 08:31:57,416:INFO:intervals after: [(256, 265)], [(427169, 427286)]\n",
+ "2023-11-03 08:31:57,417:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:31:57,418:INFO:intervals before: [(256, 265)], [(427169, 427286)]\n",
+ "2023-11-03 08:31:57,420:INFO:the tp solutions: (255.56273399999998, 265.993866)\n",
+ "2023-11-03 08:31:57,421:INFO:the tn solutions: (427168.427686, 427353.64071400004)\n",
+ "2023-11-03 08:31:57,422:INFO:intervals after: [(256, 265)], [(427169, 427286)]\n",
+ "2023-11-03 08:31:57,423:INFO:testing acc, feasible tptn pairs: 10\n",
+ "2023-11-03 08:31:57,425:INFO:testing sens, feasible tptn pairs: 10\n",
+ "2023-11-03 08:31:57,425:INFO:testing spec, feasible tptn pairs: 10\n",
+ "2023-11-03 08:31:57,426:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:31:57,427:INFO:final number of intervals: 10\n",
+ "2023-11-03 08:31:57,428:INFO:final number of pairs: 1105\n"
]
}
],
diff --git a/notebooks/illustration/bundles/03_retina_hrf.ipynb b/notebooks/illustration/bundles/03_retina_hrf.ipynb
new file mode 100644
index 0000000..5b931bb
--- /dev/null
+++ b/notebooks/illustration/bundles/03_retina_hrf.ipynb
@@ -0,0 +1,227 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Testing the consistency of performance scored reported for the HRF retinal vessel segmentation dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.bundles.retina import (check_hrf_vessel_image,\n",
+ " check_hrf_vessel_aggregated)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Aggregated scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}\n",
+ "k = 4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 08:26:16,122:INFO:checking the scores {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}\n",
+ "2023-11-03 08:26:16,124:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:26:16,125:INFO:intervals before: (0, 28395044), (0, 282712163)\n",
+ "2023-11-03 08:26:16,126:INFO:the tp solutions: (16082896.131512, 16088688.720488)\n",
+ "2023-11-03 08:26:16,127:INFO:the tn solutions: (134486577.25309798, 134555835.712302)\n",
+ "2023-11-03 08:26:16,128:INFO:intervals after: [(16082897, 16088688)], [(134486578, 134555835)]\n",
+ "2023-11-03 08:26:16,128:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:26:16,129:INFO:intervals before: [(16082897, 16088688)], [(134486578, 134555835)]\n",
+ "2023-11-03 08:26:16,130:INFO:the tp solutions: (16258151.907959998, 16379291.059439987)\n",
+ "2023-11-03 08:26:16,131:INFO:the tn solutions: (134259440.784374, 134317114.065626)\n",
+ "2023-11-03 08:26:16,132:INFO:intervals after: [], []\n",
+ "2023-11-03 08:26:16,133:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:26:16,134:INFO:intervals before: [], []\n",
+ "2023-11-03 08:26:16,135:INFO:the tp solutions: (16082896.131512, 16088688.720488)\n",
+ "2023-11-03 08:26:16,136:INFO:the tn solutions: (134259440.784374, 134317114.065626)\n",
+ "2023-11-03 08:26:16,137:INFO:intervals after: [], []\n",
+ "2023-11-03 08:26:16,138:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:26:16,139:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:26:16,140:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:26:16,141:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:26:16,142:INFO:final number of pairs: 0\n",
+ "2023-11-03 08:26:16,182:INFO:checking the scores {'acc': 0.4841, 'sens': 0.5665, 'spec': 0.475}\n",
+ "2023-11-03 08:26:16,183:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:26:16,185:INFO:intervals before: (0, 28399210), (0, 339941270)\n",
+ "2023-11-03 08:26:16,185:INFO:the tp solutions: (16085255.745579999, 16091049.18442)\n",
+ "2023-11-03 08:26:16,186:INFO:the tn solutions: (162185006.45462, 162265941.35138002)\n",
+ "2023-11-03 08:26:16,187:INFO:intervals after: [(16085256, 16091049)], [(162185007, 162265941)]\n",
+ "2023-11-03 08:26:16,190:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:26:16,191:INFO:intervals before: [(16085256, 16091049)], [(162185007, 162265941)]\n",
+ "2023-11-03 08:26:16,192:INFO:the tp solutions: (16769278.379500002, 16913767.8565)\n",
+ "2023-11-03 08:26:16,192:INFO:the tn solutions: (161437429.24046, 161506777.25954)\n",
+ "2023-11-03 08:26:16,193:INFO:intervals after: [], []\n",
+ "2023-11-03 08:26:16,194:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:26:16,196:INFO:intervals before: [], []\n",
+ "2023-11-03 08:26:16,197:INFO:the tp solutions: (16085255.745579999, 16091049.18442)\n",
+ "2023-11-03 08:26:16,198:INFO:the tn solutions: (161437429.24046, 161506777.25954)\n",
+ "2023-11-03 08:26:16,199:INFO:intervals after: [], []\n",
+ "2023-11-03 08:26:16,200:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:26:16,201:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:26:16,202:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:26:16,203:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:26:16,204:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{'inconsistency_fov_mos': False,\n",
+ " 'inconsistency_fov_som': True,\n",
+ " 'inconsistency_all_mos': False,\n",
+ " 'inconsistency_all_som': True}"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_hrf_vessel_aggregated(scores=scores,\n",
+ " eps=10**(-k),\n",
+ " imageset='all',\n",
+ " verbosity=0)\n",
+ "results['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Image level scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "img_identifier = '13_h'\n",
+ "scores = {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 08:26:16,244:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:26:16,246:INFO:calling the score check with scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}, uncertainty 0.0001, p 712528 and n 6199936\n",
+ "2023-11-03 08:26:16,247:INFO:checking the scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}\n",
+ "2023-11-03 08:26:16,249:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:26:16,250:INFO:intervals before: (0, 712528), (0, 6199936)\n",
+ "2023-11-03 08:26:16,251:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
+ "2023-11-03 08:26:16,253:INFO:the tn solutions: (3484179.340416, 3485734.8387840004)\n",
+ "2023-11-03 08:26:16,255:INFO:intervals after: [(359683, 359828)], [(3484180, 3485734)]\n",
+ "2023-11-03 08:26:16,256:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:26:16,257:INFO:intervals before: [(359683, 359828)], [(3484180, 3485734)]\n",
+ "2023-11-03 08:26:16,258:INFO:the tp solutions: (358390.98639999935, 361065.91600000067)\n",
+ "2023-11-03 08:26:16,259:INFO:the tn solutions: (3484351.632128, 3485616.419072001)\n",
+ "2023-11-03 08:26:16,259:INFO:intervals after: [(359683, 359828)], [(3484352, 3485616)]\n",
+ "2023-11-03 08:26:16,260:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:26:16,261:INFO:intervals before: [(359683, 359828)], [(3484352, 3485616)]\n",
+ "2023-11-03 08:26:16,261:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
+ "2023-11-03 08:26:16,262:INFO:the tn solutions: (3484351.632128, 3485616.419072001)\n",
+ "2023-11-03 08:26:16,262:INFO:intervals after: [(359683, 359828)], [(3484352, 3485616)]\n",
+ "2023-11-03 08:26:16,267:INFO:testing acc, feasible tptn pairs: 146\n",
+ "2023-11-03 08:26:16,278:INFO:testing sens, feasible tptn pairs: 146\n",
+ "2023-11-03 08:26:16,279:INFO:testing spec, feasible tptn pairs: 144\n",
+ "2023-11-03 08:26:16,284:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:26:16,285:INFO:final number of intervals: 144\n",
+ "2023-11-03 08:26:16,286:INFO:final number of pairs: 180054\n",
+ "2023-11-03 08:26:16,287:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:26:16,288:INFO:calling the score check with scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}, uncertainty 0.0001, p 712528 and n 7472816\n",
+ "2023-11-03 08:26:16,289:INFO:checking the scores {'acc': 0.5562, 'sens': 0.5049, 'spec': 0.5621}\n",
+ "2023-11-03 08:26:16,290:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:26:16,291:INFO:intervals before: (0, 712528), (0, 7472816)\n",
+ "2023-11-03 08:26:16,292:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
+ "2023-11-03 08:26:16,295:INFO:the tn solutions: (4192025.362656, 4193840.528544001)\n",
+ "2023-11-03 08:26:16,296:INFO:intervals after: [(359683, 359828)], [(4192026, 4193840)]\n",
+ "2023-11-03 08:26:16,297:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:26:16,298:INFO:intervals before: [(359683, 359828)], [(4192026, 4193840)]\n",
+ "2023-11-03 08:26:16,299:INFO:the tp solutions: (350621.3268799996, 353815.5915200012)\n",
+ "2023-11-03 08:26:16,300:INFO:the tn solutions: (4199707.646368, 4201232.100832)\n",
+ "2023-11-03 08:26:16,301:INFO:intervals after: [], []\n",
+ "2023-11-03 08:26:16,302:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:26:16,303:INFO:intervals before: [], []\n",
+ "2023-11-03 08:26:16,304:INFO:the tp solutions: (359682.709344, 359828.065056)\n",
+ "2023-11-03 08:26:16,305:INFO:the tn solutions: (4199707.646368, 4201232.100832)\n",
+ "2023-11-03 08:26:16,306:INFO:intervals after: [], []\n",
+ "2023-11-03 08:26:16,307:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:26:16,308:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:26:16,308:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:26:16,313:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:26:16,314:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{'inconsistency_fov': False, 'inconsistency_all': True}"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_hrf_vessel_image(image_identifier=img_identifier,\n",
+ " scores=scores,\n",
+ " eps=10**(-k))\n",
+ "results['inconsistency']"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/24_retina_diaretdb0.ipynb b/notebooks/illustration/bundles/04_retina_diaretdb0.ipynb
similarity index 70%
rename from notebooks/illustration/24_retina_diaretdb0.ipynb
rename to notebooks/illustration/bundles/04_retina_diaretdb0.ipynb
index 3291b7c..a2d68f4 100644
--- a/notebooks/illustration/24_retina_diaretdb0.ipynb
+++ b/notebooks/illustration/bundles/04_retina_diaretdb0.ipynb
@@ -13,7 +13,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.retina import check_diaretdb0_class"
+ "from mlscorecheck.check.bundles.retina import check_diaretdb0_class"
]
},
{
@@ -36,27 +36,27 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:39,720:INFO:checking the scores {'acc': 0.4271, 'sens': 0.406, 'spec': 0.4765}\n",
- "2023-10-26 23:42:39,722:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:39,723:INFO:intervals before: (0, 419), (0, 301)\n",
- "2023-10-26 23:42:39,724:INFO:the tp solutions: (170.07126200000002, 170.15673800000002)\n",
- "2023-10-26 23:42:39,726:INFO:the tn solutions: (137.28182199999998, 137.51417799999993)\n",
- "2023-10-26 23:42:39,727:INFO:intervals after: [(1, 0)], [(1, 0)]\n",
- "2023-10-26 23:42:39,728:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:39,729:INFO:intervals before: [(1, 0)], [(1, 0)]\n",
- "2023-10-26 23:42:39,729:INFO:the tp solutions: (163.981358, 164.18964199999996)\n",
- "2023-10-26 23:42:39,731:INFO:the tn solutions: (143.39579799999998, 143.457202)\n",
- "2023-10-26 23:42:39,733:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:39,734:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:39,735:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:39,737:INFO:the tp solutions: (170.07126200000002, 170.15673800000002)\n",
- "2023-10-26 23:42:39,738:INFO:the tn solutions: (143.39579799999998, 143.457202)\n",
- "2023-10-26 23:42:39,739:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:39,740:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:39,741:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:39,742:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:39,743:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:39,744:INFO:final number of pairs: 0\n"
+ "2023-11-03 08:36:52,978:INFO:checking the scores {'acc': 0.4271, 'sens': 0.406, 'spec': 0.4765}\n",
+ "2023-11-03 08:36:52,980:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:36:52,982:INFO:intervals before: (0, 419), (0, 301)\n",
+ "2023-11-03 08:36:52,986:INFO:the tp solutions: (170.07126200000002, 170.15673800000002)\n",
+ "2023-11-03 08:36:52,987:INFO:the tn solutions: (137.28182199999998, 137.51417799999993)\n",
+ "2023-11-03 08:36:52,988:INFO:intervals after: [(1, 0)], [(1, 0)]\n",
+ "2023-11-03 08:36:52,989:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:36:52,990:INFO:intervals before: [(1, 0)], [(1, 0)]\n",
+ "2023-11-03 08:36:52,992:INFO:the tp solutions: (163.981358, 164.18964199999996)\n",
+ "2023-11-03 08:36:52,996:INFO:the tn solutions: (143.39579799999998, 143.457202)\n",
+ "2023-11-03 08:36:52,998:INFO:intervals after: [], []\n",
+ "2023-11-03 08:36:53,000:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:36:53,002:INFO:intervals before: [], []\n",
+ "2023-11-03 08:36:53,004:INFO:the tp solutions: (170.07126200000002, 170.15673800000002)\n",
+ "2023-11-03 08:36:53,006:INFO:the tn solutions: (143.39579799999998, 143.457202)\n",
+ "2023-11-03 08:36:53,007:INFO:intervals after: [], []\n",
+ "2023-11-03 08:36:53,008:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:36:53,010:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:36:53,013:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:36:53,015:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:36:53,017:INFO:final number of pairs: 0\n"
]
},
{
@@ -67,7 +67,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de62e6a75dc2403486b0c6eb659a1778-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de62e6a75dc2403486b0c6eb659a1778-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/655bcca08de849fb80a0beb1b22ad102-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/655bcca08de849fb80a0beb1b22ad102-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -81,26 +81,26 @@
"Cgl0004I processed model has 3 rows, 18 columns (18 integer (0 of which binary)) and 36 elements\n",
"Cbc0031I 4 added rows had average density of 13.25\n",
"Cbc0013I At root node, 4 cuts changed objective from 0 to 0 in 68 passes\n",
- "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 1 (Gomory) - 194 row cuts average 16.5 elements, 0 column cuts (0 active) in 0.002 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.002 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 194 row cuts average 16.5 elements, 0 column cuts (0 active) in 0.005 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 13 row cuts average 10.8 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 13 row cuts average 10.8 elements, 0 column cuts (0 active) in 0.002 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
- "Cbc0014I Cut generator 6 (TwoMirCuts) - 19 row cuts average 10.4 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 6 (TwoMirCuts) - 19 row cuts average 10.4 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
"Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.03 seconds)\n",
- "Cbc0016I Integer solution of 0 found by strong branching after 224 iterations and 53 nodes (0.04 seconds)\n",
- "Cbc0001I Search completed - best objective 0, took 224 iterations and 53 nodes (0.04 seconds)\n",
+ "Cbc0016I Integer solution of 0 found by strong branching after 224 iterations and 53 nodes (0.03 seconds)\n",
+ "Cbc0001I Search completed - best objective 0, took 224 iterations and 53 nodes (0.03 seconds)\n",
"Cbc0032I Strong branching done 150 times (226 iterations), fathomed 11 nodes and fixed 4 variables\n",
"Cbc0035I Maximum depth 28, 0 variables fixed on reduced cost\n",
"Cuts at root node changed objective from 0 to 0\n",
- "Probing was tried 68 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
- "Gomory was tried 68 times and created 194 cuts of which 0 were active after adding rounds of cuts (0.002 seconds)\n",
+ "Probing was tried 68 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.002 seconds)\n",
+ "Gomory was tried 68 times and created 194 cuts of which 0 were active after adding rounds of cuts (0.005 seconds)\n",
"Knapsack was tried 68 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
"Clique was tried 68 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
- "MixedIntegerRounding2 was tried 68 times and created 13 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "MixedIntegerRounding2 was tried 68 times and created 13 cuts of which 0 were active after adding rounds of cuts (0.002 seconds)\n",
"FlowCover was tried 68 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
- "TwoMirCuts was tried 68 times and created 19 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 68 times and created 19 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
"ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
"\n",
"Result - Optimal solution found\n",
@@ -108,11 +108,11 @@
"Objective value: 0.00000000\n",
"Enumerated nodes: 53\n",
"Total iterations: 224\n",
- "Time (CPU seconds): 0.02\n",
- "Time (Wallclock seconds): 0.04\n",
+ "Time (CPU seconds): 0.03\n",
+ "Time (Wallclock seconds): 0.03\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.02 (Wallclock seconds): 0.04\n",
+ "Total time (CPU seconds): 0.03 (Wallclock seconds): 0.03\n",
"\n"
]
},
diff --git a/notebooks/illustration/25_retina_diaretdb1.ipynb b/notebooks/illustration/bundles/05_retina_diaretdb1.ipynb
similarity index 53%
rename from notebooks/illustration/25_retina_diaretdb1.ipynb
rename to notebooks/illustration/bundles/05_retina_diaretdb1.ipynb
index a47e475..85276d7 100644
--- a/notebooks/illustration/25_retina_diaretdb1.ipynb
+++ b/notebooks/illustration/bundles/05_retina_diaretdb1.ipynb
@@ -9,13 +9,13 @@
},
{
"cell_type": "code",
- "execution_count": 8,
+ "execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.retina import (check_diaretdb1_class,\n",
- " check_diaretdb1_segmentation_image,\n",
- " check_diaretdb1_segmentation_aggregated)"
+ "from mlscorecheck.check.bundles.retina import (check_diaretdb1_class,\n",
+ " check_diaretdb1_segmentation_image,\n",
+ " check_diaretdb1_segmentation_aggregated)"
]
},
{
@@ -27,7 +27,7 @@
},
{
"cell_type": "code",
- "execution_count": 9,
+ "execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
@@ -36,53 +36,53 @@
},
{
"cell_type": "code",
- "execution_count": 10,
+ "execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:45,697:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:45,725:INFO:calling the score check with scores {'acc': 0.3115, 'sens': 1.0, 'spec': 0.0455, 'f1p': 0.4474}, uncertainty 0.0001, p 17 and n 44\n",
- "2023-10-26 23:42:45,737:INFO:checking the scores {'acc': 0.3115, 'sens': 1.0, 'spec': 0.0455, 'f1p': 0.4474}\n",
- "2023-10-26 23:42:45,781:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:45,797:INFO:intervals before: (0, 17), (0, 44)\n",
- "2023-10-26 23:42:45,801:INFO:the tp solutions: (16.998266, 17)\n",
- "2023-10-26 23:42:45,804:INFO:the tn solutions: (1.995277999999999, 2.009456)\n",
- "2023-10-26 23:42:45,817:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,821:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:45,824:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,827:INFO:the tp solutions: (16.988789999999998, 17.01021)\n",
- "2023-10-26 23:42:45,828:INFO:the tn solutions: (1.997512, 2.006488)\n",
- "2023-10-26 23:42:45,830:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,832:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:42:45,837:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,838:INFO:the tp solutions: (16.992033649999456, 17.011099682210617)\n",
- "2023-10-26 23:42:45,840:INFO:the tn solutions: (1.9784082467188469, 2.021469888719955)\n",
- "2023-10-26 23:42:45,843:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,845:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:45,850:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,852:INFO:the tp solutions: (16.998266, 17)\n",
- "2023-10-26 23:42:45,855:INFO:the tn solutions: (1.997512, 2.006488)\n",
- "2023-10-26 23:42:45,857:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,859:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:42:45,861:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,863:INFO:the tp solutions: (16.998266, 17)\n",
- "2023-10-26 23:42:45,865:INFO:the tn solutions: (1.9853953396141368, 2.0313616425738457)\n",
- "2023-10-26 23:42:45,867:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,869:INFO:evaluating the tp and tn solution for spec and f1p\n",
- "2023-10-26 23:42:45,873:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,875:INFO:the tp solutions: (16.994684060802395, 17.00725629596689)\n",
- "2023-10-26 23:42:45,877:INFO:the tn solutions: (1.997512, 2.006488)\n",
- "2023-10-26 23:42:45,878:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
- "2023-10-26 23:42:45,891:INFO:testing acc, feasible tptn pairs: 1\n",
- "2023-10-26 23:42:45,911:INFO:testing sens, feasible tptn pairs: 1\n",
- "2023-10-26 23:42:45,913:INFO:testing spec, feasible tptn pairs: 1\n",
- "2023-10-26 23:42:45,919:INFO:testing f1p, feasible tptn pairs: 1\n",
- "2023-10-26 23:42:45,923:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:45,926:INFO:final number of intervals: 1\n",
- "2023-10-26 23:42:45,929:INFO:final number of pairs: 1\n"
+ "2023-11-03 08:39:51,362:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:39:51,363:INFO:calling the score check with scores {'acc': 0.3115, 'sens': 1.0, 'spec': 0.0455, 'f1p': 0.4474}, uncertainty 0.0001, p 17 and n 44\n",
+ "2023-11-03 08:39:51,363:INFO:checking the scores {'acc': 0.3115, 'sens': 1.0, 'spec': 0.0455, 'f1p': 0.4474}\n",
+ "2023-11-03 08:39:51,365:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:39:51,365:INFO:intervals before: (0, 17), (0, 44)\n",
+ "2023-11-03 08:39:51,366:INFO:the tp solutions: (16.998266, 17)\n",
+ "2023-11-03 08:39:51,367:INFO:the tn solutions: (1.995277999999999, 2.009456)\n",
+ "2023-11-03 08:39:51,367:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,370:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:39:51,371:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,371:INFO:the tp solutions: (16.988789999999998, 17.01021)\n",
+ "2023-11-03 08:39:51,372:INFO:the tn solutions: (1.997512, 2.006488)\n",
+ "2023-11-03 08:39:51,373:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,374:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 08:39:51,376:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,377:INFO:the tp solutions: (16.992033649999456, 17.011099682210617)\n",
+ "2023-11-03 08:39:51,378:INFO:the tn solutions: (1.9784082467188469, 2.021469888719955)\n",
+ "2023-11-03 08:39:51,380:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,381:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:39:51,382:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,383:INFO:the tp solutions: (16.998266, 17)\n",
+ "2023-11-03 08:39:51,384:INFO:the tn solutions: (1.997512, 2.006488)\n",
+ "2023-11-03 08:39:51,384:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,385:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 08:39:51,387:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,389:INFO:the tp solutions: (16.998266, 17)\n",
+ "2023-11-03 08:39:51,390:INFO:the tn solutions: (1.9853953396141368, 2.0313616425738457)\n",
+ "2023-11-03 08:39:51,392:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,393:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 08:39:51,394:INFO:intervals before: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,394:INFO:the tp solutions: (16.994684060802395, 17.00725629596689)\n",
+ "2023-11-03 08:39:51,395:INFO:the tn solutions: (1.997512, 2.006488)\n",
+ "2023-11-03 08:39:51,396:INFO:intervals after: [(17, 17)], [(2, 2)]\n",
+ "2023-11-03 08:39:51,396:INFO:testing acc, feasible tptn pairs: 1\n",
+ "2023-11-03 08:39:51,397:INFO:testing sens, feasible tptn pairs: 1\n",
+ "2023-11-03 08:39:51,398:INFO:testing spec, feasible tptn pairs: 1\n",
+ "2023-11-03 08:39:51,398:INFO:testing f1p, feasible tptn pairs: 1\n",
+ "2023-11-03 08:39:51,399:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:39:51,400:INFO:final number of intervals: 1\n",
+ "2023-11-03 08:39:51,401:INFO:final number of pairs: 1\n"
]
},
{
@@ -91,7 +91,7 @@
"False"
]
},
- "execution_count": 10,
+ "execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
@@ -114,7 +114,7 @@
},
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
@@ -123,91 +123,91 @@
},
{
"cell_type": "code",
- "execution_count": 12,
+ "execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:46,057:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:46,059:INFO:calling the score check with scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}, uncertainty 0.0001, p 132038 and n 1275197\n",
- "2023-10-26 23:42:46,062:INFO:checking the scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}\n",
- "2023-10-26 23:42:46,064:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:46,066:INFO:intervals before: (0, 132038), (0, 1275197)\n",
- "2023-10-26 23:42:46,067:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
- "2023-10-26 23:42:46,068:INFO:the tn solutions: (802783.778254, 803097.7899460001)\n",
- "2023-10-26 23:42:46,071:INFO:intervals after: [(6629, 6654)], [(802784, 803097)]\n",
- "2023-10-26 23:42:46,072:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:46,073:INFO:intervals before: [(6629, 6654)], [(802784, 803097)]\n",
- "2023-10-26 23:42:46,074:INFO:the tp solutions: (20344.303535999963, 20891.519664000138)\n",
- "2023-10-26 23:42:46,075:INFO:the tn solutions: (788834.3138059999, 789094.4539940001)\n",
- "2023-10-26 23:42:46,076:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,078:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:42:46,081:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:46,084:INFO:the tp solutions: (5382.620227408646, 5448.411631373825)\n",
- "2023-10-26 23:42:46,085:INFO:the tn solutions: (803820.7609366671, 804512.8769120976)\n",
- "2023-10-26 23:42:46,086:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,087:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:46,089:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:46,090:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
- "2023-10-26 23:42:46,091:INFO:the tn solutions: (788834.3138059999, 789094.4539940001)\n",
- "2023-10-26 23:42:46,092:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,094:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:42:46,097:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:46,100:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
- "2023-10-26 23:42:46,102:INFO:the tn solutions: (654261.5122493437, 681171.3956943698)\n",
- "2023-10-26 23:42:46,116:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,119:INFO:evaluating the tp and tn solution for spec and f1p\n",
- "2023-10-26 23:42:46,121:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:46,122:INFO:the tp solutions: (5518.761209550405, 5585.29854943852)\n",
- "2023-10-26 23:42:46,123:INFO:the tn solutions: (788834.3138059999, 789094.4539940001)\n",
- "2023-10-26 23:42:46,125:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,126:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:46,128:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:46,129:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:46,130:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:46,132:INFO:final number of pairs: 0\n",
- "2023-10-26 23:42:46,135:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:42:46,138:INFO:calling the score check with scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}, uncertainty 0.0001, p 132038 and n 1595962\n",
- "2023-10-26 23:42:46,141:INFO:checking the scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}\n",
- "2023-10-26 23:42:46,143:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:46,144:INFO:intervals before: (0, 132038), (0, 1595962)\n",
- "2023-10-26 23:42:46,146:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
- "2023-10-26 23:42:46,148:INFO:the tn solutions: (987287.1647239999, 987666.6124760002)\n",
- "2023-10-26 23:42:46,156:INFO:intervals after: [(6629, 6654)], [(987288, 987666)]\n",
- "2023-10-26 23:42:46,157:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:46,159:INFO:intervals before: [(6629, 6654)], [(987288, 987666)]\n",
- "2023-10-26 23:42:46,161:INFO:the tp solutions: (6357.6664759998675, 7035.754724000231)\n",
- "2023-10-26 23:42:46,163:INFO:the tn solutions: (987258.901276, 987584.4775240001)\n",
- "2023-10-26 23:42:46,164:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,166:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:42:46,168:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,170:INFO:the tp solutions: (6609.5341239822355, 6690.322013746083)\n",
- "2023-10-26 23:42:46,172:INFO:the tn solutions: (987043.5818456482, 987893.4586647607)\n",
- "2023-10-26 23:42:46,173:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,174:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:46,180:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,183:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
- "2023-10-26 23:42:46,184:INFO:the tn solutions: (987258.901276, 987584.4775240001)\n",
- "2023-10-26 23:42:46,185:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,187:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:42:46,188:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,190:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
- "2023-10-26 23:42:46,191:INFO:the tn solutions: (971371.2748457015, 1005633.7660187001)\n",
- "2023-10-26 23:42:46,193:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,194:INFO:evaluating the tp and tn solution for spec and f1p\n",
- "2023-10-26 23:42:46,196:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,198:INFO:the tp solutions: (6610.432677150226, 6690.258074705212)\n",
- "2023-10-26 23:42:46,199:INFO:the tn solutions: (987258.901276, 987584.4775240001)\n",
- "2023-10-26 23:42:46,201:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
- "2023-10-26 23:42:46,205:INFO:testing acc, feasible tptn pairs: 26\n",
- "2023-10-26 23:42:46,211:INFO:testing sens, feasible tptn pairs: 26\n",
- "2023-10-26 23:42:46,213:INFO:testing spec, feasible tptn pairs: 26\n",
- "2023-10-26 23:42:46,218:INFO:testing f1p, feasible tptn pairs: 26\n",
- "2023-10-26 23:42:46,224:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:46,231:INFO:final number of intervals: 26\n",
- "2023-10-26 23:42:46,235:INFO:final number of pairs: 7293\n"
+ "2023-11-03 08:39:51,428:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:39:51,429:INFO:calling the score check with scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}, uncertainty 0.0001, p 132038 and n 1275197\n",
+ "2023-11-03 08:39:51,430:INFO:checking the scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}\n",
+ "2023-11-03 08:39:51,431:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:39:51,432:INFO:intervals before: (0, 132038), (0, 1275197)\n",
+ "2023-11-03 08:39:51,433:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
+ "2023-11-03 08:39:51,435:INFO:the tn solutions: (802783.778254, 803097.7899460001)\n",
+ "2023-11-03 08:39:51,436:INFO:intervals after: [(6629, 6654)], [(802784, 803097)]\n",
+ "2023-11-03 08:39:51,437:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:39:51,438:INFO:intervals before: [(6629, 6654)], [(802784, 803097)]\n",
+ "2023-11-03 08:39:51,439:INFO:the tp solutions: (20344.303535999963, 20891.519664000138)\n",
+ "2023-11-03 08:39:51,440:INFO:the tn solutions: (788834.3138059999, 789094.4539940001)\n",
+ "2023-11-03 08:39:51,441:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,442:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 08:39:51,443:INFO:intervals before: [], []\n",
+ "2023-11-03 08:39:51,443:INFO:the tp solutions: (5382.620227408646, 5448.411631373825)\n",
+ "2023-11-03 08:39:51,444:INFO:the tn solutions: (803820.7609366671, 804512.8769120976)\n",
+ "2023-11-03 08:39:51,445:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,445:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:39:51,446:INFO:intervals before: [], []\n",
+ "2023-11-03 08:39:51,447:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
+ "2023-11-03 08:39:51,447:INFO:the tn solutions: (788834.3138059999, 789094.4539940001)\n",
+ "2023-11-03 08:39:51,448:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,449:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 08:39:51,450:INFO:intervals before: [], []\n",
+ "2023-11-03 08:39:51,450:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
+ "2023-11-03 08:39:51,453:INFO:the tn solutions: (654261.5122493437, 681171.3956943698)\n",
+ "2023-11-03 08:39:51,454:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,455:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 08:39:51,457:INFO:intervals before: [], []\n",
+ "2023-11-03 08:39:51,458:INFO:the tp solutions: (5518.761209550405, 5585.29854943852)\n",
+ "2023-11-03 08:39:51,459:INFO:the tn solutions: (788834.3138059999, 789094.4539940001)\n",
+ "2023-11-03 08:39:51,459:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,460:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:39:51,461:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:39:51,461:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:39:51,462:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:39:51,463:INFO:final number of pairs: 0\n",
+ "2023-11-03 08:39:51,464:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:39:51,464:INFO:calling the score check with scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}, uncertainty 0.0001, p 132038 and n 1595962\n",
+ "2023-11-03 08:39:51,465:INFO:checking the scores {'acc': 0.5753, 'sens': 0.0503, 'spec': 0.6187, 'f1p': 0.0178}\n",
+ "2023-11-03 08:39:51,466:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:39:51,467:INFO:intervals before: (0, 132038), (0, 1595962)\n",
+ "2023-11-03 08:39:51,468:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
+ "2023-11-03 08:39:51,468:INFO:the tn solutions: (987287.1647239999, 987666.6124760002)\n",
+ "2023-11-03 08:39:51,474:INFO:intervals after: [(6629, 6654)], [(987288, 987666)]\n",
+ "2023-11-03 08:39:51,475:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:39:51,476:INFO:intervals before: [(6629, 6654)], [(987288, 987666)]\n",
+ "2023-11-03 08:39:51,477:INFO:the tp solutions: (6357.6664759998675, 7035.754724000231)\n",
+ "2023-11-03 08:39:51,478:INFO:the tn solutions: (987258.901276, 987584.4775240001)\n",
+ "2023-11-03 08:39:51,479:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,480:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 08:39:51,481:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,482:INFO:the tp solutions: (6609.5341239822355, 6690.322013746083)\n",
+ "2023-11-03 08:39:51,483:INFO:the tn solutions: (987043.5818456482, 987893.4586647607)\n",
+ "2023-11-03 08:39:51,484:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,485:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:39:51,485:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,486:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
+ "2023-11-03 08:39:51,487:INFO:the tn solutions: (987258.901276, 987584.4775240001)\n",
+ "2023-11-03 08:39:51,489:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,489:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 08:39:51,491:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,492:INFO:the tp solutions: (6628.043524, 6654.979275999999)\n",
+ "2023-11-03 08:39:51,493:INFO:the tn solutions: (971371.2748457015, 1005633.7660187001)\n",
+ "2023-11-03 08:39:51,498:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,499:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 08:39:51,500:INFO:intervals before: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,500:INFO:the tp solutions: (6610.432677150226, 6690.258074705212)\n",
+ "2023-11-03 08:39:51,501:INFO:the tn solutions: (987258.901276, 987584.4775240001)\n",
+ "2023-11-03 08:39:51,502:INFO:intervals after: [(6629, 6654)], [(987288, 987584)]\n",
+ "2023-11-03 08:39:51,504:INFO:testing acc, feasible tptn pairs: 26\n",
+ "2023-11-03 08:39:51,507:INFO:testing sens, feasible tptn pairs: 26\n",
+ "2023-11-03 08:39:51,508:INFO:testing spec, feasible tptn pairs: 26\n",
+ "2023-11-03 08:39:51,509:INFO:testing f1p, feasible tptn pairs: 26\n",
+ "2023-11-03 08:39:51,511:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:39:51,511:INFO:final number of intervals: 26\n",
+ "2023-11-03 08:39:51,512:INFO:final number of pairs: 7293\n"
]
},
{
@@ -216,7 +216,7 @@
"{'inconsistency_fov': True, 'inconsistency_all': False}"
]
},
- "execution_count": 12,
+ "execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
@@ -239,7 +239,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
@@ -248,55 +248,55 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:42:46,316:INFO:checking the scores {'acc': 0.7143, 'sens': 0.3775, 'spec': 0.7244}\n",
- "2023-10-26 23:42:46,318:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:46,319:INFO:intervals before: (0, 1250927), (0, 26893773)\n",
- "2023-10-26 23:42:46,321:INFO:the tp solutions: (472097.347946, 472352.537054)\n",
- "2023-10-26 23:42:46,323:INFO:the tn solutions: (19628535.913546003, 19634532.621454004)\n",
- "2023-10-26 23:42:46,324:INFO:intervals after: [(472098, 472352)], [(19628536, 19634532)]\n",
- "2023-10-26 23:42:46,325:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:46,327:INFO:intervals before: [(472098, 472352)], [(19628536, 19634532)]\n",
- "2023-10-26 23:42:46,327:INFO:the tp solutions: (616296.1245540008, 627523.9730460048)\n",
- "2023-10-26 23:42:46,330:INFO:the tn solutions: (19479105.996354, 19484592.326046)\n",
- "2023-10-26 23:42:46,331:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,334:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:46,342:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:46,343:INFO:the tp solutions: (472097.347946, 472352.537054)\n",
- "2023-10-26 23:42:46,345:INFO:the tn solutions: (19479105.996354, 19484592.326046)\n",
- "2023-10-26 23:42:46,347:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,348:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:46,351:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:46,353:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:46,355:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:46,356:INFO:final number of pairs: 0\n",
- "2023-10-26 23:42:46,493:INFO:checking the scores {'acc': 0.7143, 'sens': 0.3775, 'spec': 0.7244}\n",
- "2023-10-26 23:42:46,495:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:42:46,500:INFO:intervals before: (0, 1252069), (0, 33307931)\n",
- "2023-10-26 23:42:46,501:INFO:the tp solutions: (472528.33646200004, 472783.758538)\n",
- "2023-10-26 23:42:46,502:INFO:the tn solutions: (24209899.121462, 24217204.783538006)\n",
- "2023-10-26 23:42:46,504:INFO:intervals after: [(472529, 472783)], [(24209900, 24217204)]\n",
- "2023-10-26 23:42:46,505:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:42:46,506:INFO:intervals before: [(472529, 472783)], [(24209900, 24217204)]\n",
- "2023-10-26 23:42:46,507:INFO:the tp solutions: (551020.2546379976, 564865.3125620037)\n",
- "2023-10-26 23:42:46,512:INFO:the tn solutions: (24124867.807438, 24131662.625362)\n",
- "2023-10-26 23:42:46,514:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,515:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:42:46,517:INFO:intervals before: [], []\n",
- "2023-10-26 23:42:46,518:INFO:the tp solutions: (472528.33646200004, 472783.758538)\n",
- "2023-10-26 23:42:46,519:INFO:the tn solutions: (24124867.807438, 24131662.625362)\n",
- "2023-10-26 23:42:46,520:INFO:intervals after: [], []\n",
- "2023-10-26 23:42:46,522:INFO:testing acc, feasible tptn pairs: 0\n",
- "2023-10-26 23:42:46,523:INFO:no more feasible tp,tn pairs left\n",
- "2023-10-26 23:42:46,524:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:42:46,525:INFO:final number of intervals: 0\n",
- "2023-10-26 23:42:46,527:INFO:final number of pairs: 0\n"
+ "2023-11-03 08:39:51,539:INFO:checking the scores {'acc': 0.7143, 'sens': 0.3775, 'spec': 0.7244}\n",
+ "2023-11-03 08:39:51,541:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:39:51,542:INFO:intervals before: (0, 1250927), (0, 26893773)\n",
+ "2023-11-03 08:39:51,543:INFO:the tp solutions: (472097.347946, 472352.537054)\n",
+ "2023-11-03 08:39:51,543:INFO:the tn solutions: (19628535.913546003, 19634532.621454004)\n",
+ "2023-11-03 08:39:51,544:INFO:intervals after: [(472098, 472352)], [(19628536, 19634532)]\n",
+ "2023-11-03 08:39:51,545:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:39:51,546:INFO:intervals before: [(472098, 472352)], [(19628536, 19634532)]\n",
+ "2023-11-03 08:39:51,546:INFO:the tp solutions: (616296.1245540008, 627523.9730460048)\n",
+ "2023-11-03 08:39:51,547:INFO:the tn solutions: (19479105.996354, 19484592.326046)\n",
+ "2023-11-03 08:39:51,548:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,548:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:39:51,549:INFO:intervals before: [], []\n",
+ "2023-11-03 08:39:51,550:INFO:the tp solutions: (472097.347946, 472352.537054)\n",
+ "2023-11-03 08:39:51,551:INFO:the tn solutions: (19479105.996354, 19484592.326046)\n",
+ "2023-11-03 08:39:51,552:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,558:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:39:51,559:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:39:51,560:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:39:51,561:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:39:51,562:INFO:final number of pairs: 0\n",
+ "2023-11-03 08:39:51,582:INFO:checking the scores {'acc': 0.7143, 'sens': 0.3775, 'spec': 0.7244}\n",
+ "2023-11-03 08:39:51,583:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:39:51,584:INFO:intervals before: (0, 1252069), (0, 33307931)\n",
+ "2023-11-03 08:39:51,585:INFO:the tp solutions: (472528.33646200004, 472783.758538)\n",
+ "2023-11-03 08:39:51,586:INFO:the tn solutions: (24209899.121462, 24217204.783538006)\n",
+ "2023-11-03 08:39:51,587:INFO:intervals after: [(472529, 472783)], [(24209900, 24217204)]\n",
+ "2023-11-03 08:39:51,588:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:39:51,590:INFO:intervals before: [(472529, 472783)], [(24209900, 24217204)]\n",
+ "2023-11-03 08:39:51,591:INFO:the tp solutions: (551020.2546379976, 564865.3125620037)\n",
+ "2023-11-03 08:39:51,592:INFO:the tn solutions: (24124867.807438, 24131662.625362)\n",
+ "2023-11-03 08:39:51,593:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,594:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:39:51,595:INFO:intervals before: [], []\n",
+ "2023-11-03 08:39:51,595:INFO:the tp solutions: (472528.33646200004, 472783.758538)\n",
+ "2023-11-03 08:39:51,596:INFO:the tn solutions: (24124867.807438, 24131662.625362)\n",
+ "2023-11-03 08:39:51,597:INFO:intervals after: [], []\n",
+ "2023-11-03 08:39:51,597:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-03 08:39:51,602:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-03 08:39:51,603:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:39:51,604:INFO:final number of intervals: 0\n",
+ "2023-11-03 08:39:51,605:INFO:final number of pairs: 0\n"
]
},
{
@@ -307,7 +307,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f5a1320a23cf4bfe93c723fe6dacf0e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f5a1320a23cf4bfe93c723fe6dacf0e9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5aff7dc80e034994b4911dff3c0e9491-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5aff7dc80e034994b4911dff3c0e9491-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -347,7 +347,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d02cdd183b84f8280f1a08cdad1b774-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d02cdd183b84f8280f1a08cdad1b774-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4983d761810e47df8533a8df8a880cc7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4983d761810e47df8533a8df8a880cc7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -381,7 +381,7 @@
"Time (Wallclock seconds): 0.00\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
"\n"
]
},
@@ -394,7 +394,7 @@
" 'inconsistency_all_mos': False}"
]
},
- "execution_count": 14,
+ "execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
diff --git a/notebooks/illustration/bundles/06_drishti_gs.ipynb b/notebooks/illustration/bundles/06_drishti_gs.ipynb
new file mode 100644
index 0000000..e428419
--- /dev/null
+++ b/notebooks/illustration/bundles/06_drishti_gs.ipynb
@@ -0,0 +1,273 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Testing the consistency of performance scored reported for the DRISHTI_GS retinal image dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.bundles.retina import (check_drishti_gs_segmentation_image,\n",
+ " check_drishti_gs_segmentation_aggregated)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Image level scores, optic disk segmentation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 08:45:43,688:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:45:43,689:INFO:calling the score check with scores {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}, uncertainty 0.0001, p 118933 and n 3487307\n",
+ "2023-11-03 08:45:43,691:INFO:checking the scores {'acc': 0.5966, 'sens': 0.3, 'spec': 0.6067, 'f1p': 0.0468}\n",
+ "2023-11-03 08:45:43,693:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:45:43,695:INFO:intervals before: (0, 118933), (0, 3487307)\n",
+ "2023-11-03 08:45:43,696:INFO:the tp solutions: (35667.768834, 35692.031166)\n",
+ "2023-11-03 08:45:43,697:INFO:the tn solutions: (2115422.916354, 2116182.851646)\n",
+ "2023-11-03 08:45:43,697:INFO:intervals after: [(35668, 35692)], [(2115423, 2116182)]\n",
+ "2023-11-03 08:45:43,698:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:45:43,699:INFO:intervals before: [(35668, 35692)], [(2115423, 2116182)]\n",
+ "2023-11-03 08:45:43,700:INFO:the tp solutions: (35010.0853059995, 36457.16889400035)\n",
+ "2023-11-03 08:45:43,702:INFO:the tn solutions: (2115393.4515859997, 2116104.8622140004)\n",
+ "2023-11-03 08:45:43,703:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,703:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 08:45:43,705:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,707:INFO:the tp solutions: (35622.01445335527, 35803.38811508206)\n",
+ "2023-11-03 08:45:43,709:INFO:the tn solutions: (2114840.8000603416, 2116699.5836843182)\n",
+ "2023-11-03 08:45:43,710:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,711:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:45:43,712:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,713:INFO:the tp solutions: (35667.768834, 35692.031166)\n",
+ "2023-11-03 08:45:43,713:INFO:the tn solutions: (2115393.4515859997, 2116104.8622140004)\n",
+ "2023-11-03 08:45:43,714:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,715:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 08:45:43,716:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,716:INFO:the tp solutions: (35667.768834, 35692.031166)\n",
+ "2023-11-03 08:45:43,717:INFO:the tn solutions: (2104083.976312527, 2130248.7999003753)\n",
+ "2023-11-03 08:45:43,719:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,719:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 08:45:43,720:INFO:intervals before: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,721:INFO:the tp solutions: (35624.9728226002, 35801.4215434727)\n",
+ "2023-11-03 08:45:43,721:INFO:the tn solutions: (2115393.4515859997, 2116104.8622140004)\n",
+ "2023-11-03 08:45:43,722:INFO:intervals after: [(35668, 35692)], [(2115423, 2116104)]\n",
+ "2023-11-03 08:45:43,723:INFO:testing acc, feasible tptn pairs: 25\n",
+ "2023-11-03 08:45:43,725:INFO:testing sens, feasible tptn pairs: 25\n",
+ "2023-11-03 08:45:43,726:INFO:testing spec, feasible tptn pairs: 24\n",
+ "2023-11-03 08:45:43,726:INFO:testing f1p, feasible tptn pairs: 24\n",
+ "2023-11-03 08:45:43,729:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:45:43,730:INFO:final number of intervals: 24\n",
+ "2023-11-03 08:45:43,731:INFO:final number of pairs: 15900\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_drishti_gs_segmentation_image(image_identifier='053',\n",
+ " confidence=0.75,\n",
+ " target='OD',\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "results['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Aggregated scores, optic disk segmentation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = {'acc': 0.4767, 'sens': 0.4845, 'spec': 0.4765, 'f1p': 0.0512}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 08:45:43,783:INFO:checking the scores {'acc': 0.4767, 'sens': 0.4845, 'spec': 0.4765, 'f1p': 0.0512}\n",
+ "2023-11-03 08:45:43,784:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:45:43,785:INFO:intervals before: (0, 5414553), (0, 180494984)\n",
+ "2023-11-03 08:45:43,786:INFO:the tp solutions: (2622798.644094, 2623903.2129059997)\n",
+ "2023-11-03 08:45:43,787:INFO:the tn solutions: (85980210.30222, 86019240.41657999)\n",
+ "2023-11-03 08:45:43,787:INFO:intervals after: [(2622799, 2623903)], [(85980211, 86019240)]\n",
+ "2023-11-03 08:45:43,790:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:45:43,791:INFO:intervals before: [(2622799, 2623903)], [(85980211, 86019240)]\n",
+ "2023-11-03 08:45:43,791:INFO:the tp solutions: (2579843.150758013, 2654589.6730419993)\n",
+ "2023-11-03 08:45:43,792:INFO:the tn solutions: (85987449.387632, 86024270.36436799)\n",
+ "2023-11-03 08:45:43,794:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,794:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 08:45:43,795:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,798:INFO:the tp solutions: (2618908.280150996, 2630954.653441351)\n",
+ "2023-11-03 08:45:43,799:INFO:the tn solutions: (85953650.68004246, 86042649.8249844)\n",
+ "2023-11-03 08:45:43,800:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,801:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:45:43,801:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,803:INFO:the tp solutions: (2622798.644094, 2623903.2129059997)\n",
+ "2023-11-03 08:45:43,803:INFO:the tn solutions: (85987449.387632, 86024270.36436799)\n",
+ "2023-11-03 08:45:43,804:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,804:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 08:45:43,806:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,807:INFO:the tp solutions: (2622798.644094, 2623903.2129059997)\n",
+ "2023-11-03 08:45:43,807:INFO:the tn solutions: (85490212.09562817, 86628542.44030103)\n",
+ "2023-11-03 08:45:43,810:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,811:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-03 08:45:43,811:INFO:intervals before: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,812:INFO:the tp solutions: (2618878.4015550935, 2630578.4368279246)\n",
+ "2023-11-03 08:45:43,813:INFO:the tn solutions: (85987449.387632, 86024270.36436799)\n",
+ "2023-11-03 08:45:43,814:INFO:intervals after: [(2622799, 2623903)], [(85987450, 86019240)]\n",
+ "2023-11-03 08:45:43,833:INFO:testing acc, feasible tptn pairs: 1105\n",
+ "2023-11-03 08:45:43,886:INFO:testing sens, feasible tptn pairs: 1105\n",
+ "2023-11-03 08:45:43,887:INFO:testing spec, feasible tptn pairs: 1093\n",
+ "2023-11-03 08:45:43,956:INFO:testing f1p, feasible tptn pairs: 1093\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/66c9789ac4b9417c935be1ef903cdc9a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/66c9789ac4b9417c935be1ef903cdc9a-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 625 RHS\n",
+ "At line 632 BOUNDS\n",
+ "At line 736 ENDATA\n",
+ "Problem MODEL has 6 rows, 103 columns and 408 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0004I processed model has 3 rows, 102 columns (102 integer (0 of which binary)) and 204 elements\n",
+ "Cbc0012I Integer solution of 0 found by DiveCoefficient after 0 iterations and 0 nodes (0.00 seconds)\n",
+ "Cbc0001I Search completed - best objective 0, took 0 iterations and 0 nodes (0.00 seconds)\n",
+ "Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
+ "Cuts at root node changed objective from 0 to 0\n",
+ "Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "\n",
+ "Result - Optimal solution found\n",
+ "\n",
+ "Objective value: 0.00000000\n",
+ "Enumerated nodes: 0\n",
+ "Total iterations: 0\n",
+ "Time (CPU seconds): 0.00\n",
+ "Time (Wallclock seconds): 0.00\n",
+ "\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-03 08:45:44,009:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:45:44,010:INFO:final number of intervals: 1093\n",
+ "2023-11-03 08:45:44,011:INFO:final number of pairs: 33744189\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{'inconsistency_som': False, 'inconsistency_mos': False}"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_drishti_gs_segmentation_aggregated(subset='test',\n",
+ " confidence=0.75,\n",
+ " target='OD',\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "results['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/27_tpehg.ipynb b/notebooks/illustration/bundles/07_tpehg.ipynb
similarity index 86%
rename from notebooks/illustration/27_tpehg.ipynb
rename to notebooks/illustration/bundles/07_tpehg.ipynb
index ac2299f..832486b 100644
--- a/notebooks/illustration/27_tpehg.ipynb
+++ b/notebooks/illustration/bundles/07_tpehg.ipynb
@@ -9,11 +9,11 @@
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.ehg import check_tpehg\n",
+ "from mlscorecheck.check.bundles.ehg import check_tpehg\n",
"from mlscorecheck.aggregated import kfolds_generator"
]
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
@@ -37,7 +37,7 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": 4,
"metadata": {},
"outputs": [
{
@@ -46,7 +46,7 @@
"918"
]
},
- "execution_count": 3,
+ "execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
@@ -57,7 +57,7 @@
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
@@ -68,7 +68,7 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 6,
"metadata": {},
"outputs": [
{
@@ -79,7 +79,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4dd632419d9548c589242210ffa06d3f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4dd632419d9548c589242210ffa06d3f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00b3060a3747439db96cd95f7cc02e63-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00b3060a3747439db96cd95f7cc02e63-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -99,7 +99,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3b71d503954844bbab3a0b069eb7c3c1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3b71d503954844bbab3a0b069eb7c3c1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/77bec2eceb334bcabb6c56001993a0cc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/77bec2eceb334bcabb6c56001993a0cc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -119,7 +119,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f00c63685314b37b368b47e3bcabec7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f00c63685314b37b368b47e3bcabec7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7438e2c0da64492293ba007449623380-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7438e2c0da64492293ba007449623380-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -140,7 +140,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/092604b93e0c411095db142653c946ac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/092604b93e0c411095db142653c946ac-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eff72845268d4c718cf3e2421244c6ac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eff72845268d4c718cf3e2421244c6ac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -161,7 +161,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c679c48903374be39254804dc0406d07-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c679c48903374be39254804dc0406d07-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2eccd3397ee4458cb270d4efecd69504-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2eccd3397ee4458cb270d4efecd69504-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -182,7 +182,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5938edff21dd48a89121a3c26e0865fa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5938edff21dd48a89121a3c26e0865fa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bbc1450b340344118ee6be43dede823d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bbc1450b340344118ee6be43dede823d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -203,7 +203,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/648200eef79d4f03ac704f9275a7276e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/648200eef79d4f03ac704f9275a7276e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6787c821c9a945bb8f6e9d73f9d35c73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6787c821c9a945bb8f6e9d73f9d35c73-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -224,7 +224,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/402d2f32356e45c3bf9934391f5bdb58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/402d2f32356e45c3bf9934391f5bdb58-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/905a1c9d1e204d0cb83a06ec7bfe73ec-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/905a1c9d1e204d0cb83a06ec7bfe73ec-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -245,7 +245,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e8523a8d26c241819e635c81206bd54f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e8523a8d26c241819e635c81206bd54f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dfa1150ddccc4757a5b4affb80a69e18-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dfa1150ddccc4757a5b4affb80a69e18-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -266,7 +266,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/888825520f6a4ea88216185816bb2e5a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/888825520f6a4ea88216185816bb2e5a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/edd0050fba8643b7b2cbe4852cbf69ec-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/edd0050fba8643b7b2cbe4852cbf69ec-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -288,7 +288,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f7a7ec18e654fadaf22e00885209416-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f7a7ec18e654fadaf22e00885209416-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/45cae1c64e6346aaa231be0edd466192-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/45cae1c64e6346aaa231be0edd466192-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -310,7 +310,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6cc1373f038641389c0f2ae8806606aa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6cc1373f038641389c0f2ae8806606aa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c70bcc78acca47da99b95d1b441110f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c70bcc78acca47da99b95d1b441110f9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -349,7 +349,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d0e18a55f6aa46eba5b65630d50792d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d0e18a55f6aa46eba5b65630d50792d8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1e2567e04aed43418e5d8ec9093414f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1e2567e04aed43418e5d8ec9093414f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -388,7 +388,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53b697e3f5e0472eae43ff1708ec81c1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53b697e3f5e0472eae43ff1708ec81c1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba8ce96b9a2a452d9b819f74c92d1637-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba8ce96b9a2a452d9b819f74c92d1637-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -427,7 +427,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e26d9919e41f437db114513dbfb99312-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e26d9919e41f437db114513dbfb99312-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5c73eab3a1164e559a2d4a0c98ee2a52-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5c73eab3a1164e559a2d4a0c98ee2a52-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -445,7 +445,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea5a6672766144c68bd85579057aecc5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea5a6672766144c68bd85579057aecc5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4d88ed80781f485794826db28e226e56-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4d88ed80781f485794826db28e226e56-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -463,7 +463,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/738a578e9ba74f8d89ce3d85fddc0d40-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/738a578e9ba74f8d89ce3d85fddc0d40-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/09e252288cc949f28543af4574fba4a2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/09e252288cc949f28543af4574fba4a2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -481,7 +481,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9583835dc0f140c18221f8baeba33753-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9583835dc0f140c18221f8baeba33753-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f89234ceb77c4e50b264e5527cc7ecdd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f89234ceb77c4e50b264e5527cc7ecdd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -501,7 +501,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5eac10c77b948188b94a6a7c06d51a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5eac10c77b948188b94a6a7c06d51a1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68f2f8f6371849f198bad791216b77fe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68f2f8f6371849f198bad791216b77fe-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -540,7 +540,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7281e0f814fe421ba166525e40fcca2f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7281e0f814fe421ba166525e40fcca2f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/55f7ef1f653c4bab8d15acf403854817-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/55f7ef1f653c4bab8d15acf403854817-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -561,7 +561,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/384d7618acac4eb180a3d523ff75bed4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/384d7618acac4eb180a3d523ff75bed4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d7b3273fe5e84ac9896776c04d2e329e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d7b3273fe5e84ac9896776c04d2e329e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -582,7 +582,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4ed53a5d6a6c49e696904b2868dd64f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4ed53a5d6a6c49e696904b2868dd64f5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc50ba4b1fdb430a9c31e3c89cd65d95-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc50ba4b1fdb430a9c31e3c89cd65d95-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -603,7 +603,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c690f91fcf054c949a1b6ac879647917-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c690f91fcf054c949a1b6ac879647917-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fff5702ff95242fd95a2fc1e1de7bffa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fff5702ff95242fd95a2fc1e1de7bffa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -624,7 +624,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e11de2982c824a479e41a18a0318aadb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e11de2982c824a479e41a18a0318aadb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d36fe69b12b419ab684599714ca84da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d36fe69b12b419ab684599714ca84da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -645,7 +645,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/114f29dfda234ce9b81cb618f903d1c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/114f29dfda234ce9b81cb618f903d1c0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f5ac3b918f84c69992c1c2eb1649d62-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f5ac3b918f84c69992c1c2eb1649d62-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -666,7 +666,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c850d332370443a1a8281adc00279fd3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c850d332370443a1a8281adc00279fd3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42cf68d18e8c4ebe933b017163008ca1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42cf68d18e8c4ebe933b017163008ca1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -688,7 +688,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78b80cbaf805476bae47238a7d5d8624-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78b80cbaf805476bae47238a7d5d8624-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2f8fce39a2d84b2a8a539762d53e2369-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2f8fce39a2d84b2a8a539762d53e2369-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -710,7 +710,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5450ecf11ba04580bc186446f645c3e4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5450ecf11ba04580bc186446f645c3e4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f88b241ca08d4943bc2e7129a2535039-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f88b241ca08d4943bc2e7129a2535039-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -744,12 +744,18 @@
"\n",
"Option for printingOptions changed from normal to all\n",
"Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
- "\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a3cb741c1a47477ab36b42989a36e974-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a3cb741c1a47477ab36b42989a36e974-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7b531b08564541678a54be98b5f5a769-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7b531b08564541678a54be98b5f5a769-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -788,7 +794,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/838c6631e7a5487e870965d1c495b53c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/838c6631e7a5487e870965d1c495b53c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/38bcec6f26ff4d699874d416bec950f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/38bcec6f26ff4d699874d416bec950f2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -806,7 +812,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/222cbaf87c1d4a1ba609a073aa8524d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/222cbaf87c1d4a1ba609a073aa8524d9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2270f2d5ae6f4f7b8a849ae3b10d8e69-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2270f2d5ae6f4f7b8a849ae3b10d8e69-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -824,7 +830,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de7cd8844342470489253e33a482031c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de7cd8844342470489253e33a482031c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1b108a3777d4379ab3cf1639e1f4c3d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1b108a3777d4379ab3cf1639e1f4c3d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -842,7 +848,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d399551cee9c43c99cb9ce5a0289f066-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d399551cee9c43c99cb9ce5a0289f066-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89c5ab58413b4693bfeeb8e587ad83ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89c5ab58413b4693bfeeb8e587ad83ee-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -860,7 +866,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a5b97cfc130240f4b6ca697da32914e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a5b97cfc130240f4b6ca697da32914e7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/65683525555849148c3251e562f2b947-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/65683525555849148c3251e562f2b947-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -881,7 +887,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/70170435d41b4cdc9de422661b30c8ce-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/70170435d41b4cdc9de422661b30c8ce-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6257328f6c1a4ea9927140ea903f0b5c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6257328f6c1a4ea9927140ea903f0b5c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -930,7 +936,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3076cc7160074e4eb42a6e902edd4167-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3076cc7160074e4eb42a6e902edd4167-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d32568843d8944f29c482c61d285e98b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d32568843d8944f29c482c61d285e98b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -979,7 +985,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/422a1ba4f5594e538dd5689d97148fb2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/422a1ba4f5594e538dd5689d97148fb2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82ae2b74575e45898b55ae9386a2f4ff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82ae2b74575e45898b55ae9386a2f4ff-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1028,7 +1034,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/67b05623cfba4809b0d6c4735fb8dafe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/67b05623cfba4809b0d6c4735fb8dafe-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/098514e04a0d45829aeb9fb265b772d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/098514e04a0d45829aeb9fb265b772d8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1049,7 +1055,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/79813c5964114c3485fde3e69bef4934-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/79813c5964114c3485fde3e69bef4934-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/71b85f4549054ce1965ff2306ed12ce8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/71b85f4549054ce1965ff2306ed12ce8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1071,7 +1077,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d1e4f507df2a4b01a291abbc137d5c7a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d1e4f507df2a4b01a291abbc137d5c7a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f783dce65c6b443db422c08ae123acd5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f783dce65c6b443db422c08ae123acd5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1111,7 +1117,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a1828de81be04fc18a71915d8c8c344a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a1828de81be04fc18a71915d8c8c344a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4289a0b52b864ae8bf6f0c9b5bbfc4d1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4289a0b52b864ae8bf6f0c9b5bbfc4d1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1151,7 +1157,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e789f15840f40f7a05a208948487b28-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e789f15840f40f7a05a208948487b28-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc1b666c4874462cb098d30be48cb471-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc1b666c4874462cb098d30be48cb471-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1173,7 +1179,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6c12fded3bd24847b82a15ace5a77174-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6c12fded3bd24847b82a15ace5a77174-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a1d73872b0004f17b2ec592dff5e2a3c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a1d73872b0004f17b2ec592dff5e2a3c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1196,7 +1202,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d39fe77df874bd7b41ab57c40b0ea03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d39fe77df874bd7b41ab57c40b0ea03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a19802abeff401dab894de3adefed8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a19802abeff401dab894de3adefed8c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1214,7 +1220,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0473d491a6a34980bc43ec9b67c22efc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0473d491a6a34980bc43ec9b67c22efc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/85f39b907fb9460780922c64fbc582d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/85f39b907fb9460780922c64fbc582d9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1232,7 +1238,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46e19c9ef0ca4c71a0c3552f2e86e073-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46e19c9ef0ca4c71a0c3552f2e86e073-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0327468d682b4896a90b9d2b8d868076-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0327468d682b4896a90b9d2b8d868076-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1250,7 +1256,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8ea9cff89e647c4a53a5491275b8ffb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8ea9cff89e647c4a53a5491275b8ffb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bea0b22448ee40f6b09196f442ee62e4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bea0b22448ee40f6b09196f442ee62e4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1268,7 +1274,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7cf019661462490eb64ebe41bf25cb99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7cf019661462490eb64ebe41bf25cb99-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/72504b8ec0d54bd79b62c9664f334116-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/72504b8ec0d54bd79b62c9664f334116-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1289,7 +1295,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e6e932c17ac54f3aa7e9d85258fe07cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e6e932c17ac54f3aa7e9d85258fe07cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a88ef26e2a4e4eb9866a18ed5b083d3e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a88ef26e2a4e4eb9866a18ed5b083d3e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1338,7 +1344,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5090dd0f3f50429d98c6703460a2be84-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5090dd0f3f50429d98c6703460a2be84-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3401333a83c240c4ab4392a482da4306-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3401333a83c240c4ab4392a482da4306-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1387,7 +1393,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4ed3fd03c9e84deb989b42a7fd7bf949-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4ed3fd03c9e84deb989b42a7fd7bf949-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d2e6bec429f4706a6f1186c783271a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d2e6bec429f4706a6f1186c783271a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1409,7 +1415,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82fa0c1f63ce48e9b6d3013cb1cc64cb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82fa0c1f63ce48e9b6d3013cb1cc64cb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d017d6f7e2a54d04a6390ced04215e7c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d017d6f7e2a54d04a6390ced04215e7c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1431,7 +1437,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/861bdc5a3b3b443fbcdb08035936845d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/861bdc5a3b3b443fbcdb08035936845d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b7e28ced93d4448b64cec28bef3d780-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b7e28ced93d4448b64cec28bef3d780-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1452,7 +1458,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e4f4712d17b142f49d21924bc4c408d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e4f4712d17b142f49d21924bc4c408d0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35e77a9b5305487889b158d2aa0577cc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35e77a9b5305487889b158d2aa0577cc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1474,7 +1480,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a915c5d1c3543648d925a509ffe1023-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a915c5d1c3543648d925a509ffe1023-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/77c0d182aebd412894caf6dfb0250c24-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/77c0d182aebd412894caf6dfb0250c24-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1496,7 +1502,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c655ae07de8d4f2397ac487000fc2952-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c655ae07de8d4f2397ac487000fc2952-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/86515215e8b5431b94373f3fdc19da41-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/86515215e8b5431b94373f3fdc19da41-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1514,7 +1520,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32a0d001d4a64f33be0a16f824c9a6bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32a0d001d4a64f33be0a16f824c9a6bd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/972bcd2f10ce4d75906d0349b7845726-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/972bcd2f10ce4d75906d0349b7845726-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1532,7 +1538,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc7a33549ee845da950a2fc56a2e8b04-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc7a33549ee845da950a2fc56a2e8b04-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b552ac0ed07246ea86c5079970355a50-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b552ac0ed07246ea86c5079970355a50-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1550,7 +1556,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7d5cb53cfa864478b32a02554d9a556a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7d5cb53cfa864478b32a02554d9a556a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46d072352f874d19b034ea5da3a2dfff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46d072352f874d19b034ea5da3a2dfff-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1568,7 +1574,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b4c7a88623f4476bd4ef21aaa0bdd4a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b4c7a88623f4476bd4ef21aaa0bdd4a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/69688793fcd247b89bbcbf90486c8459-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/69688793fcd247b89bbcbf90486c8459-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1586,7 +1592,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/989cd0f5d3d340ac97d6f2e4c17c9007-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/989cd0f5d3d340ac97d6f2e4c17c9007-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/37b36271315c484fa4576dbd4e33665a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/37b36271315c484fa4576dbd4e33665a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1625,7 +1631,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f7ed1c3f06ae45719b641394b47fdee0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f7ed1c3f06ae45719b641394b47fdee0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c364241ec5f44f10b83f39b42a3c5962-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c364241ec5f44f10b83f39b42a3c5962-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1672,7 +1678,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db4dbfb7f3ca4bc4bf49f276ef09a9ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db4dbfb7f3ca4bc4bf49f276ef09a9ee-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d94cb1ce5b85420c9a269637436165cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d94cb1ce5b85420c9a269637436165cd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1719,7 +1725,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6415585922454999b6d51ea0ee8988b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6415585922454999b6d51ea0ee8988b5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c78f1f775a5847c89b03f8188c24219c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c78f1f775a5847c89b03f8188c24219c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1767,7 +1773,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/84c9fcfdaedc471e97cfbb2e27cdb135-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/84c9fcfdaedc471e97cfbb2e27cdb135-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/71a56399380f4aad83a655df1623ae18-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/71a56399380f4aad83a655df1623ae18-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1814,7 +1820,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/105dc79203b64c749fa66019dd2d5bf4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/105dc79203b64c749fa66019dd2d5bf4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ec57b074a424727a9cc4e3130944ca9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ec57b074a424727a9cc4e3130944ca9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1862,7 +1868,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f62a3c0bbd144feaa1893744f40d54e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f62a3c0bbd144feaa1893744f40d54e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c7e2d6bf6be5499fbb5a2f8205f55986-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c7e2d6bf6be5499fbb5a2f8205f55986-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1880,7 +1886,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3155f58b70e4aaeb5f8fb9e83438629-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3155f58b70e4aaeb5f8fb9e83438629-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/90dd37b06cb3447db944ed0310b1426a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/90dd37b06cb3447db944ed0310b1426a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1898,7 +1904,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46dfb0d813954c84b56c9d805f456b44-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46dfb0d813954c84b56c9d805f456b44-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8bb0518da09488a8f6bdaa54cb4550c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8bb0518da09488a8f6bdaa54cb4550c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1916,7 +1922,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/118f487e8b9f4a07afbaacbb3eb11ba0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/118f487e8b9f4a07afbaacbb3eb11ba0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b84b3ff561f24b0380bbc0bb51f18a8b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b84b3ff561f24b0380bbc0bb51f18a8b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1934,7 +1940,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bba71fa892984fbd9b63a37035060ffc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bba71fa892984fbd9b63a37035060ffc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4eea78d1883441439b80b4d927ef601c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4eea78d1883441439b80b4d927ef601c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1952,7 +1958,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89c7e26a67d74990b45a9f2ee6c9cbe6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89c7e26a67d74990b45a9f2ee6c9cbe6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e6cdf081aae644ce95f2ab95207aa599-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e6cdf081aae644ce95f2ab95207aa599-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -1973,7 +1979,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f7940bacfaa4ae5a7d2973df614bdd8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f7940bacfaa4ae5a7d2973df614bdd8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3706163267842298c7b8683c7d1ba72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3706163267842298c7b8683c7d1ba72-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2021,7 +2027,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f2098e2f8f545c8a29f5b7ba3af73cc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f2098e2f8f545c8a29f5b7ba3af73cc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1325376bba948788cb22b5f7e5cf822-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1325376bba948788cb22b5f7e5cf822-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2069,7 +2075,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a3ff2adcb85e47a8ae5b3ddf3712fee1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a3ff2adcb85e47a8ae5b3ddf3712fee1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7a70e18e408f4ffeb161cc0df5833968-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7a70e18e408f4ffeb161cc0df5833968-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2117,7 +2123,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f1969bacbfb6478798797357a6265496-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f1969bacbfb6478798797357a6265496-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0b10d2fcf87d47bdaae8e4624b55a9ba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0b10d2fcf87d47bdaae8e4624b55a9ba-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2135,7 +2141,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5127376a20b849a18fb15cdee9ec6edb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5127376a20b849a18fb15cdee9ec6edb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/349f01542f0445d98c74303b4052626d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/349f01542f0445d98c74303b4052626d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2153,7 +2159,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/deab5427f1d241e48234ddc20d692e2d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/deab5427f1d241e48234ddc20d692e2d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4d3511175c2d4cb480a6b7b65f22853d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4d3511175c2d4cb480a6b7b65f22853d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2171,7 +2177,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4a8dbc55f7274754a4f3492dc1274449-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4a8dbc55f7274754a4f3492dc1274449-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a1df2d893be48a981c4d850321e8ac7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a1df2d893be48a981c4d850321e8ac7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2189,7 +2195,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eb9f9b97c1a54074be367ca7d99ec08a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eb9f9b97c1a54074be367ca7d99ec08a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e4834350bb6a4ac78a3033159aa0e729-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e4834350bb6a4ac78a3033159aa0e729-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2207,7 +2213,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b9679a4a53cb4fb49447eab445bedd51-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b9679a4a53cb4fb49447eab445bedd51-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a29cb0884e374e3e90b95b8457ca349d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a29cb0884e374e3e90b95b8457ca349d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2225,7 +2231,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0526cada9c6e4197920778c058effcc7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0526cada9c6e4197920778c058effcc7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b00379c3a9943c486ca88761729be8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b00379c3a9943c486ca88761729be8f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2246,7 +2252,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b50c049758e44eeaee13069a3a689b2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b50c049758e44eeaee13069a3a689b2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/94f49c29db344e669267ebdee236e025-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/94f49c29db344e669267ebdee236e025-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2294,7 +2300,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f085f270960440bb839d220a541c2a4a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f085f270960440bb839d220a541c2a4a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c0694b33db474cf298d2a8b713bf3c23-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c0694b33db474cf298d2a8b713bf3c23-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2312,7 +2318,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cc24a2f009804287a1dbe568f753bebe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cc24a2f009804287a1dbe568f753bebe-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b774d37ef544253a9eac73f352c7c50-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b774d37ef544253a9eac73f352c7c50-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2330,7 +2336,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00469d84888d4c1b9c59afde586d9fb4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00469d84888d4c1b9c59afde586d9fb4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8accd2609e04d61ba6abcae4f07cb9c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8accd2609e04d61ba6abcae4f07cb9c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2348,7 +2354,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7834306c30214dafaccbfebcf6be5c1d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7834306c30214dafaccbfebcf6be5c1d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f2fc1b002ca444ab49cc20f0615cf63-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f2fc1b002ca444ab49cc20f0615cf63-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2366,7 +2372,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4c2f919b18f94069a5c707a9af8557c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4c2f919b18f94069a5c707a9af8557c6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/070502ddd5d04e33a4c4fde1df545e4b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/070502ddd5d04e33a4c4fde1df545e4b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2384,7 +2390,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2324406cf4c9431e9675975d65503403-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2324406cf4c9431e9675975d65503403-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2d9675c869141f3a0f2f46a4f16ee91-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2d9675c869141f3a0f2f46a4f16ee91-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2402,7 +2408,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b07d5e718c5b4dbba7295a3d0d9199c8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b07d5e718c5b4dbba7295a3d0d9199c8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3166c9c0768c4c248d2e8423ca89322d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3166c9c0768c4c248d2e8423ca89322d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2420,7 +2426,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78f9178490514e4db709de6dcfe2c8f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78f9178490514e4db709de6dcfe2c8f9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/083fc07022994c72879d457dc12d1ccd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/083fc07022994c72879d457dc12d1ccd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2438,7 +2444,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c28dbc73281f41f3bd100666f7b336a0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c28dbc73281f41f3bd100666f7b336a0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d1373724c5c245f1a6d2b03592c4459e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d1373724c5c245f1a6d2b03592c4459e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2456,7 +2462,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/92b2a9bbb86549408708a2f4019bf58c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/92b2a9bbb86549408708a2f4019bf58c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f1a8185f710a492b8dedae2a4dc41f22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f1a8185f710a492b8dedae2a4dc41f22-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2474,7 +2480,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e9891f0026434043a4982e90ef7f8c12-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e9891f0026434043a4982e90ef7f8c12-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2ba3bbe316949108b9e774a9174b0a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2ba3bbe316949108b9e774a9174b0a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2492,7 +2498,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a9e70f7c324f4886ab11b9c155d8348e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a9e70f7c324f4886ab11b9c155d8348e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea86e6325cbc447b8dfa752263a5c903-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea86e6325cbc447b8dfa752263a5c903-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2510,7 +2516,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/627910d197854669989c43e705caaca1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/627910d197854669989c43e705caaca1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee06a8264e8a4e3cbcda71cd4cca4af8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee06a8264e8a4e3cbcda71cd4cca4af8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2528,7 +2534,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f85a9f82016b4e15882b7231c02acae8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f85a9f82016b4e15882b7231c02acae8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc2ba71f3925409d8078b542dbdbad17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc2ba71f3925409d8078b542dbdbad17-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2546,7 +2552,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea147ad4b39a40da9e731296bb94a5bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea147ad4b39a40da9e731296bb94a5bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b771b4e037c1442f9b0cd3d6e56c40c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b771b4e037c1442f9b0cd3d6e56c40c6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2564,7 +2570,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/489c36154e1e4d249450f4899dd1611b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/489c36154e1e4d249450f4899dd1611b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/812e7d97b0974a9cb0dacfc9a83f475c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/812e7d97b0974a9cb0dacfc9a83f475c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2582,7 +2588,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f5069691da354836a0c50b89d61edd19-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f5069691da354836a0c50b89d61edd19-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9cd0a771fbf94f63af953da0156f9a91-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9cd0a771fbf94f63af953da0156f9a91-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2600,7 +2606,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/feb78a1a70e3400d9e6e2ade73017e67-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/feb78a1a70e3400d9e6e2ade73017e67-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7dbe4651dc5445bda6544d786e8a9b24-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7dbe4651dc5445bda6544d786e8a9b24-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2618,7 +2624,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d7329af807d84c89af912d0cf442ead6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d7329af807d84c89af912d0cf442ead6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9e0d26d25c5453bbac8fe9b6e3e0567-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9e0d26d25c5453bbac8fe9b6e3e0567-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2636,7 +2642,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d50e8c341f9c46d1ac6bb85f1b41e7ba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d50e8c341f9c46d1ac6bb85f1b41e7ba-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd056e2506714729b2b1410efb95dc7f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd056e2506714729b2b1410efb95dc7f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2654,7 +2660,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/404bb5483744436aa74f240ed034a132-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/404bb5483744436aa74f240ed034a132-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b9cc871698c14008b0f9a47855730b65-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b9cc871698c14008b0f9a47855730b65-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2672,7 +2678,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/14cc1342c10447798188ca6927ecbea0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/14cc1342c10447798188ca6927ecbea0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f222df8c194e4ed68e26e85c0f243f7f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f222df8c194e4ed68e26e85c0f243f7f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2690,7 +2696,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/024364e78ed142f2a71013953e11371a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/024364e78ed142f2a71013953e11371a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/567bf140d9a9478e9329709e794183aa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/567bf140d9a9478e9329709e794183aa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2708,7 +2714,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8082d674f67d4a7eb0cf838be3e55a06-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8082d674f67d4a7eb0cf838be3e55a06-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6e7aa92744eb41bf9b9b27ac9699b9da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6e7aa92744eb41bf9b9b27ac9699b9da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2726,7 +2732,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87389887b4714e419dd1a8ead540eb17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87389887b4714e419dd1a8ead540eb17-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8a2550bbf00349b2b19c84d3e72f7e9e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8a2550bbf00349b2b19c84d3e72f7e9e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2744,7 +2750,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dd9b83b5171740bf9858fa61cda36ed5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dd9b83b5171740bf9858fa61cda36ed5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d5d61f68dd0488497f5c581c9733f3c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d5d61f68dd0488497f5c581c9733f3c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2764,7 +2770,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f541dc1b443149c4a58248e881075922-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f541dc1b443149c4a58248e881075922-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/728b14810262449d86ca97276437938b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/728b14810262449d86ca97276437938b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2785,7 +2791,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3660e01145d1409bb228d2967cec7552-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3660e01145d1409bb228d2967cec7552-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8a7999cf30e94c538a3b5f07767dd765-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8a7999cf30e94c538a3b5f07767dd765-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2806,7 +2812,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4988f31f10424450983465f8811c2654-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4988f31f10424450983465f8811c2654-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57a87369e2484f5d9847de3a48b02cf4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57a87369e2484f5d9847de3a48b02cf4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2827,7 +2833,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e5d798549f564bab84a6a0c992ca9569-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e5d798549f564bab84a6a0c992ca9569-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f290741ecbe3425b9ade731d7916ab80-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f290741ecbe3425b9ade731d7916ab80-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2848,7 +2854,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a3ff599762ac40c7a6beb69cdd4aad39-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a3ff599762ac40c7a6beb69cdd4aad39-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/37bfa91eef304c0db5f77e568229219c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/37bfa91eef304c0db5f77e568229219c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2869,7 +2875,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cdc3452bec3c435cbb4968413707fa6d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cdc3452bec3c435cbb4968413707fa6d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3c485bc9f70412fa718236bd4dfe8a6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3c485bc9f70412fa718236bd4dfe8a6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2890,7 +2896,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0f0c4eaff75c4f738188da8fa1c2e5c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0f0c4eaff75c4f738188da8fa1c2e5c9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/882de284e3944eb2b5abef8b0a5dbf9f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/882de284e3944eb2b5abef8b0a5dbf9f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2930,7 +2936,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1faedc91af174f9aaa07e56543843e2e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1faedc91af174f9aaa07e56543843e2e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9c372895e59441d5b6b4f4555426e9cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9c372895e59441d5b6b4f4555426e9cf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2969,7 +2975,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ee5ad20822e40b088dda35b3cbe8c98-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ee5ad20822e40b088dda35b3cbe8c98-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2e4cdbd09ab46d4bb402536f3115d55-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2e4cdbd09ab46d4bb402536f3115d55-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -2991,7 +2997,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/76388459f51e48619a59d5403293fe81-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/76388459f51e48619a59d5403293fe81-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6deb317b619e4990abdaa0b34d3bbc27-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6deb317b619e4990abdaa0b34d3bbc27-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3030,7 +3036,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5ada6cc115dc449ab43d53905192190a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5ada6cc115dc449ab43d53905192190a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b2417fee9c2440cfbfe3fd7d90586051-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b2417fee9c2440cfbfe3fd7d90586051-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3048,7 +3054,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2fccf8add9514e65bf74a21770569635-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2fccf8add9514e65bf74a21770569635-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e6503709496d425a8a98af6dbf75e0c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e6503709496d425a8a98af6dbf75e0c0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3066,7 +3072,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b726a956d69f4ecfb50e0e99b1d3aeb0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b726a956d69f4ecfb50e0e99b1d3aeb0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8dc52af2e9949dc8150bee62596b15c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8dc52af2e9949dc8150bee62596b15c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3084,7 +3090,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/69eb21a9fa8844b68df2ab5ac55a23dc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/69eb21a9fa8844b68df2ab5ac55a23dc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/28658464f3de432fb9163c73cea8e0f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/28658464f3de432fb9163c73cea8e0f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3102,7 +3108,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6eb42834b6cf4d9eb55fbc7c350e8fc9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6eb42834b6cf4d9eb55fbc7c350e8fc9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1e312bcde6ae492e8fe18c56f0984516-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1e312bcde6ae492e8fe18c56f0984516-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3123,7 +3129,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e7236589f5bc43cdb8996bfb6b27850c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e7236589f5bc43cdb8996bfb6b27850c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3900a678e10847ceabaa65ef18e51614-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3900a678e10847ceabaa65ef18e51614-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3172,7 +3178,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2160b90109f44ce7acbac0fd5fface6a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2160b90109f44ce7acbac0fd5fface6a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/897b7f3e5a3c4a548000b66d73bc0755-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/897b7f3e5a3c4a548000b66d73bc0755-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3221,7 +3227,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/346c7eb64e2240e2849a1a45f93246a0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/346c7eb64e2240e2849a1a45f93246a0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f30938b49aa74faba6bce507fd2956a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f30938b49aa74faba6bce507fd2956a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3270,7 +3276,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3671796a1f1c4359a0fc73802361cc37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3671796a1f1c4359a0fc73802361cc37-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ea786e84bd64264afa351c96c6b73de-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ea786e84bd64264afa351c96c6b73de-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3292,7 +3298,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/30b3c3cb72074c6db5312571c1883a99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/30b3c3cb72074c6db5312571c1883a99-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/095fbb2fa9d04a7bb61e5d29e71c1690-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/095fbb2fa9d04a7bb61e5d29e71c1690-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3314,7 +3320,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a54bbbc949f340e8a0a2d7a7c8a269f4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a54bbbc949f340e8a0a2d7a7c8a269f4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/34317652692f4f5399d637fe10cbd462-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/34317652692f4f5399d637fe10cbd462-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3336,7 +3342,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/399dc474826d419e85ce13b8e848332f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/399dc474826d419e85ce13b8e848332f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fba91d038ca44e5da894ae6a29b76787-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fba91d038ca44e5da894ae6a29b76787-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3358,7 +3364,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ec3d0898c3e84b18bea3402dec89427d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ec3d0898c3e84b18bea3402dec89427d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33db401805dd4182a6e295d23af9bdc5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33db401805dd4182a6e295d23af9bdc5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3380,7 +3386,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/83e47a2bf1cb4fa6a05908eeaf14c0b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/83e47a2bf1cb4fa6a05908eeaf14c0b5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d9d06cbe590346da9ce6ca88964adeb2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d9d06cbe590346da9ce6ca88964adeb2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3398,7 +3404,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e29ce9c52a394aa38f21fa3e645b0967-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e29ce9c52a394aa38f21fa3e645b0967-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee702985cbd94e60a7d3fd086f5deebd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee702985cbd94e60a7d3fd086f5deebd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3416,7 +3422,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78a98fa55be8498eb4310d13a420810e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78a98fa55be8498eb4310d13a420810e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e39e73ed7134ef19412b5d63bc4cfc7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e39e73ed7134ef19412b5d63bc4cfc7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3434,7 +3440,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d1938c4774b34ce4b597cce33a87f9a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d1938c4774b34ce4b597cce33a87f9a1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3879eb4052534247ad93605bf96ce3a0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3879eb4052534247ad93605bf96ce3a0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3452,7 +3458,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87400cb37f87445bac0d82080bbeb1b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87400cb37f87445bac0d82080bbeb1b8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e884fc4cdab84b599978607f3b18ed89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e884fc4cdab84b599978607f3b18ed89-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3470,7 +3476,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/466f686227094a8f8c5fbe76582ec86d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/466f686227094a8f8c5fbe76582ec86d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/816382328cd04421bfd06b5e086943f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/816382328cd04421bfd06b5e086943f2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3491,7 +3497,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9f394d5cc494a24b7126cc4e6617b92-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9f394d5cc494a24b7126cc4e6617b92-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64243a87711f490e8df748725e0225a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64243a87711f490e8df748725e0225a1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3530,7 +3536,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8fd56e2d759c4d66a739c45d92c3c348-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8fd56e2d759c4d66a739c45d92c3c348-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3848261c83184ffabb9ab37cef28b67f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3848261c83184ffabb9ab37cef28b67f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3579,7 +3585,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/88435f93db574782a6b5a8656cc102da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/88435f93db574782a6b5a8656cc102da-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7af5172dbd804ec2a98be952f2bfc7b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7af5172dbd804ec2a98be952f2bfc7b8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3601,7 +3607,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/27b6eede559e4380bcc2be8c95704ea3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/27b6eede559e4380bcc2be8c95704ea3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7dfc6c4f8c9d4f22a5a6d30a00a987e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7dfc6c4f8c9d4f22a5a6d30a00a987e9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3622,7 +3628,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ac8a7f67b6c49f2b589d82af8bb7577-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ac8a7f67b6c49f2b589d82af8bb7577-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ddd2afc9b0d84657872826dd44f5f611-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ddd2afc9b0d84657872826dd44f5f611-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3643,7 +3649,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/601c1925fb004578973380d50a18bdcb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/601c1925fb004578973380d50a18bdcb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02bfa0c949f948a299ba42ae1a1c2de0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02bfa0c949f948a299ba42ae1a1c2de0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3665,7 +3671,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e4170f1248f74660904d0479ca112567-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e4170f1248f74660904d0479ca112567-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d6af64ab702b44a3b572909feb57f9da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d6af64ab702b44a3b572909feb57f9da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3683,7 +3689,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a393205687d24cab8879f4c298c86c8b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a393205687d24cab8879f4c298c86c8b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/41af4d0b3d25430b8361ff462ed747b7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/41af4d0b3d25430b8361ff462ed747b7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3701,7 +3707,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/020470a0e87c4803aebdef148d26cff7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/020470a0e87c4803aebdef148d26cff7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78f9dfd8e0064d5ebfbf6c1eba66503f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78f9dfd8e0064d5ebfbf6c1eba66503f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3719,7 +3725,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/36125dee5b574796b33c98df1633f66d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/36125dee5b574796b33c98df1633f66d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8de0c9a130a442f2b60f454544b6663c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8de0c9a130a442f2b60f454544b6663c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3737,7 +3743,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e6da4d8fd964648a6d56a486420927b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e6da4d8fd964648a6d56a486420927b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/da053e4638954f0c951d96ae5043899a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/da053e4638954f0c951d96ae5043899a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3755,7 +3761,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/983669684cf947829d52033e343308bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/983669684cf947829d52033e343308bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/08b84d4facc64d2aa3a33da560d0797f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/08b84d4facc64d2aa3a33da560d0797f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3776,7 +3782,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b472be77b8264510af2a12e306b67379-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b472be77b8264510af2a12e306b67379-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ad272167487a42e5b40aeb6d215b5285-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ad272167487a42e5b40aeb6d215b5285-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3823,7 +3829,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2499bb39c6ff4dd59d6dc740e37894bf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2499bb39c6ff4dd59d6dc740e37894bf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef38b89e7a5243c38c5d291c9989090e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef38b89e7a5243c38c5d291c9989090e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3871,7 +3877,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e899f566a2ca4b84b79c289d33dc4a72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e899f566a2ca4b84b79c289d33dc4a72-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73ce8f4c6f2848bcbb207884e7958182-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73ce8f4c6f2848bcbb207884e7958182-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3919,7 +3925,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aae304fdd3334022aa29fe3c0b302328-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aae304fdd3334022aa29fe3c0b302328-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/990e25dd9f8d4cfc9d9922f1ef2c680c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/990e25dd9f8d4cfc9d9922f1ef2c680c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3967,7 +3973,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/28dcc04cfb4242e8b13bcd52cb36d045-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/28dcc04cfb4242e8b13bcd52cb36d045-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c2d50ac442704a17864a7bc04f16a7f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c2d50ac442704a17864a7bc04f16a7f2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -3985,7 +3991,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68a2ccbec0bf476d96d480b9b5d67bae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68a2ccbec0bf476d96d480b9b5d67bae-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f342d22f1c24952bc16a8b3755ac572-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f342d22f1c24952bc16a8b3755ac572-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4003,7 +4009,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e929b658969943b0be3d2370fe7e9b5a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e929b658969943b0be3d2370fe7e9b5a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e04f590d13f94e4f8c36aff6d941bdff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e04f590d13f94e4f8c36aff6d941bdff-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4021,7 +4027,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2a7fa1f5803d4848bf7c9fd51a7a085a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2a7fa1f5803d4848bf7c9fd51a7a085a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/567d0d12cbbf4880bf8e7a48443dc31e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/567d0d12cbbf4880bf8e7a48443dc31e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4039,7 +4045,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53eecf1bf52c4abeb137e5feb4e0e575-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53eecf1bf52c4abeb137e5feb4e0e575-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/584818ed56e64808ba292d33f6bd33c4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/584818ed56e64808ba292d33f6bd33c4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4057,7 +4063,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a4ac8cae71f442e89942f32836399498-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a4ac8cae71f442e89942f32836399498-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/baf5bdcd7c0d43bfa7d4b1495371a36d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/baf5bdcd7c0d43bfa7d4b1495371a36d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4075,7 +4081,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a1687abf606647d8a8a1a6d70c24e271-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a1687abf606647d8a8a1a6d70c24e271-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c2043ad1fa8e453f82d8331993b61a76-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c2043ad1fa8e453f82d8331993b61a76-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4096,7 +4102,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b962f62c64804ddb9abd361e813b8496-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b962f62c64804ddb9abd361e813b8496-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7b189cf1636e4f07aafa1b3f36f09900-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7b189cf1636e4f07aafa1b3f36f09900-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4144,7 +4150,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc4c912f4b174e1493ca9dd2aff01f97-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc4c912f4b174e1493ca9dd2aff01f97-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9231b6773214cb3b8a3d873c6bfa9f7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9231b6773214cb3b8a3d873c6bfa9f7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4192,7 +4198,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/10d8c8eed15e4753a081a0fe7f0ca34e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/10d8c8eed15e4753a081a0fe7f0ca34e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/681045f79bb64f3c8440b7e1e1874771-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/681045f79bb64f3c8440b7e1e1874771-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4210,7 +4216,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ba74c4c6dd24ca4b8b06f2bd77d7c17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ba74c4c6dd24ca4b8b06f2bd77d7c17-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e78d775bf40e4ccd9f574aba36d8adc6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e78d775bf40e4ccd9f574aba36d8adc6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4228,7 +4234,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7c2c8579f1144521b6099846432ed6d1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7c2c8579f1144521b6099846432ed6d1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/900e30ac1d7141bb9facd630b505b52b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/900e30ac1d7141bb9facd630b505b52b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4246,7 +4252,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9a7bfbaa8a545bea506d8e6fef330ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9a7bfbaa8a545bea506d8e6fef330ca-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e48949d6e9744c1f9dd4254ca2e9538a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e48949d6e9744c1f9dd4254ca2e9538a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4264,7 +4270,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4b5385ba9e2e48bcb3fecb1e7b8622be-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4b5385ba9e2e48bcb3fecb1e7b8622be-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/98a4e333eabf421ba4a14c1c69a5035d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/98a4e333eabf421ba4a14c1c69a5035d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4282,7 +4288,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bcd577f8d28d499594813226fe04b854-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bcd577f8d28d499594813226fe04b854-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3ab5af2c782143eea4c417c60c2ff092-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3ab5af2c782143eea4c417c60c2ff092-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4300,7 +4306,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4e42bb81b372429cb133f9e549668457-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4e42bb81b372429cb133f9e549668457-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b7ce0cf4b55b430891f4e9f38c40924f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b7ce0cf4b55b430891f4e9f38c40924f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4321,7 +4327,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c3ddd496fb0e4300a9a99f55d094bac3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c3ddd496fb0e4300a9a99f55d094bac3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0b00936a1ce14380bb27909a4598d38f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0b00936a1ce14380bb27909a4598d38f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4339,7 +4345,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aa0864bb3ce1492b98bc8a7dbed8ba31-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aa0864bb3ce1492b98bc8a7dbed8ba31-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/446f0c1d52ce4e9fb415b56fa876e486-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/446f0c1d52ce4e9fb415b56fa876e486-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4357,7 +4363,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02105a0804554e0a9210ccb0a31014f8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02105a0804554e0a9210ccb0a31014f8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e9378c1481f24291a152f222ebbbcadc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e9378c1481f24291a152f222ebbbcadc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4375,7 +4381,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b328d2905c79464289003fc68d489a22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b328d2905c79464289003fc68d489a22-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b88b5760d2ed4c19af7be862f2dfff78-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b88b5760d2ed4c19af7be862f2dfff78-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4393,7 +4399,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3fbbf91e6fcb41fea96c8c14c3c68f7f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3fbbf91e6fcb41fea96c8c14c3c68f7f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5bfc51bbe5654f8fa2546398add558f4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5bfc51bbe5654f8fa2546398add558f4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4411,7 +4417,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/72293fcb074e48b2a3f1377dc27f39f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/72293fcb074e48b2a3f1377dc27f39f5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02a86e2a819a4559b4e7c2e85d0748e0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02a86e2a819a4559b4e7c2e85d0748e0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4429,7 +4435,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f2b3aa53c167495cb0d04487d7346b0e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f2b3aa53c167495cb0d04487d7346b0e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/785353811d404776928dac0b238f87b7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/785353811d404776928dac0b238f87b7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4447,7 +4453,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87e75acc42e043ebb1d171a75124d105-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87e75acc42e043ebb1d171a75124d105-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4bf216da759b4aa8908a5378eb304f98-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4bf216da759b4aa8908a5378eb304f98-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4465,7 +4471,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/edc0f1c7e81b434c8e98cd188e2ebbdd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/edc0f1c7e81b434c8e98cd188e2ebbdd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/454ea23fc1b54097a2188e4b066d780d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/454ea23fc1b54097a2188e4b066d780d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4483,7 +4489,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5c7c664c970a45f9b756f8196e128332-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5c7c664c970a45f9b756f8196e128332-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/77f12c729fea4512897eba94364d49e6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/77f12c729fea4512897eba94364d49e6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4501,7 +4507,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/744b27892acb4c64b1a5139e47f8adf1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/744b27892acb4c64b1a5139e47f8adf1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef18e67e39a84526b475eac33c1bec8b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef18e67e39a84526b475eac33c1bec8b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4519,7 +4525,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/98166e4a39534d9d9bbd3a15368e3014-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/98166e4a39534d9d9bbd3a15368e3014-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c9c808591d64417386f86e7f005454d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c9c808591d64417386f86e7f005454d6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4537,7 +4543,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ccf696d6e6647edbc3c32c14bd81435-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ccf696d6e6647edbc3c32c14bd81435-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cc2fea99557b43d088dca25ac362a949-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cc2fea99557b43d088dca25ac362a949-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4555,7 +4561,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a982e4992cf940ebb710fef4aca1dfd7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a982e4992cf940ebb710fef4aca1dfd7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f935c36bdc704b4cac45d74df1329790-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f935c36bdc704b4cac45d74df1329790-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4573,7 +4579,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/88eb10d70e394aaa8669b034e7767e08-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/88eb10d70e394aaa8669b034e7767e08-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e3861c4894694497b31952adb131cd6b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e3861c4894694497b31952adb131cd6b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4591,7 +4597,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d388b49b0f74651be69785fb7246045-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d388b49b0f74651be69785fb7246045-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c06b515377e3424fba6664ec28985f26-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c06b515377e3424fba6664ec28985f26-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4609,7 +4615,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/678cd19ebea74e90a501d04aefd80492-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/678cd19ebea74e90a501d04aefd80492-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bbc745b94c074edc88f98bb5f0c27e3c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bbc745b94c074edc88f98bb5f0c27e3c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4627,7 +4633,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b391e8ffb814ccdaf1b31373e009ade-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b391e8ffb814ccdaf1b31373e009ade-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/49bdd2304ca74a7da664e80e63661cfe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/49bdd2304ca74a7da664e80e63661cfe-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4645,7 +4651,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/62200d55b56e42778e5beed686d91dc0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/62200d55b56e42778e5beed686d91dc0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/80b6035bd6d44cee90f8d1d5e1ff8882-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/80b6035bd6d44cee90f8d1d5e1ff8882-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4663,7 +4669,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/313ed2a1568b40a3a804856601156d32-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/313ed2a1568b40a3a804856601156d32-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d71c88203e8c42ce8e95c4d010690efc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d71c88203e8c42ce8e95c4d010690efc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4681,7 +4687,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9668de0e235a410e8f01ed6349cf491c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9668de0e235a410e8f01ed6349cf491c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f5763707868840f0b4b3f89ac2d08250-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f5763707868840f0b4b3f89ac2d08250-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4699,7 +4705,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3040789686a74da18bd31ee976a12e2a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3040789686a74da18bd31ee976a12e2a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/551235ed299d41cca4c92d18ffe86d5d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/551235ed299d41cca4c92d18ffe86d5d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4717,7 +4723,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/82fab14bf9c24e5e99148dbf96ed2296-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/82fab14bf9c24e5e99148dbf96ed2296-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/164650d61f37494fa5f11e4794e39f10-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/164650d61f37494fa5f11e4794e39f10-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4735,7 +4741,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/162a91e3bc1d4eb2ac0e1dd08b0a8ac1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/162a91e3bc1d4eb2ac0e1dd08b0a8ac1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/26356302c24c434c88089922579522aa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/26356302c24c434c88089922579522aa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4756,7 +4762,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c9a28527e70743af8a9d26b7ced8700d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c9a28527e70743af8a9d26b7ced8700d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ffcfe845e81848f29f27fa075daed92b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ffcfe845e81848f29f27fa075daed92b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4805,7 +4811,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f1ac952c110246e7a8d13ab3dc4d8521-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f1ac952c110246e7a8d13ab3dc4d8521-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57969e42fe1a467cb60cbb94283f44d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57969e42fe1a467cb60cbb94283f44d6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4844,7 +4850,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7217c4328bdd4e86a7f8f7398e7781ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7217c4328bdd4e86a7f8f7398e7781ca-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/340d054b1dcf44dc8336f5fa1aff07c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/340d054b1dcf44dc8336f5fa1aff07c0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4866,7 +4872,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/90842800e62f403eb5352f55a9af1d2f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/90842800e62f403eb5352f55a9af1d2f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f92c83db0cc4aab8be47ca1786959c8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f92c83db0cc4aab8be47ca1786959c8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4888,7 +4894,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1c2f38606a7a4e03aabc4c06d9bf6ea9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1c2f38606a7a4e03aabc4c06d9bf6ea9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b392a262da8d42e7955675afd46100f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b392a262da8d42e7955675afd46100f3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4910,7 +4916,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2cc1a769c5a94c5593d7f16615012c01-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2cc1a769c5a94c5593d7f16615012c01-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8574c7326234aa8b8a04fcbfbeecec7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8574c7326234aa8b8a04fcbfbeecec7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4932,7 +4938,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bb514a1903104aaeb88654e89090ec67-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bb514a1903104aaeb88654e89090ec67-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3df5958f54348f3a012cb1da98a9662-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3df5958f54348f3a012cb1da98a9662-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4954,7 +4960,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0aeabeb55c9c4df5975e87e12ec37165-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0aeabeb55c9c4df5975e87e12ec37165-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc293c6662ab4be28401efc5ad8ed301-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc293c6662ab4be28401efc5ad8ed301-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4972,7 +4978,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/94e05492e94b4f1f9fc1fc78269a0355-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/94e05492e94b4f1f9fc1fc78269a0355-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/05fb27442ae2464b9040c1c430581d96-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/05fb27442ae2464b9040c1c430581d96-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -4990,7 +4996,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/375ddd6eeac045c880eb892e90111a02-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/375ddd6eeac045c880eb892e90111a02-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7f84d262c4ce482ab6bae18283cb90dc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7f84d262c4ce482ab6bae18283cb90dc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5008,7 +5014,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a50327c557004045bfb64d3922056869-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a50327c557004045bfb64d3922056869-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/964658393f9c4910aeb16917a4803f35-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/964658393f9c4910aeb16917a4803f35-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5026,7 +5032,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0f2bf7c535ea40869b2f1a5af31870cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0f2bf7c535ea40869b2f1a5af31870cd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b7cf7ec165c64055a14e3c3d7c3d60e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b7cf7ec165c64055a14e3c3d7c3d60e7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5044,7 +5050,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2a9228555cf143f3b43edeb020d95389-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2a9228555cf143f3b43edeb020d95389-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bfbdf6e1840442f6b18701bd496e0983-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bfbdf6e1840442f6b18701bd496e0983-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5093,7 +5099,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef6a26918aa0437f8fa3980a03ff8ed5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef6a26918aa0437f8fa3980a03ff8ed5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4a3a72d040854c9f9c42f2896ae5f477-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4a3a72d040854c9f9c42f2896ae5f477-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5115,7 +5121,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c8412c397d6e4cb2ac2b2dd4881bb643-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c8412c397d6e4cb2ac2b2dd4881bb643-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/731e8ff2a3b74e479afb6c4af4011ee6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/731e8ff2a3b74e479afb6c4af4011ee6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5137,7 +5143,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/66174556a7184e15970b13340348bc11-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/66174556a7184e15970b13340348bc11-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3481fa25c6cc4c1e95e69fdca338e180-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3481fa25c6cc4c1e95e69fdca338e180-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5159,7 +5165,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6904f5b52569463aafa0c19fb549b46c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6904f5b52569463aafa0c19fb549b46c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/020f26f3b101451c91f338b795d43144-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/020f26f3b101451c91f338b795d43144-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5181,7 +5187,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f0a3db9e30454d0a8d999fd183d43ecc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f0a3db9e30454d0a8d999fd183d43ecc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c01f63c3c064bcebf6b38445e1711f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c01f63c3c064bcebf6b38445e1711f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5203,7 +5209,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/340c0fb03a4948edae9a81916cfaab6e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/340c0fb03a4948edae9a81916cfaab6e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/139f6919537c45c4a6a15acd96dee39e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/139f6919537c45c4a6a15acd96dee39e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5221,7 +5227,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fbb0f9aac161467781195579f0e8c6a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fbb0f9aac161467781195579f0e8c6a1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6443142c208b493e9b44c64b560bc07f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6443142c208b493e9b44c64b560bc07f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5239,7 +5245,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/105e525ec64f4df58cbeeb6585776f4f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/105e525ec64f4df58cbeeb6585776f4f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/31d0482351da4ade9fdf5a00c913d428-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/31d0482351da4ade9fdf5a00c913d428-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5257,7 +5263,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a639f4c42c5b4bf0840feb095ae926d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a639f4c42c5b4bf0840feb095ae926d3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6115042d15b248c9a91ab9f2148c042e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6115042d15b248c9a91ab9f2148c042e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5275,7 +5281,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6208064ec448415baedf562e9381be24-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6208064ec448415baedf562e9381be24-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a07c10ce139d4b2191f3205a6430bd0c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a07c10ce139d4b2191f3205a6430bd0c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5293,7 +5299,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/083c95d236b44645914f3aec883fd671-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/083c95d236b44645914f3aec883fd671-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/863648b005484f60953378f9e9dce9dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/863648b005484f60953378f9e9dce9dd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5311,7 +5317,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d385ce961eb14558b6322b969b5f8a48-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d385ce961eb14558b6322b969b5f8a48-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46900b4d134e4ff082f080ebec8808ab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46900b4d134e4ff082f080ebec8808ab-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5333,7 +5339,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75a71eb9669b423282a68a4c31aef423-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75a71eb9669b423282a68a4c31aef423-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f051009bd62e485f8cbceacc205595ae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f051009bd62e485f8cbceacc205595ae-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5355,7 +5361,7 @@
"Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
"Cbc0014I Cut generator 6 (TwoMirCuts) - 3 row cuts average 3.3 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
- "Cbc0001I Search completed - best objective 1e+50, took 0 iterations and 0 nodes (0.00 seconds)\n",
+ "Cbc0001I Search completed - best objective 1e+50, took 0 iterations and 0 nodes (0.01 seconds)\n",
"Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
"Cuts at root node changed objective from 0 to 0\n",
"Probing was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
@@ -5372,17 +5378,17 @@
"No feasible solution found\n",
"Enumerated nodes: 0\n",
"Total iterations: 0\n",
- "Time (CPU seconds): 0.00\n",
- "Time (Wallclock seconds): 0.00\n",
+ "Time (CPU seconds): 0.01\n",
+ "Time (Wallclock seconds): 0.01\n",
"\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4e76349f007a4d4ab6984a449117e78e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4e76349f007a4d4ab6984a449117e78e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9db8c72aab2e482aaf7f50489e939c68-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9db8c72aab2e482aaf7f50489e939c68-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5431,7 +5437,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/888f82801b9141de90f195da0e7caaf5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/888f82801b9141de90f195da0e7caaf5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/93d57fc02ecd4da0843c70ecfdfdc85e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/93d57fc02ecd4da0843c70ecfdfdc85e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5480,7 +5486,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bba062984ae04ed691d1c7c7d9e9b9ae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bba062984ae04ed691d1c7c7d9e9b9ae-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a610f0358f01494cb4dc8f45b3880eb7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a610f0358f01494cb4dc8f45b3880eb7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5498,7 +5504,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6bceb4a71a5749d2ae6b08849d2125b4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6bceb4a71a5749d2ae6b08849d2125b4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7c45b6b7f86940eea263b146b7aa5f13-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7c45b6b7f86940eea263b146b7aa5f13-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5516,7 +5522,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4d9fa9f5f96d475e987e0612cbda335e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4d9fa9f5f96d475e987e0612cbda335e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3479b17ef2ef4f4c8542354b871717b8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3479b17ef2ef4f4c8542354b871717b8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5534,7 +5540,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dcd53196663d4754bb8f0eb56178289e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dcd53196663d4754bb8f0eb56178289e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ca70d63013514a0dbc99fc7f4e87ed17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ca70d63013514a0dbc99fc7f4e87ed17-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5552,7 +5558,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/543f74967e6d46aaa48de850279e34d7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/543f74967e6d46aaa48de850279e34d7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/af16eb2ebd5e4ded86f4a1eb09c0d780-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/af16eb2ebd5e4ded86f4a1eb09c0d780-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5570,7 +5576,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9bcea30ef63344c5a151102060fcbdbf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9bcea30ef63344c5a151102060fcbdbf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3d83cb9eaecc435ca137f2db295d02d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3d83cb9eaecc435ca137f2db295d02d3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5582,13 +5588,13 @@
"Option for timeMode changed from cpu to elapsed\n",
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9da0b84f4b8b4d3998b79fd9b42e49e3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9da0b84f4b8b4d3998b79fd9b42e49e3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/655cd0cd6fcc43848628061596902836-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/655cd0cd6fcc43848628061596902836-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5610,7 +5616,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5bf0d5e6668490195bd99a6f2d1682b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5bf0d5e6668490195bd99a6f2d1682b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dd593acb6540442397db9e9f508e2579-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dd593acb6540442397db9e9f508e2579-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5659,7 +5665,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b4f8203a4bd74045acef2e45a0a49017-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b4f8203a4bd74045acef2e45a0a49017-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dd73a2ec0f154e6f9453751ccec1a62f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dd73a2ec0f154e6f9453751ccec1a62f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5677,7 +5683,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/47fbd79dac1e48348472f94a15070c78-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/47fbd79dac1e48348472f94a15070c78-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/727324152aaa499ab84d24ce8d5a40be-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/727324152aaa499ab84d24ce8d5a40be-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5695,7 +5701,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f61cecd976eb405697c85e60f19c1235-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f61cecd976eb405697c85e60f19c1235-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8d4d13e9980341e89c670a7ab95d0cae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8d4d13e9980341e89c670a7ab95d0cae-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5713,7 +5719,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2268474eef7541099e17dad02a28ad47-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2268474eef7541099e17dad02a28ad47-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d463b924fc5343de96b1a847ab5297ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d463b924fc5343de96b1a847ab5297ca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5731,7 +5737,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3b8ecc29dae6465bb5f532d91fa674e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3b8ecc29dae6465bb5f532d91fa674e7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/54c84406bcbb46fc9595a6e5a9e7193e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/54c84406bcbb46fc9595a6e5a9e7193e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5749,7 +5755,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a2fc08714f3b4c92890a4e0ed592e3ce-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a2fc08714f3b4c92890a4e0ed592e3ce-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bfa7f58cda554e1cab2dff276868cf13-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bfa7f58cda554e1cab2dff276868cf13-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5767,7 +5773,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/623b218e7b7c445cb6527bddd740e12d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/623b218e7b7c445cb6527bddd740e12d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/60c404cbc485490998da4f396b006e9e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/60c404cbc485490998da4f396b006e9e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5785,7 +5791,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/822566839f5d491ab386e5d3cd94c0e6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/822566839f5d491ab386e5d3cd94c0e6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c5db7e7a262941608d1e69b0a81e5dac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c5db7e7a262941608d1e69b0a81e5dac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5803,7 +5809,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6764cb6256654058ae7224774e3bd709-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6764cb6256654058ae7224774e3bd709-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c7ebbf5cc174a26bd7698486cb1e50e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c7ebbf5cc174a26bd7698486cb1e50e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5821,7 +5827,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/efa9e190c22f43b7afa5b26b549c0727-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/efa9e190c22f43b7afa5b26b549c0727-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ed07f984cd6c4a528287a2f539613180-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ed07f984cd6c4a528287a2f539613180-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5839,7 +5845,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2bb92fb236f94bb3b0613422428bf3b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2bb92fb236f94bb3b0613422428bf3b5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b524a72db4a445debc2d4336c26b493a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b524a72db4a445debc2d4336c26b493a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5857,7 +5863,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3fa5664ce1ae4bd1804cf57ff08e5abc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3fa5664ce1ae4bd1804cf57ff08e5abc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/44e6bb744cf547348cd79c9f7df1f7bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/44e6bb744cf547348cd79c9f7df1f7bd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5875,7 +5881,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3c1237e86b4472e92c54c862495f265-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3c1237e86b4472e92c54c862495f265-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2fa49be7f7c945b8b44e7c6edd18bfb0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2fa49be7f7c945b8b44e7c6edd18bfb0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5893,7 +5899,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cccaeac856246448ed1b72ce6838061-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cccaeac856246448ed1b72ce6838061-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/120cf124bac744448d8ec7f591eb2573-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/120cf124bac744448d8ec7f591eb2573-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5911,7 +5917,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c040992370914b3294e6240159b2c460-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c040992370914b3294e6240159b2c460-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/218d5edab2964511af144c746da1bf6e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/218d5edab2964511af144c746da1bf6e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5929,7 +5935,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d2a97facd8c4351aee08a1435540897-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d2a97facd8c4351aee08a1435540897-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3b87857bacb463bad8ad04cf113e528-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3b87857bacb463bad8ad04cf113e528-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5947,7 +5953,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dbd63c5a782e4b6b9098fba44cfae2d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dbd63c5a782e4b6b9098fba44cfae2d2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0f757ce08e094de0ad7310b329932b45-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0f757ce08e094de0ad7310b329932b45-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5965,7 +5971,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57403e48857d4c9da5c3bfe2f31eca22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57403e48857d4c9da5c3bfe2f31eca22-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/da28b82a903f4eafa946910a6e59ca22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/da28b82a903f4eafa946910a6e59ca22-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -5983,7 +5989,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9652d2c0b43d4ad5a4b22e4fdd275776-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9652d2c0b43d4ad5a4b22e4fdd275776-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3430a368ffc14e2c83363d1ea509cb25-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3430a368ffc14e2c83363d1ea509cb25-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6001,7 +6007,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9dd3a224b5ce4af7a7c396ab0f4aa319-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9dd3a224b5ce4af7a7c396ab0f4aa319-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8db9836de4644293a35205eae6749412-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8db9836de4644293a35205eae6749412-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6019,7 +6025,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/874e32efe9704f779d14c1a371da40c7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/874e32efe9704f779d14c1a371da40c7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/383138cd8f67416f959618e32d6cba86-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/383138cd8f67416f959618e32d6cba86-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6037,7 +6043,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e1a17d46bf6e45d8a09567159079da76-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e1a17d46bf6e45d8a09567159079da76-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73fb808863d54427b854eebb38d73356-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73fb808863d54427b854eebb38d73356-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6055,7 +6061,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c680e208e8be479aa2da2642822b8621-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c680e208e8be479aa2da2642822b8621-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e44bebcf45664abb97b42dad55f388e2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e44bebcf45664abb97b42dad55f388e2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6073,7 +6079,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3519434bb6dc45fbbaca6ff3eadfa232-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3519434bb6dc45fbbaca6ff3eadfa232-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5c63a993a3014b2d8b888c9040445921-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5c63a993a3014b2d8b888c9040445921-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6091,7 +6097,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/856c8d0d2a484e7b8b5f82306613968f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/856c8d0d2a484e7b8b5f82306613968f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4125b3d53cfb4d5a802d972c8e1c9a6b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4125b3d53cfb4d5a802d972c8e1c9a6b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6109,7 +6115,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/747d515300bf44f49d16df8c13d8e3ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/747d515300bf44f49d16df8c13d8e3ef-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/83eb3eb484eb4798bfebacd0e0171626-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/83eb3eb484eb4798bfebacd0e0171626-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6121,13 +6127,13 @@
"Option for timeMode changed from cpu to elapsed\n",
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1c76145ca2ed4afd8a89dbf7c01810b6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1c76145ca2ed4afd8a89dbf7c01810b6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0255a3dd062b4294a84c42a80439c638-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0255a3dd062b4294a84c42a80439c638-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6139,13 +6145,13 @@
"Option for timeMode changed from cpu to elapsed\n",
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3817b181dd94aef8f3612fed1d45014-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3817b181dd94aef8f3612fed1d45014-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9ea2f7af7a09432e8c40960baee19e5f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9ea2f7af7a09432e8c40960baee19e5f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6163,7 +6169,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c72e6f9f3b134bcabe8810fd2f9897bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c72e6f9f3b134bcabe8810fd2f9897bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/839fb5cf17124602b05276037b14e012-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/839fb5cf17124602b05276037b14e012-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6184,7 +6190,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b5dc08acc2549fd89ccb2859d3b6305-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b5dc08acc2549fd89ccb2859d3b6305-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a6bbd8f52464959ad0b58968d2cf9fd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a6bbd8f52464959ad0b58968d2cf9fd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6206,7 +6212,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2ce8418dc712498ca95467ec9749a8d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2ce8418dc712498ca95467ec9749a8d6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fa1ca97b785b472ab348e1269eafb6e1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fa1ca97b785b472ab348e1269eafb6e1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6255,7 +6261,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/857829e2af5745e98cd8d4ed4b246441-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/857829e2af5745e98cd8d4ed4b246441-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5bdbff3d459547b8b3afc3f8079e67da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5bdbff3d459547b8b3afc3f8079e67da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6277,7 +6283,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/20d3a1bee52d427a8d624505a68e9eea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/20d3a1bee52d427a8d624505a68e9eea-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/248a9bcd11924fdc9db24ca739c061c4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/248a9bcd11924fdc9db24ca739c061c4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6298,7 +6304,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/516fae332f444197ade9c4b360bb4747-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/516fae332f444197ade9c4b360bb4747-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5d5072d1f7a24e238d3d8c5fa35da167-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5d5072d1f7a24e238d3d8c5fa35da167-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6316,7 +6322,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/84b1363e40024fe788bc8d48d1001073-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/84b1363e40024fe788bc8d48d1001073-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9e25a7ae0ea8428eb0b1994e276451c7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9e25a7ae0ea8428eb0b1994e276451c7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6334,7 +6340,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b8e335ed20c4433b7450b0c1452727a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b8e335ed20c4433b7450b0c1452727a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db9e27a3e9364b3c927778d7bb894ff2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db9e27a3e9364b3c927778d7bb894ff2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6352,7 +6358,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6df50acc1fd745adbad2b215f6b7c646-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6df50acc1fd745adbad2b215f6b7c646-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d249245714b34fc4ad7ec1170108ba86-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d249245714b34fc4ad7ec1170108ba86-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6370,7 +6376,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42cffa01dcbc4e00b8ab5e24a9de9e90-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42cffa01dcbc4e00b8ab5e24a9de9e90-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f470aacde37a4b5b862354f79377bbc8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f470aacde37a4b5b862354f79377bbc8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6388,7 +6394,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de8e8c86d49f4baf804842d0ebbb59ff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de8e8c86d49f4baf804842d0ebbb59ff-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35be996e3a6e4f90a22ec36ccc72fed8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35be996e3a6e4f90a22ec36ccc72fed8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6406,7 +6412,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2459f079e39c4895ac0380c8ce393566-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2459f079e39c4895ac0380c8ce393566-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f0b8448f5c2e4e27ae1e7f16d267fbd1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f0b8448f5c2e4e27ae1e7f16d267fbd1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6455,7 +6461,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2bebe270481c4214b12016fc53e646ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2bebe270481c4214b12016fc53e646ca-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bd04244809fd49a2a28cd002d4db9f37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bd04244809fd49a2a28cd002d4db9f37-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6504,7 +6510,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e3c83b09f74c4290ad0f8c00b0d5c621-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e3c83b09f74c4290ad0f8c00b0d5c621-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fcc8a88061f94635a5f6b75c44c8bd94-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fcc8a88061f94635a5f6b75c44c8bd94-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6553,7 +6559,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5c1dd398141a4639b92825e9fd0205c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5c1dd398141a4639b92825e9fd0205c9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/595dd7bbfb794022b54192415e9d6eed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/595dd7bbfb794022b54192415e9d6eed-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6571,7 +6577,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42334c2a9acf45e395f84b4f8efb915b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42334c2a9acf45e395f84b4f8efb915b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/97f7d5981cb640bba36c249d2c6de553-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/97f7d5981cb640bba36c249d2c6de553-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6589,7 +6595,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/793ffa29f7364898928358804ce5a814-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/793ffa29f7364898928358804ce5a814-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/adccbb225aeb445c89bb41fa47c77c20-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/adccbb225aeb445c89bb41fa47c77c20-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6607,7 +6613,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8cb180f6391145f18d429ad23aedb11a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8cb180f6391145f18d429ad23aedb11a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/df3ca4f7901f4f53b597c2e30218b143-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/df3ca4f7901f4f53b597c2e30218b143-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6625,7 +6631,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1c6e3b3b402a4f5e982a9f25c1e2e7c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1c6e3b3b402a4f5e982a9f25c1e2e7c0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/213eae9c30144e77bf636f258cbb737c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/213eae9c30144e77bf636f258cbb737c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6643,7 +6649,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c67c7d2046b24d758cf51c2e606e08f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c67c7d2046b24d758cf51c2e606e08f3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9837b2513a184f4986fc37b1d796034c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9837b2513a184f4986fc37b1d796034c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6661,7 +6667,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6ef3ae4386904dfbb02a7e6cd66773de-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6ef3ae4386904dfbb02a7e6cd66773de-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8dcbb730969459f8f8b31e4514db9b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8dcbb730969459f8f8b31e4514db9b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6679,7 +6685,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89279b05a696401f9b51c5037205eef8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89279b05a696401f9b51c5037205eef8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0163b02bafe7490ca11b087732f25126-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0163b02bafe7490ca11b087732f25126-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6728,7 +6734,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d138389f08694ddfbc43a93f9fa60e08-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d138389f08694ddfbc43a93f9fa60e08-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9fca54bbf6ce4d37a1a9bb981aa974e8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9fca54bbf6ce4d37a1a9bb981aa974e8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6746,7 +6752,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/008b2227c9c642e0bbd9a4f2ae946885-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/008b2227c9c642e0bbd9a4f2ae946885-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e28c89176424a69b6984e2f778346c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e28c89176424a69b6984e2f778346c6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6764,7 +6770,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/883ef7558eea466797892a52453e6fc2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/883ef7558eea466797892a52453e6fc2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a063202b9f24088b25827efb8620ae3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a063202b9f24088b25827efb8620ae3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6782,7 +6788,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/95318ae9f8994ffbbd21331c52266b92-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/95318ae9f8994ffbbd21331c52266b92-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/04cf6fd3f33b495ea32c25938ed4c600-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/04cf6fd3f33b495ea32c25938ed4c600-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6800,7 +6806,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ceaf18c900e848608e3f2655084c083e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ceaf18c900e848608e3f2655084c083e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ed1b19a6a2f44845b71f92f227f75383-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ed1b19a6a2f44845b71f92f227f75383-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6818,7 +6824,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7b860fc88e6448738bbbf9e009d17688-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7b860fc88e6448738bbbf9e009d17688-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/54c49bd3d3a54ee580267c1003f18fd8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/54c49bd3d3a54ee580267c1003f18fd8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6836,7 +6842,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9b431eefebba4a33a6267f2a2fb9e7e5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9b431eefebba4a33a6267f2a2fb9e7e5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6cf109282aab439cab224460d86a8ef3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6cf109282aab439cab224460d86a8ef3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6854,7 +6860,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c8a47bd01514da0af95f87442b8b1b1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c8a47bd01514da0af95f87442b8b1b1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8a511deef5e446a6b24de7375274bd1e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8a511deef5e446a6b24de7375274bd1e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6872,7 +6878,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/264e977342a44c148ff1d4f8bd9df71a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/264e977342a44c148ff1d4f8bd9df71a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d6e32bdeca8a4f3cb04febf5b2d78d91-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d6e32bdeca8a4f3cb04febf5b2d78d91-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6890,7 +6896,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5247407d61a24955ad8cff3c5f8e7789-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5247407d61a24955ad8cff3c5f8e7789-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8a070b8aab0941d2a7d0a66eb2cf6a55-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8a070b8aab0941d2a7d0a66eb2cf6a55-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6908,7 +6914,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/144cf395197a47cf9d47912d120c1b7b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/144cf395197a47cf9d47912d120c1b7b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a6ab4a9f37a4ec48a046d5e2507cd6b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a6ab4a9f37a4ec48a046d5e2507cd6b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6926,7 +6932,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d9d1fa9287dc4ea6ab17304278ee5b68-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d9d1fa9287dc4ea6ab17304278ee5b68-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7a4e35fb91b44cb0864a07d7355b7346-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7a4e35fb91b44cb0864a07d7355b7346-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6944,7 +6950,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/868cef53c21d4b25acb8802964c30897-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/868cef53c21d4b25acb8802964c30897-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c8d30463339b4600b964e042333d6bda-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c8d30463339b4600b964e042333d6bda-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6962,7 +6968,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/068eb76a7a9045c7b8ee01605c7f5592-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/068eb76a7a9045c7b8ee01605c7f5592-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ce47e222bfe54cea9385935a86149fbf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ce47e222bfe54cea9385935a86149fbf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6980,7 +6986,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/18c333b18574415fb3947668aa66e73c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/18c333b18574415fb3947668aa66e73c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1c2b70780d504231b00d4f274a74e042-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1c2b70780d504231b00d4f274a74e042-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -6998,7 +7004,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2a77acab744e4c3f9b5ffb3317f738ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2a77acab744e4c3f9b5ffb3317f738ea-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2cf88ad8f2a94634bebea0ef6e4f40a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2cf88ad8f2a94634bebea0ef6e4f40a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7016,7 +7022,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4e2f40b06b5b43b8bbf9458d42d51618-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4e2f40b06b5b43b8bbf9458d42d51618-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/25a8bf291f4043b099cd03c5c0108407-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/25a8bf291f4043b099cd03c5c0108407-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7034,7 +7040,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7783c62f5eb840f39f878d27ec0f7658-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7783c62f5eb840f39f878d27ec0f7658-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ebf08f98b9f246c1a173929531135597-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ebf08f98b9f246c1a173929531135597-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7052,7 +7058,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5405a54207b143deb27cad6bb55e000d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5405a54207b143deb27cad6bb55e000d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b5d2f206a8848f591ffdd036193dcc3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b5d2f206a8848f591ffdd036193dcc3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7070,7 +7076,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/efec6407926e438c919f73bbeca13685-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/efec6407926e438c919f73bbeca13685-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/19c67bbe5d6a4d5c9fddc76cdd205c73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/19c67bbe5d6a4d5c9fddc76cdd205c73-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7088,7 +7094,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fe5ca4cf00374b42bcd68d9ca0f4833a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fe5ca4cf00374b42bcd68d9ca0f4833a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/42c687115e37425b819719b799beea36-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/42c687115e37425b819719b799beea36-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7106,7 +7112,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/11c6841860564050ae571c155ac6b218-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/11c6841860564050ae571c155ac6b218-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0671268529e14355875d15e11be7d4e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0671268529e14355875d15e11be7d4e9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7124,7 +7130,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/331d2d4b39cc42078be54d37e07d9146-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/331d2d4b39cc42078be54d37e07d9146-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/571de66453884c1abc3ade369e363ce0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/571de66453884c1abc3ade369e363ce0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7136,13 +7142,13 @@
"Option for timeMode changed from cpu to elapsed\n",
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fe41bf81e39644db8d932ac55fc51675-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fe41bf81e39644db8d932ac55fc51675-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1790cbae35f44a119c210fc8ef21bbe6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1790cbae35f44a119c210fc8ef21bbe6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7160,7 +7166,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8193ebbe7c934b2eb2baa8939e11ca4e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8193ebbe7c934b2eb2baa8939e11ca4e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cffa24be67a24ea49639b8640e324db6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cffa24be67a24ea49639b8640e324db6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7178,7 +7184,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3bd98da988a4efdb814edace8cd7e61-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3bd98da988a4efdb814edace8cd7e61-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c87a160dcb2b4a5f83d5882113458c35-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c87a160dcb2b4a5f83d5882113458c35-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7190,13 +7196,13 @@
"Option for timeMode changed from cpu to elapsed\n",
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5889ac97c0924ec8901dae12be0d5ccb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5889ac97c0924ec8901dae12be0d5ccb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1684e29363194badb973ba685990d62c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1684e29363194badb973ba685990d62c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7217,7 +7223,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/daa115562993455b8cba33d17ec335ae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/daa115562993455b8cba33d17ec335ae-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b05cc7ab546426fb067d66c0f39d1cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b05cc7ab546426fb067d66c0f39d1cf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7265,7 +7271,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7916a405941e4d1ba1234f61ff313eb7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7916a405941e4d1ba1234f61ff313eb7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f10c81cfa8db4cdda2a1f516d3d2a46b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f10c81cfa8db4cdda2a1f516d3d2a46b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7283,7 +7289,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9df89ba4150d414dbf7f527bd0469411-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9df89ba4150d414dbf7f527bd0469411-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f0221bbff9d84330b999993b1a4c5f88-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f0221bbff9d84330b999993b1a4c5f88-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7301,7 +7307,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/67b083dba5934c5d8ee065c89b860195-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/67b083dba5934c5d8ee065c89b860195-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a7466c26247e4e7895786f1268f6b57f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a7466c26247e4e7895786f1268f6b57f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7319,7 +7325,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3bbb8beb041448e7a87f84e68c205e0b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3bbb8beb041448e7a87f84e68c205e0b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b6b72ee5a12e49bf8ad8c421a5c26814-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b6b72ee5a12e49bf8ad8c421a5c26814-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7337,7 +7343,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ad857d2809b491ba5d0a4255a5ef4e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ad857d2809b491ba5d0a4255a5ef4e9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/efa2f9d4772043deab8c536acff1c382-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/efa2f9d4772043deab8c536acff1c382-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7355,7 +7361,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73b0bfa203ae47fdb96e3542dabb1468-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73b0bfa203ae47fdb96e3542dabb1468-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5c8be55b5584bcbbbc0eb936bbb772e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5c8be55b5584bcbbbc0eb936bbb772e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7373,7 +7379,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f11ea90ddff43c6b10750b5b9bccfa8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f11ea90ddff43c6b10750b5b9bccfa8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/10df3d177eae41a796b8a9b4a1a48095-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/10df3d177eae41a796b8a9b4a1a48095-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7391,7 +7397,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2584b0f4a5ff429f8f98dd15b1655b8b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2584b0f4a5ff429f8f98dd15b1655b8b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f4f004638144f00b74302b8032f3c99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f4f004638144f00b74302b8032f3c99-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7409,7 +7415,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1b27cf4f7fb3493da17e998a593bb3ad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1b27cf4f7fb3493da17e998a593bb3ad-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8db2eb3deeb4e0bacfc63d1396df28c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8db2eb3deeb4e0bacfc63d1396df28c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7427,7 +7433,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/604da1f36905491d824d40562b997831-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/604da1f36905491d824d40562b997831-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2be2901358941f4a1a0429189dd9604-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2be2901358941f4a1a0429189dd9604-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7445,7 +7451,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1b659c6621cd41ec914fe5b430962369-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1b659c6621cd41ec914fe5b430962369-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e45af2de208840ebb812bb67eb98c367-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e45af2de208840ebb812bb67eb98c367-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7463,7 +7469,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e0cae2d20b7f47ad8c63f1028c9b0a85-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e0cae2d20b7f47ad8c63f1028c9b0a85-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ea1889f4e424fc1a86f086243f1b724-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ea1889f4e424fc1a86f086243f1b724-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7481,7 +7487,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/74f769a4ace24b598b6a903573848b75-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/74f769a4ace24b598b6a903573848b75-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87bd872a4c934d6d8ad4811b700f079b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87bd872a4c934d6d8ad4811b700f079b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7499,7 +7505,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/631a6b8e68ab441ba6a9166364b53507-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/631a6b8e68ab441ba6a9166364b53507-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/85143d1a230747448ded2db126f73fc2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/85143d1a230747448ded2db126f73fc2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7517,7 +7523,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/068393705675408bb07cb24bc7c3d7f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/068393705675408bb07cb24bc7c3d7f2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/71757896f9bf499782ce8bf9cbbf10da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/71757896f9bf499782ce8bf9cbbf10da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7535,7 +7541,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6bf0d353ff6243beba1c2571547122f6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6bf0d353ff6243beba1c2571547122f6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73605d147fd14145bd1d0c8e1a051d47-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73605d147fd14145bd1d0c8e1a051d47-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7553,7 +7559,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e90102ec7b084716b563ee0d9013ad9c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e90102ec7b084716b563ee0d9013ad9c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f959fbcd15c448ee920a56fb274c255f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f959fbcd15c448ee920a56fb274c255f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7571,7 +7577,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a20769837ff94f85a4cd56371b10d97f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a20769837ff94f85a4cd56371b10d97f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ce167bb7c6fe414794996c86ff1a9cb5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ce167bb7c6fe414794996c86ff1a9cb5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7589,7 +7595,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4a056ae24d9c4556b8a8b0b33bf57335-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4a056ae24d9c4556b8a8b0b33bf57335-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c72ab84e59e14fd9a4382967905fb38d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c72ab84e59e14fd9a4382967905fb38d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7607,7 +7613,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e422a14d08ad4333b5dc72abe08cd657-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e422a14d08ad4333b5dc72abe08cd657-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fcd5bd6e39ac490284c2dc7413b38d82-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fcd5bd6e39ac490284c2dc7413b38d82-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7625,7 +7631,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/34e493ec6cf043b0a51de90de3dd228e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/34e493ec6cf043b0a51de90de3dd228e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7d96080d4f3343a9b1af4b7f29bd94e8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7d96080d4f3343a9b1af4b7f29bd94e8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7643,7 +7649,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ff4d86f772fb49acb88ea0276657de8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ff4d86f772fb49acb88ea0276657de8c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae7b1b0cfa82494cb2b504ca494a3262-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae7b1b0cfa82494cb2b504ca494a3262-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7661,7 +7667,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/df419082c61d4a8fa29fc5eadd4813c7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/df419082c61d4a8fa29fc5eadd4813c7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/03345cdefd2347ac9eb5336c243790f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/03345cdefd2347ac9eb5336c243790f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7679,7 +7685,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/70ce75b587dc4d8d87f80d1c625825d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/70ce75b587dc4d8d87f80d1c625825d3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/29532fe6e829482ca12d8a4e2b0d42f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/29532fe6e829482ca12d8a4e2b0d42f9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7697,7 +7703,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb73d20aa1b6421187185e1ef9e2cfd4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb73d20aa1b6421187185e1ef9e2cfd4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/92e7e6cf6bd84fbd9a0af0a26084d277-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/92e7e6cf6bd84fbd9a0af0a26084d277-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7715,7 +7721,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53e9d3708c1248569096c199469bfeeb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53e9d3708c1248569096c199469bfeeb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/93ffff9b0ea3466c9f186502366248a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/93ffff9b0ea3466c9f186502366248a8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7733,7 +7739,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/599d8c8ca95b4142a85c7ef5e9731cc1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/599d8c8ca95b4142a85c7ef5e9731cc1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/305b8d981d784cd6b26599d4739631b6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/305b8d981d784cd6b26599d4739631b6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7751,7 +7757,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5377615b12c446c9b2b17465633d221b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5377615b12c446c9b2b17465633d221b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e027727bf434edd9c68093b62e632c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e027727bf434edd9c68093b62e632c9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7769,7 +7775,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/001ff1ca15774c36b8d4e0223abce717-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/001ff1ca15774c36b8d4e0223abce717-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/150e5671671e4819a3012d72f3752e66-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/150e5671671e4819a3012d72f3752e66-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7787,7 +7793,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/df08c5c3e9aa48f68d69bf3e74ad530d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/df08c5c3e9aa48f68d69bf3e74ad530d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cda753ce51434c8a90ecdd3b0c5ec041-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cda753ce51434c8a90ecdd3b0c5ec041-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7805,7 +7811,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ede84f62e9bb4801977f86c5b3356120-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ede84f62e9bb4801977f86c5b3356120-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b632c3d3a940434e944dbf6e109523e0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b632c3d3a940434e944dbf6e109523e0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7823,7 +7829,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f358b2d91084131bf9da3ed0546d3be-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f358b2d91084131bf9da3ed0546d3be-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a23d1c4fe2ef440c8501c743b440366b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a23d1c4fe2ef440c8501c743b440366b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7841,7 +7847,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0e3e800c8ac24ec4add226c2231a8e88-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0e3e800c8ac24ec4add226c2231a8e88-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6c597bb46f01469cbc6cfb6a59f5de5e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6c597bb46f01469cbc6cfb6a59f5de5e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7859,7 +7865,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d4ca70482dcc4bffab20739be9cab280-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d4ca70482dcc4bffab20739be9cab280-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/133e1a2f77694bd0b8077201e3c61154-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/133e1a2f77694bd0b8077201e3c61154-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7877,7 +7883,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/786e1d5d2b2b4086a5a4a47007476f5f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/786e1d5d2b2b4086a5a4a47007476f5f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ca8a354b4d26409a992c68272e6ac4af-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ca8a354b4d26409a992c68272e6ac4af-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7895,7 +7901,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd6a4c30d3224c9498fb3648d326a978-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd6a4c30d3224c9498fb3648d326a978-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f43fbfa6182843ed87c7aff83de2fae0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f43fbfa6182843ed87c7aff83de2fae0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7913,7 +7919,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c5c74b325dbd430f995ecda8eca7a0dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c5c74b325dbd430f995ecda8eca7a0dd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8af14e41cae14119a4142829f82e0462-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8af14e41cae14119a4142829f82e0462-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7931,7 +7937,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a739bb1be2b4f378d0ed4d3db42fcf6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a739bb1be2b4f378d0ed4d3db42fcf6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/600d1553f5394e6fa1c322a62945bbc4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/600d1553f5394e6fa1c322a62945bbc4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7949,7 +7955,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a9f65e2ebb0c41ea87ee3025539bb211-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a9f65e2ebb0c41ea87ee3025539bb211-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/27b2c96a73f6444d979a36be7c6d6b51-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/27b2c96a73f6444d979a36be7c6d6b51-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7967,7 +7973,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aa52be4e3b8d49c8b4d920def47aab72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aa52be4e3b8d49c8b4d920def47aab72-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4b186ecf2cc44545a995f8cc1ab05361-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4b186ecf2cc44545a995f8cc1ab05361-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -7985,7 +7991,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3cee73616a3407ba1a989186705b0e6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3cee73616a3407ba1a989186705b0e6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f6378d533aa54194b007959bb677ab0e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f6378d533aa54194b007959bb677ab0e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8003,7 +8009,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7f440c106d2e49e285a6750a171fa36b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7f440c106d2e49e285a6750a171fa36b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b64811920b3a49b3abda0b1cfe61bf69-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b64811920b3a49b3abda0b1cfe61bf69-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8021,7 +8027,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1a28cc596ab441aadd45ab6819fa302-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1a28cc596ab441aadd45ab6819fa302-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d74644b9dad482193c797c170ba5359-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d74644b9dad482193c797c170ba5359-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8039,7 +8045,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f65eba5abfb4893bb39f9df85721839-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f65eba5abfb4893bb39f9df85721839-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/735fc91cd75e4eb5b3277a2c4fcadf15-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/735fc91cd75e4eb5b3277a2c4fcadf15-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8057,7 +8063,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d56ceab269894d1d9c3dd68c6f903e30-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d56ceab269894d1d9c3dd68c6f903e30-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3eca6bb94e7d4a8989ac9460aa3fc030-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3eca6bb94e7d4a8989ac9460aa3fc030-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8075,7 +8081,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53ace5d2b6224b6ab4656cd8558da263-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53ace5d2b6224b6ab4656cd8558da263-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/72fd3b128fec499ea9611a0aa159edea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/72fd3b128fec499ea9611a0aa159edea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8093,7 +8099,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7e9c0267bb70490383824f16d2319528-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7e9c0267bb70490383824f16d2319528-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e1cbbeeaad4b4f1dbe03c46c4e9e767f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e1cbbeeaad4b4f1dbe03c46c4e9e767f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8111,7 +8117,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3881cb96a0845ff89bc589578c836c8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3881cb96a0845ff89bc589578c836c8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d720aa97f9db47528b49a995c1b223c1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d720aa97f9db47528b49a995c1b223c1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8129,7 +8135,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6cf1ede6485444158270c2aa04f17bb0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6cf1ede6485444158270c2aa04f17bb0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8a6c5f904b5e4ec78240be0670a1c286-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8a6c5f904b5e4ec78240be0670a1c286-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8147,7 +8153,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b68410beadf64fb7b63c2cad84f2a455-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b68410beadf64fb7b63c2cad84f2a455-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dd5d39cd2b094cc989c61a8fa2767047-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dd5d39cd2b094cc989c61a8fa2767047-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8165,7 +8171,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b7a4946c60f046719a8fcb1bbecfc94e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b7a4946c60f046719a8fcb1bbecfc94e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/321f2edb826a45aa957024af90fa6b99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/321f2edb826a45aa957024af90fa6b99-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8183,7 +8189,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a3fe1e147fb4418ba0421a5cd87f082e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a3fe1e147fb4418ba0421a5cd87f082e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/397c86c544084d5bbca1c8400a7fc269-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/397c86c544084d5bbca1c8400a7fc269-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8201,7 +8207,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5d8365e7adc4d6f9c670419bc2d336f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5d8365e7adc4d6f9c670419bc2d336f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f5625d2f63a49ae9fb367db6deb603e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f5625d2f63a49ae9fb367db6deb603e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8219,7 +8225,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e0a41f6c820546ccbd372929f3dd1d37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e0a41f6c820546ccbd372929f3dd1d37-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3e66ef0273924d20b404b81d70075bc4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3e66ef0273924d20b404b81d70075bc4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8237,7 +8243,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a6bf98697d9846d386c661a4772c6e10-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a6bf98697d9846d386c661a4772c6e10-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8da08557d7e5487c9aca7a157c35979a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8da08557d7e5487c9aca7a157c35979a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8255,7 +8261,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d8e574aac0a43a49e629f5e65851855-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d8e574aac0a43a49e629f5e65851855-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aaca8c04e1da4daba1dee95d4c2cbb85-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aaca8c04e1da4daba1dee95d4c2cbb85-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8273,7 +8279,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4dcf18cca8d142ff84dabd5e7658aee4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4dcf18cca8d142ff84dabd5e7658aee4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13a18c94bc94458d8a7306210341edbf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13a18c94bc94458d8a7306210341edbf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8291,7 +8297,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9cf0bf80e3624f20a0bf310ff752661a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9cf0bf80e3624f20a0bf310ff752661a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cc9ec6945462413c9f398bf93225ac02-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cc9ec6945462413c9f398bf93225ac02-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8309,7 +8315,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de1b0117cd3a435782188040e41bd0dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de1b0117cd3a435782188040e41bd0dd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2de592727fad44249abd9ecb556d6ac5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2de592727fad44249abd9ecb556d6ac5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8327,7 +8333,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c6b9d262eda7431382a23f034fbbcf89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c6b9d262eda7431382a23f034fbbcf89-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cc3508c09bf24e42b1f561bf32e2d43a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cc3508c09bf24e42b1f561bf32e2d43a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8345,7 +8351,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eae86e0afc6e4cec9a01f160f80f3303-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eae86e0afc6e4cec9a01f160f80f3303-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/01aadeb334784120984e5992064022b7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/01aadeb334784120984e5992064022b7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8363,7 +8369,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ed1d875c53a24cf4a933d20511b15caa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ed1d875c53a24cf4a933d20511b15caa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/03f9d10901cb45eb82941d7f0db0f46d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/03f9d10901cb45eb82941d7f0db0f46d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8381,7 +8387,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f04795142f174f2e9ac61ca399f672d4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f04795142f174f2e9ac61ca399f672d4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb7483d8ed6b46a0b70d3e824b7d758d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb7483d8ed6b46a0b70d3e824b7d758d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8399,7 +8405,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b330f85ecc4f490eb8b785523f972ba8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b330f85ecc4f490eb8b785523f972ba8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6417136b367549caa24e3e5bb967ce37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6417136b367549caa24e3e5bb967ce37-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8417,7 +8423,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bee673bfffd047eebb98c75d7da71c2c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bee673bfffd047eebb98c75d7da71c2c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/27e4fe48c5544f3f9c63ec7d32bcc948-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/27e4fe48c5544f3f9c63ec7d32bcc948-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8435,7 +8441,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/575f4d652c2f4f50bd3c406f5b4df376-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/575f4d652c2f4f50bd3c406f5b4df376-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/29ebecf836d94b728e94157a9ab416ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/29ebecf836d94b728e94157a9ab416ed-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8453,7 +8459,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0693762c18f44f37b55260083e71b57f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0693762c18f44f37b55260083e71b57f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/29001d3e8b0946dc9dc1cac5db5c91a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/29001d3e8b0946dc9dc1cac5db5c91a9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8471,7 +8477,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e374560f192f493fb993fe9b88b05e58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e374560f192f493fb993fe9b88b05e58-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68c8e46f22d54500981d19ef3cd9553e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68c8e46f22d54500981d19ef3cd9553e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8489,7 +8495,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7190fb4413334df98bb27cf9d198951b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7190fb4413334df98bb27cf9d198951b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1b34737f65745bfa0bd6c780f08598e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1b34737f65745bfa0bd6c780f08598e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8507,7 +8513,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bba0d663e17f4a66bb30f5a7e8a793fe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bba0d663e17f4a66bb30f5a7e8a793fe-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7044dd482c4a4830ade7455cd67a4c22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7044dd482c4a4830ade7455cd67a4c22-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8525,7 +8531,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0290337d1b36497ab4503e27d8418bbe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0290337d1b36497ab4503e27d8418bbe-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5a8b7bbab1e24a72803046c38b805a58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5a8b7bbab1e24a72803046c38b805a58-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8545,7 +8551,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1f790bee32f74ed2b0a643c9e0bd4ec7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1f790bee32f74ed2b0a643c9e0bd4ec7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/92cfa2b5d04b4510868930ab82b4f359-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/92cfa2b5d04b4510868930ab82b4f359-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8566,7 +8572,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b4a9f5ae9b444a5381fa1fbcc811f846-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b4a9f5ae9b444a5381fa1fbcc811f846-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3a42779ed7ff43999cb84e44d891acd6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3a42779ed7ff43999cb84e44d891acd6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8587,7 +8593,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b0cacf046aec43d1851c6bb323bd022e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b0cacf046aec43d1851c6bb323bd022e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2058867de18c421dbb0e61e44d489bc4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2058867de18c421dbb0e61e44d489bc4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8608,7 +8614,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc0358ebeb574126b8bdd75fc2cc0256-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc0358ebeb574126b8bdd75fc2cc0256-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/919b906c087f454fb1b15767ff654575-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/919b906c087f454fb1b15767ff654575-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8629,7 +8635,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a315c3d8fb7c4d5987a7de6736e1c8ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a315c3d8fb7c4d5987a7de6736e1c8ea-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cf0e250d1c1b406d91aa3ece32581a2b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cf0e250d1c1b406d91aa3ece32581a2b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8650,7 +8656,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c03f91f0ef4c4c6c831bc5b7f51847a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c03f91f0ef4c4c6c831bc5b7f51847a4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8d570ce2055d4fb8bb22bf9a6c5b787b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8d570ce2055d4fb8bb22bf9a6c5b787b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8671,7 +8677,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/19035d7db5cf4d8eb79b7e487c8baa89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/19035d7db5cf4d8eb79b7e487c8baa89-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3bc9327a85d14e5ca5f5ad19e623edf7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3bc9327a85d14e5ca5f5ad19e623edf7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8692,7 +8698,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/316e50c2990d47bd81a19060bec53eb8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/316e50c2990d47bd81a19060bec53eb8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e90dfbabbff14854a425692a6c911805-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e90dfbabbff14854a425692a6c911805-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8714,7 +8720,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4b1a66ed112d4d31ae4a05f5ae034523-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4b1a66ed112d4d31ae4a05f5ae034523-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5e631bf0c58a429ea1974c96b322fb52-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5e631bf0c58a429ea1974c96b322fb52-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8736,7 +8742,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1193145eab734b85967586cef3e1af48-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1193145eab734b85967586cef3e1af48-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f5cdea6e38441b5b18147103be16fec-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f5cdea6e38441b5b18147103be16fec-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8754,7 +8760,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d73ecc606ab0460ab73bef917643c73c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d73ecc606ab0460ab73bef917643c73c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1c8934efd71c497285781ae9f6e92b0d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1c8934efd71c497285781ae9f6e92b0d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8772,7 +8778,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b6a7c0a76364d3897908367ac7149c7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b6a7c0a76364d3897908367ac7149c7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f4421e03824e41c983e357e365f5925d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f4421e03824e41c983e357e365f5925d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8790,7 +8796,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/953d580217e04a3390892cbef139b365-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/953d580217e04a3390892cbef139b365-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/897a5c4317b54a9287dfbab8aa35f58a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/897a5c4317b54a9287dfbab8aa35f58a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8808,7 +8814,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/785afe73308045fab0a417ff9bcbcf5b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/785afe73308045fab0a417ff9bcbcf5b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4fbb277cc7c24b19913390082cb5ad02-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4fbb277cc7c24b19913390082cb5ad02-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8826,7 +8832,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/296119015c1a448bb32aea4fe0c3d9fa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/296119015c1a448bb32aea4fe0c3d9fa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b9461b5af79247eaaab0351b698443e5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b9461b5af79247eaaab0351b698443e5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8847,7 +8853,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ad27fbaa633741769f897adaf8e5ef37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ad27fbaa633741769f897adaf8e5ef37-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f66a47b02bd6412ca02211a64eb2cb36-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f66a47b02bd6412ca02211a64eb2cb36-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8896,7 +8902,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b56caca810c241199fb2f30c780cfd89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b56caca810c241199fb2f30c780cfd89-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/38ea321d41c74edaa100416184b2fbad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/38ea321d41c74edaa100416184b2fbad-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8935,7 +8941,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b3f3200c41743c19ba76a5e1b198904-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b3f3200c41743c19ba76a5e1b198904-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75791f699109455e9d8dce3255b9be07-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75791f699109455e9d8dce3255b9be07-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -8984,7 +8990,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/336aaff5540344e8a1220399cd5f614e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/336aaff5540344e8a1220399cd5f614e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6ad6ade40a9a4250ac82b1dd4b0125ab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6ad6ade40a9a4250ac82b1dd4b0125ab-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9005,7 +9011,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/af6bafd753d44a57980a20ed12835f9e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/af6bafd753d44a57980a20ed12835f9e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8ada32778c7455f97d332f29ee24edc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8ada32778c7455f97d332f29ee24edc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9027,7 +9033,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/26c0662106c24bd2a604761442ccaee9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/26c0662106c24bd2a604761442ccaee9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8beb80b5e82d4300a9efcec9d5140c1a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8beb80b5e82d4300a9efcec9d5140c1a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9049,7 +9055,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/789b26e615b949e3b3710b047dd18fed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/789b26e615b949e3b3710b047dd18fed-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7d4eaa4ca40a44ea9cbfe31e5f85f0bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7d4eaa4ca40a44ea9cbfe31e5f85f0bd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9065,13 +9071,13 @@
"1 infeasibilities\n",
"Problem is infeasible!\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5179e761f32c4c7e9a063499fe5a47ce-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5179e761f32c4c7e9a063499fe5a47ce-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee049c9162a44259a09f3e5a26761ed0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee049c9162a44259a09f3e5a26761ed0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9089,7 +9095,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e8200b3c767543309d676dbb6611a58d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e8200b3c767543309d676dbb6611a58d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f89bc8b90a5a484b87f92d0ba1000cca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f89bc8b90a5a484b87f92d0ba1000cca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9107,7 +9113,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c458f0e6aa9f4d4aa924f4176f9b2fcf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c458f0e6aa9f4d4aa924f4176f9b2fcf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/317ead829dd743248e01e1318ee95798-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/317ead829dd743248e01e1318ee95798-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9125,7 +9131,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/41a4d753706d46c2b19f6100fa0fdcaf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/41a4d753706d46c2b19f6100fa0fdcaf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c4ba6b1aa30b4bbe82f248584d81e2b1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c4ba6b1aa30b4bbe82f248584d81e2b1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9143,7 +9149,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/44dbc259d79d48b993c3113b5658f16f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/44dbc259d79d48b993c3113b5658f16f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bbdac92a20204c1892c13b1990572d72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bbdac92a20204c1892c13b1990572d72-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9161,7 +9167,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/63f820d185bc495f8f61aca1b9e42e88-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/63f820d185bc495f8f61aca1b9e42e88-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eca9950de6294ad99a097a016fffebbd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eca9950de6294ad99a097a016fffebbd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9182,7 +9188,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bf7bdfb07f264698a4e4a6d494cd104f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bf7bdfb07f264698a4e4a6d494cd104f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b291849630b94bbf92534da4050412e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b291849630b94bbf92534da4050412e9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9231,7 +9237,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1efa14528ab453b9b114ac90ab6c20d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1efa14528ab453b9b114ac90ab6c20d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e5bf4f1d8405483c80a8ad7be6b82d6c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e5bf4f1d8405483c80a8ad7be6b82d6c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9280,7 +9286,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/929475798e76482581a82a07af23ea71-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/929475798e76482581a82a07af23ea71-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/51c07d0cc3344332929272db64e22ef1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/51c07d0cc3344332929272db64e22ef1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9301,7 +9307,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/590c03333ab84587ba6506c3c9b11027-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/590c03333ab84587ba6506c3c9b11027-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46cdbcc203c54dff8e46fe8f40bee877-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46cdbcc203c54dff8e46fe8f40bee877-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9323,7 +9329,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6cf35706a6ff4261b0a349bb86c5d30e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6cf35706a6ff4261b0a349bb86c5d30e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a28d427b88f344df94537521ad221d1f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a28d427b88f344df94537521ad221d1f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9344,7 +9350,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b229cf9164ab4765bd1a7d46cd7fbf82-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b229cf9164ab4765bd1a7d46cd7fbf82-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/879e09a2706d4ec1be9f6eefe1ba285c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/879e09a2706d4ec1be9f6eefe1ba285c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9362,7 +9368,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eb57e2b6d38a4279bd5247086a9cb1b1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eb57e2b6d38a4279bd5247086a9cb1b1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f28f6e70678c4355b2d8d64e259cc0c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f28f6e70678c4355b2d8d64e259cc0c0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9380,7 +9386,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7c63a0b4376a4baaa65b8fcb512e2280-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7c63a0b4376a4baaa65b8fcb512e2280-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/009af0042c1347098f64e44e386c27b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/009af0042c1347098f64e44e386c27b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9398,7 +9404,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e81108ee789a45f7bd1c7516e96fc1a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e81108ee789a45f7bd1c7516e96fc1a9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75f016aea8974e368d7936b65129cb3d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75f016aea8974e368d7936b65129cb3d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9416,7 +9422,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/053ba3cb46f34ad5a4586e68cc7e79de-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/053ba3cb46f34ad5a4586e68cc7e79de-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f194252e3ca44dea95205d6489bb39c3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f194252e3ca44dea95205d6489bb39c3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9434,7 +9440,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/76fb5228ff2c41b987e1bff114b18a5a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/76fb5228ff2c41b987e1bff114b18a5a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/026028fbbf93439888135cdb33115d09-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/026028fbbf93439888135cdb33115d09-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9452,7 +9458,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2ace1596444344c2a4bddf72e30b0f14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2ace1596444344c2a4bddf72e30b0f14-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/44673adfdf7946078836c195be19f5ac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/44673adfdf7946078836c195be19f5ac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9473,7 +9479,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7102e691812746e8a4dab6b61a24f0f9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7102e691812746e8a4dab6b61a24f0f9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a03455c1a6d41499f9fd484a706fa73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a03455c1a6d41499f9fd484a706fa73-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9521,7 +9527,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/50d51d5ffd714aed95293fe34a71d981-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/50d51d5ffd714aed95293fe34a71d981-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/064f0570051e488b8db24dbca7a5ece6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/064f0570051e488b8db24dbca7a5ece6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9569,7 +9575,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/38f531df576b4ae1b8a90ca7dd754024-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/38f531df576b4ae1b8a90ca7dd754024-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9e9dd2b026ee45f2aec3d479dba35829-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9e9dd2b026ee45f2aec3d479dba35829-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9616,7 +9622,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9f8330eb2d504342857ffd7b3444b8ab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9f8330eb2d504342857ffd7b3444b8ab-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9eabaa3d150846d1a562d7b25f03688d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9eabaa3d150846d1a562d7b25f03688d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9634,7 +9640,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/63d08ab3bba147a894da6ba0f6706397-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/63d08ab3bba147a894da6ba0f6706397-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/898aabf35af64125a44fcd9aee8ad12a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/898aabf35af64125a44fcd9aee8ad12a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9652,7 +9658,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5458d7ca8b0646f49cb83d14a943acf8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5458d7ca8b0646f49cb83d14a943acf8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/be086ddccaa64d5688ac03ba9ddbd408-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/be086ddccaa64d5688ac03ba9ddbd408-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9670,7 +9676,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4749b7dd13f64efc828feea540d6f80b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4749b7dd13f64efc828feea540d6f80b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13f6174540914b17b3913ac11097cb41-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13f6174540914b17b3913ac11097cb41-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9688,7 +9694,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b959582504f94d2992bf09143a1b4e1e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b959582504f94d2992bf09143a1b4e1e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/80553692b72443af95b296f294033c47-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/80553692b72443af95b296f294033c47-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9706,7 +9712,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/37863a3886434c44bc239e55536a448d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/37863a3886434c44bc239e55536a448d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/56f145154d9a4ecf87ddf53ec62ee785-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/56f145154d9a4ecf87ddf53ec62ee785-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9724,7 +9730,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d640f190786348e69a4b35fcf44e9440-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d640f190786348e69a4b35fcf44e9440-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4638a11c8d4945d7821285a89fd0c4d7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4638a11c8d4945d7821285a89fd0c4d7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9745,7 +9751,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c80f496c39f64f7ba70a181a5fe22467-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c80f496c39f64f7ba70a181a5fe22467-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c26cbd3e56a5458cb8f6cefe3e018047-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c26cbd3e56a5458cb8f6cefe3e018047-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9792,7 +9798,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cf97c43a758d47399b40e65bcb2a4f82-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cf97c43a758d47399b40e65bcb2a4f82-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1e92b3bb266414ab06380fdb7f181ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1e92b3bb266414ab06380fdb7f181ca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9810,7 +9816,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e065f2e630474ec6ad420c79b5d67f3a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e065f2e630474ec6ad420c79b5d67f3a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a57bf177ac294ed8be1bf0bf76e3b122-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a57bf177ac294ed8be1bf0bf76e3b122-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9828,7 +9834,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7bdd1ae3fecf452284deded6c739a662-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7bdd1ae3fecf452284deded6c739a662-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d077e482693f4f41b0727745369f974f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d077e482693f4f41b0727745369f974f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9846,7 +9852,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00ed44d0105046a69f87b820a65f632c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00ed44d0105046a69f87b820a65f632c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b013912e802843a296dd9c5fc9eaeb7c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b013912e802843a296dd9c5fc9eaeb7c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9864,7 +9870,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33c55eed72f243acab35dd34f68754e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33c55eed72f243acab35dd34f68754e9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b3044427ab64a658cdcb7722d7fcb02-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b3044427ab64a658cdcb7722d7fcb02-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9882,7 +9888,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d871a0cfe7e148dbbf5eb56daad821d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d871a0cfe7e148dbbf5eb56daad821d2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae2c4b8adcf946bf90acce5ab456cb3f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae2c4b8adcf946bf90acce5ab456cb3f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9900,7 +9906,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fce9bd557e2b47f8a22207c00f1b56e3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fce9bd557e2b47f8a22207c00f1b56e3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fcb061eae95c49a1be01361bb5369be2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fcb061eae95c49a1be01361bb5369be2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9918,7 +9924,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bad68e5d6cea4d9f8d2bb775ccf83d89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bad68e5d6cea4d9f8d2bb775ccf83d89-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9dae0da3b9804a2aabc764885f627a7e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9dae0da3b9804a2aabc764885f627a7e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9936,7 +9942,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4f1dc002a508429abc5724eadeb9ffbc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4f1dc002a508429abc5724eadeb9ffbc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ac08b12d981430ca9ebecba450c3c5b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ac08b12d981430ca9ebecba450c3c5b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9954,7 +9960,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c20f5f4da351408aadf8e0dbbb96e8a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c20f5f4da351408aadf8e0dbbb96e8a1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a34da75f95fa4c52baa8658c4335543e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a34da75f95fa4c52baa8658c4335543e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9972,7 +9978,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c7e4509ef47e4c0ab60607e9512e39ac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c7e4509ef47e4c0ab60607e9512e39ac-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c461d0b3fbcc4d758047920e1c372a67-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c461d0b3fbcc4d758047920e1c372a67-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -9990,7 +9996,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fe8df827e8c245569ac4e8bbbfb289b4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fe8df827e8c245569ac4e8bbbfb289b4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d5e745ff5b5f44c9b9867c96aefa4c1d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d5e745ff5b5f44c9b9867c96aefa4c1d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10008,7 +10014,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/39a22a42283d43b3a601ac28455395ec-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/39a22a42283d43b3a601ac28455395ec-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3b96a43e7c1d451d8b04af043267b05a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3b96a43e7c1d451d8b04af043267b05a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10026,7 +10032,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5e7cfd9126cf498f925dcea00d210caa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5e7cfd9126cf498f925dcea00d210caa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eb8dcce44055465c8aa5ee55c252da3e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eb8dcce44055465c8aa5ee55c252da3e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10044,7 +10050,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4ec8592079694d70b766fd7766dbc46a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4ec8592079694d70b766fd7766dbc46a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/750b750ed2e64897a95d47d4c13197f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/750b750ed2e64897a95d47d4c13197f2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10062,7 +10068,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13886f7a38354924a3acb573177c593d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13886f7a38354924a3acb573177c593d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9c121f19bdee41fda771f2aa5148d509-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9c121f19bdee41fda771f2aa5148d509-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10080,7 +10086,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f69df01eeecf46e7bb9a214b853d3477-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f69df01eeecf46e7bb9a214b853d3477-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b2dc7de49f4a4f018fe30a5d6b305aa3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b2dc7de49f4a4f018fe30a5d6b305aa3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10098,7 +10104,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6310211eda584b45be1606444fe0c30b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6310211eda584b45be1606444fe0c30b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c0eaf80bf76b488a887a126251503eaf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c0eaf80bf76b488a887a126251503eaf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10116,7 +10122,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eaa5658c8acd4c288286efb1b18424bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eaa5658c8acd4c288286efb1b18424bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4029193ba8bb43838a96bf4a45360143-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4029193ba8bb43838a96bf4a45360143-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10134,7 +10140,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7a77ae989939414f9fd0e626fff06628-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7a77ae989939414f9fd0e626fff06628-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ec32ecf58ed54afc947948b893fed16f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ec32ecf58ed54afc947948b893fed16f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10152,7 +10158,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8c5e522904d4a65bfc80927ebd51b95-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8c5e522904d4a65bfc80927ebd51b95-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/19889df5df4443709666a418070c1ce6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/19889df5df4443709666a418070c1ce6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10170,7 +10176,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5eccc82c532d41e8ae4d35e1bf24b2ab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5eccc82c532d41e8ae4d35e1bf24b2ab-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9cfc6272c9ca4ae88aaf58a1cf5deec2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9cfc6272c9ca4ae88aaf58a1cf5deec2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10188,7 +10194,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ab85ac4668d4ed0a987fdb6042e9c8e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ab85ac4668d4ed0a987fdb6042e9c8e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4a5071f4df3a42328f9733f24b9622e6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4a5071f4df3a42328f9733f24b9622e6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10206,7 +10212,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b6717053e7349e38740f3b01dbd698d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b6717053e7349e38740f3b01dbd698d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c7168d1aa24143a59db0ccf4dc0582c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c7168d1aa24143a59db0ccf4dc0582c5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10224,7 +10230,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de3ccc3be3394acbbd41bf74d1c8a1fd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de3ccc3be3394acbbd41bf74d1c8a1fd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87974044d68441b78f1389b9674c40d1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87974044d68441b78f1389b9674c40d1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10242,7 +10248,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f47cf82dc0bf4dc89845925d29b8d2d1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f47cf82dc0bf4dc89845925d29b8d2d1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/551fb477a0ee41d2b452cbca2130c92d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/551fb477a0ee41d2b452cbca2130c92d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10260,7 +10266,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b06877bd4a5487b94cb5c7c9e629dd9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b06877bd4a5487b94cb5c7c9e629dd9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ebbd56a30e6e4164b08cd1ef33cfdf97-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ebbd56a30e6e4164b08cd1ef33cfdf97-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10278,7 +10284,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ec5ae3193ba4403e98b43fdc3113d7fa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ec5ae3193ba4403e98b43fdc3113d7fa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e1166aee239148f7af71df3ef27b8515-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e1166aee239148f7af71df3ef27b8515-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10296,7 +10302,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/682188237ccd4cfa996af549a071e41f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/682188237ccd4cfa996af549a071e41f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7a0440d9155d48ee97ff6770c228935c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7a0440d9155d48ee97ff6770c228935c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10317,7 +10323,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8729b5ebbfbd4ac1aca03f09709bc852-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8729b5ebbfbd4ac1aca03f09709bc852-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cc9233d5902a4aa88136c260bf9c6fc9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cc9233d5902a4aa88136c260bf9c6fc9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10366,7 +10372,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd42d3b290504592aea1154d8c42c0f0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd42d3b290504592aea1154d8c42c0f0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00a58ca67d754cdd88929624dd92282d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00a58ca67d754cdd88929624dd92282d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10415,7 +10421,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eafdde4f4e814273ad0219ca5b22b98a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eafdde4f4e814273ad0219ca5b22b98a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/743adb2f4be24b05b1c455863e498930-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/743adb2f4be24b05b1c455863e498930-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10464,7 +10470,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7368fef02d504858b861bdc7f1674a91-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7368fef02d504858b861bdc7f1674a91-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/41d3e4db4ca7446bb5a8d9d45214cd74-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/41d3e4db4ca7446bb5a8d9d45214cd74-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10486,7 +10492,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/63c2243449f84b6c9c50c8effdc4b35e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/63c2243449f84b6c9c50c8effdc4b35e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee2b602f311d4a8198cece5f390ed647-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee2b602f311d4a8198cece5f390ed647-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10508,7 +10514,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0f12c6fc4c4a4ad8902442666f33c75a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0f12c6fc4c4a4ad8902442666f33c75a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e28de0ca8cfb4d16b53d1f96201c107e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e28de0ca8cfb4d16b53d1f96201c107e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10529,7 +10535,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc6206f3ce9f402a961060a835bbc012-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc6206f3ce9f402a961060a835bbc012-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7b279a0024df4a4eb04d1e2280cb9d3f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7b279a0024df4a4eb04d1e2280cb9d3f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10547,7 +10553,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e83349b8984a475f89347f4f48749002-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e83349b8984a475f89347f4f48749002-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/236e6876c02841aca728c4d9c1a9b784-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/236e6876c02841aca728c4d9c1a9b784-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10565,7 +10571,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5d1b505662a942459fb4cb52fd9ea337-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5d1b505662a942459fb4cb52fd9ea337-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f98e07d9607642619f6bdc1fc139ce22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f98e07d9607642619f6bdc1fc139ce22-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10583,7 +10589,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/14201552e7c840dfb91ba583eafd8bcf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/14201552e7c840dfb91ba583eafd8bcf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f1d00368b4f4975a63f57278a901f14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f1d00368b4f4975a63f57278a901f14-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10601,7 +10607,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bbffb92c6edd43638f71c999eb46e456-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bbffb92c6edd43638f71c999eb46e456-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c89a374af5f2484fa901128c2a359046-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c89a374af5f2484fa901128c2a359046-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10619,7 +10625,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/152208c7d632474e8391ee7f7f7a4a20-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/152208c7d632474e8391ee7f7f7a4a20-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73c619e2316947e5900b791d5935c7ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73c619e2316947e5900b791d5935c7ef-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10637,7 +10643,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de645ba6b8584511b12e708381a4efc7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de645ba6b8584511b12e708381a4efc7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba1c966a73ff4e7c8591c4eaaa5a74ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba1c966a73ff4e7c8591c4eaaa5a74ef-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10686,7 +10692,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea87a1fc067f4242b473f03ff6f55950-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea87a1fc067f4242b473f03ff6f55950-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2dd7b1f6d93a4608891b0ee634f30aea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2dd7b1f6d93a4608891b0ee634f30aea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10709,7 +10715,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e5065932932649ae864f62f089ecee05-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e5065932932649ae864f62f089ecee05-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b4732c05a1d4647a15af8b3357ee2d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b4732c05a1d4647a15af8b3357ee2d9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10732,7 +10738,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ed3f4354b744d2c857052984667a28a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ed3f4354b744d2c857052984667a28a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4fc2aceaf39a48798a9bd96c89fbb9f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4fc2aceaf39a48798a9bd96c89fbb9f5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10754,7 +10760,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9723de263bb74b91831b847580d55312-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9723de263bb74b91831b847580d55312-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/12d91fb80cc645ae9097075f9769245b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/12d91fb80cc645ae9097075f9769245b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10775,7 +10781,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d042a712a90c4e1f9c1b5a449aaefbc0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d042a712a90c4e1f9c1b5a449aaefbc0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea7747629a3a4e7ea788b0f4fb58e3a3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea7747629a3a4e7ea788b0f4fb58e3a3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10793,7 +10799,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/975184ffbf7c4fbdb19515959e6c7b44-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/975184ffbf7c4fbdb19515959e6c7b44-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b6411815919e48e49468bedfce94f8f0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b6411815919e48e49468bedfce94f8f0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10811,7 +10817,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/48a50d9148674fc19013aa1169ed21bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/48a50d9148674fc19013aa1169ed21bc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7fe02699d2d54dcdbfdf6e775ceabbeb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7fe02699d2d54dcdbfdf6e775ceabbeb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10829,7 +10835,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c0d424ba5c6c4f75a37b8c6efd82959f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c0d424ba5c6c4f75a37b8c6efd82959f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f112281765f54d8cab88df9d9d0d654e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f112281765f54d8cab88df9d9d0d654e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10847,7 +10853,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d16f56a09114fbfb9eea3576edd8f2b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d16f56a09114fbfb9eea3576edd8f2b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f0fff35eaca34321b46e43216b29badb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f0fff35eaca34321b46e43216b29badb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10865,7 +10871,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f27bd49c0484444798ca349c9103afdb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f27bd49c0484444798ca349c9103afdb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c47fc69ca2214bb18f6460c3157658fa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c47fc69ca2214bb18f6460c3157658fa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10883,7 +10889,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/18c997e5bcb4436680f9e2d836d3d103-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/18c997e5bcb4436680f9e2d836d3d103-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/859849b2103d44f9b3e9754935d2e13a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/859849b2103d44f9b3e9754935d2e13a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10932,7 +10938,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1a9aac7276da4b6a9cb7ef77bec18def-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1a9aac7276da4b6a9cb7ef77bec18def-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d8bcfabb76584b119cb6f4f66fd60372-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d8bcfabb76584b119cb6f4f66fd60372-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -10981,7 +10987,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/716b0519773f4b7787a73f1870ed258e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/716b0519773f4b7787a73f1870ed258e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/72c9d4eaae104eb6b88af35bb96a700e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/72c9d4eaae104eb6b88af35bb96a700e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11029,7 +11035,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d5bf37295910451fbcc65682f33d3754-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d5bf37295910451fbcc65682f33d3754-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8310befed33e4dee9000e34bddaeed08-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8310befed33e4dee9000e34bddaeed08-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11047,7 +11053,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a9e66a8a7ba3400ba8918bdc8573409a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a9e66a8a7ba3400ba8918bdc8573409a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4555c61a2d264a8e88d9aa0b6ecd711f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4555c61a2d264a8e88d9aa0b6ecd711f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11065,7 +11071,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fdb8665ec5004df29cc7d9e98e467ecc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fdb8665ec5004df29cc7d9e98e467ecc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7009dc68041743b1a5b7eaaa108766ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7009dc68041743b1a5b7eaaa108766ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11083,7 +11089,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef454b9260044b33a344ee8abb50afe4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef454b9260044b33a344ee8abb50afe4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0e1b401493294a88bb4dd731203ec44b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0e1b401493294a88bb4dd731203ec44b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11101,7 +11107,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f9c358012a74bb3afa0fc6a8097cfcf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f9c358012a74bb3afa0fc6a8097cfcf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2d67590dcc5417aab66edbc1b5bf0ce-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2d67590dcc5417aab66edbc1b5bf0ce-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11119,7 +11125,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef02db4966be4fd1ad33d30d8e264759-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef02db4966be4fd1ad33d30d8e264759-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f91ff276f992445d93d53725efba3da6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f91ff276f992445d93d53725efba3da6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11137,7 +11143,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9f34769e12be4bd084474d6e28547e7b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9f34769e12be4bd084474d6e28547e7b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2d25afc8c82045b3abd98cb97bbb554b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2d25afc8c82045b3abd98cb97bbb554b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11155,7 +11161,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/876589a672ef4fb6943983ce7e99c59a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/876589a672ef4fb6943983ce7e99c59a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ec7278b60f7a4b84974404fbdc253926-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ec7278b60f7a4b84974404fbdc253926-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11204,7 +11210,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5078348e0fb4960884b6333cd7ae36f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5078348e0fb4960884b6333cd7ae36f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f2c09d8abcf4402f88d9b2b1795d6b67-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f2c09d8abcf4402f88d9b2b1795d6b67-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11222,7 +11228,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef8082b835a94a72a00e15081418444d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef8082b835a94a72a00e15081418444d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/12802603548a4deeafd64440216da320-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/12802603548a4deeafd64440216da320-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11240,7 +11246,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0523a70a74104ce28d5b21d3a4d5ca63-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0523a70a74104ce28d5b21d3a4d5ca63-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/77e042546d9f4acc80bbe0ffd4307b45-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/77e042546d9f4acc80bbe0ffd4307b45-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11258,7 +11264,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33524d9119d24326ae2d15790a578751-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33524d9119d24326ae2d15790a578751-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2d45f0dacbcb49d394006817ae7a1dff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2d45f0dacbcb49d394006817ae7a1dff-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11276,7 +11282,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/30735db5cdc744dd95f36a2b49280433-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/30735db5cdc744dd95f36a2b49280433-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c16395f9d2004eeeac600e96fe4d9ebb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c16395f9d2004eeeac600e96fe4d9ebb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11294,7 +11300,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9574fbfd8e494e5693cdfe03d1fdb583-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9574fbfd8e494e5693cdfe03d1fdb583-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/18f5ecadd14d4759a0e709f325547a3b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/18f5ecadd14d4759a0e709f325547a3b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11312,7 +11318,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a2dc80ebc1904bd1a2c181d561140835-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a2dc80ebc1904bd1a2c181d561140835-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ac97b6c2d7ca48349504d656ee6430a3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ac97b6c2d7ca48349504d656ee6430a3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11330,7 +11336,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b641bf5771e946ecb616c27d9799d0e2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b641bf5771e946ecb616c27d9799d0e2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e72bc657ba734ea29328cfc28ccdf4bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e72bc657ba734ea29328cfc28ccdf4bc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11348,7 +11354,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1db750f8857d4f8fbe411e25bf142df9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1db750f8857d4f8fbe411e25bf142df9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/35fd258e1c0d4a5d86dc2c2d3e549a2e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/35fd258e1c0d4a5d86dc2c2d3e549a2e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11366,7 +11372,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/84c59afc21104cb0a5075fc031d58788-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/84c59afc21104cb0a5075fc031d58788-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a49bad8a8f8a4c6cbb11ae1d9a053d00-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a49bad8a8f8a4c6cbb11ae1d9a053d00-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11384,7 +11390,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/efea0361546549c5a1cd4c37d6447c03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/efea0361546549c5a1cd4c37d6447c03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00bccbcf104744728b6ff64cbecf3f73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00bccbcf104744728b6ff64cbecf3f73-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11402,7 +11408,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cfec16e283504029a80ad7f176da52d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cfec16e283504029a80ad7f176da52d6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3dfb2a96f874ac1bf0eb90e87f448d1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3dfb2a96f874ac1bf0eb90e87f448d1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11420,7 +11426,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/80541171ed45477fbbd156690dbaed4f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/80541171ed45477fbbd156690dbaed4f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/93d5fcb4e2b24f72b2fa2680426068ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/93d5fcb4e2b24f72b2fa2680426068ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11438,7 +11444,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4d47ff7d250340d39c75164cd72eb143-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4d47ff7d250340d39c75164cd72eb143-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/54973a2e21c7497ea40d2cca9d2bb143-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/54973a2e21c7497ea40d2cca9d2bb143-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11456,7 +11462,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b16a538014946649fa28bb7af57e650-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b16a538014946649fa28bb7af57e650-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9359be6fc60e4f20b70aecc2d4f194cb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9359be6fc60e4f20b70aecc2d4f194cb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11474,7 +11480,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/caef72c53c9a439d83c92a23f344a8b4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/caef72c53c9a439d83c92a23f344a8b4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/59cacf57df4f4c44974725df759fc6fd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/59cacf57df4f4c44974725df759fc6fd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11492,7 +11498,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bedd473426a744bd9cc50b926b3edd96-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bedd473426a744bd9cc50b926b3edd96-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ccb8130c7c434b2e87d8aaa2328678f6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ccb8130c7c434b2e87d8aaa2328678f6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11510,7 +11516,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/176e7065c4324afdaba66b70293df0a2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/176e7065c4324afdaba66b70293df0a2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/104c06b310bc4f1683a830dc9664056e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/104c06b310bc4f1683a830dc9664056e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11528,7 +11534,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6bea87bf11974ea7b93f5f07020976b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6bea87bf11974ea7b93f5f07020976b5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eb416d3beb094f67aa251c714d54f25a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eb416d3beb094f67aa251c714d54f25a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11546,7 +11552,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d8d6a3fda9604883afaace131557d3b9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d8d6a3fda9604883afaace131557d3b9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fda2e95ceef64ac8a63b2f198ab65e0c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fda2e95ceef64ac8a63b2f198ab65e0c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11564,7 +11570,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/34891fb98dad46eaa4bc304ea090e967-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/34891fb98dad46eaa4bc304ea090e967-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/38a28c60168a40b3becde685a6b90bc9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/38a28c60168a40b3becde685a6b90bc9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11582,7 +11588,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db5bf2d7c32d491685623e4b9fcf0f37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db5bf2d7c32d491685623e4b9fcf0f37-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/baf9d7c446b443aca96062b367ee5a78-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/baf9d7c446b443aca96062b367ee5a78-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11600,7 +11606,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1904e7eb75bd4c37be4dbd0383991825-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1904e7eb75bd4c37be4dbd0383991825-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/86a1eb142e884fdeb0fde1a1d02220c8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/86a1eb142e884fdeb0fde1a1d02220c8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11618,7 +11624,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/88a32ee402b34e5499ab3eeb699f6c72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/88a32ee402b34e5499ab3eeb699f6c72-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/984ead7ff3504cd5ac8f0b705e9b26f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/984ead7ff3504cd5ac8f0b705e9b26f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11636,7 +11642,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5cffbec502c400a883efa0cc5140224-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5cffbec502c400a883efa0cc5140224-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1f474c725d04d2094c764a3b8cf0a06-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1f474c725d04d2094c764a3b8cf0a06-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11654,7 +11660,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb31378b0f7340a9b7acbefd5b2c47f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb31378b0f7340a9b7acbefd5b2c47f3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/580be3a0a2054b99afbc22af48bfd250-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/580be3a0a2054b99afbc22af48bfd250-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11672,7 +11678,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0c9c5beb06b8463ab77f6f1401573d42-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0c9c5beb06b8463ab77f6f1401573d42-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4ac029bc334644e4b0c66f254ae260c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4ac029bc334644e4b0c66f254ae260c6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11693,7 +11699,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8885f819c7a45abbd35fd3faf94e6c7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8885f819c7a45abbd35fd3faf94e6c7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5d9c2f46400e476db3146785c9b9bedf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5d9c2f46400e476db3146785c9b9bedf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11742,7 +11748,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/df369e666592471fa63279c5df486065-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/df369e666592471fa63279c5df486065-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dad4b69e85aa40e7a12f43491db63b36-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dad4b69e85aa40e7a12f43491db63b36-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11791,7 +11797,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3d13792018f44329e4265cbef76f7d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3d13792018f44329e4265cbef76f7d0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1063c8f323c845089fc60f4ca944a6ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1063c8f323c845089fc60f4ca944a6ee-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11812,7 +11818,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c8a0a0218ee741858deae18e062677a0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c8a0a0218ee741858deae18e062677a0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ac14e8ff46e343929dd9e5cb00e5ccd6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ac14e8ff46e343929dd9e5cb00e5ccd6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11830,7 +11836,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5d7a4378a1cb4060b5d1851c2138fcb3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5d7a4378a1cb4060b5d1851c2138fcb3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b2a4bedbf0049979439ea636471bdfd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b2a4bedbf0049979439ea636471bdfd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11848,7 +11854,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3a9761959b024d93a1022d508028aa5b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3a9761959b024d93a1022d508028aa5b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fbbdff2fbf34401b9dcaad34ef3b4c2d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fbbdff2fbf34401b9dcaad34ef3b4c2d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11866,7 +11872,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/18ebe6ee67ea403b9065f948bf35401e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/18ebe6ee67ea403b9065f948bf35401e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2a0c064167547028221a5a72662e3db-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2a0c064167547028221a5a72662e3db-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11884,7 +11890,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/280663344ac3486c9d6fd6b376a0b9f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/280663344ac3486c9d6fd6b376a0b9f1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/07cd065f801844878401445352025f2e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/07cd065f801844878401445352025f2e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11902,7 +11908,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc9aacbd2f4140c39183c0b86ea86a3a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc9aacbd2f4140c39183c0b86ea86a3a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/538b10ad39814c7d8373e00caabb2654-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/538b10ad39814c7d8373e00caabb2654-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11920,7 +11926,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7af302fea63c48cfbe2e180c3661ea9a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7af302fea63c48cfbe2e180c3661ea9a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/71267ec8940a450987a124d2826f6e3c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/71267ec8940a450987a124d2826f6e3c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11938,7 +11944,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c5110ffc97894c828c540fb78b619c71-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c5110ffc97894c828c540fb78b619c71-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4a3cab4056c748c784260d7df55afe8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4a3cab4056c748c784260d7df55afe8c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -11987,7 +11993,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7b8390647bc9407ca834cd9302b06916-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7b8390647bc9407ca834cd9302b06916-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/58ee79e6d360455795d85b1fc9ce8f29-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/58ee79e6d360455795d85b1fc9ce8f29-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12035,7 +12041,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e456e6a21c0945b0a2a224410aa503ab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e456e6a21c0945b0a2a224410aa503ab-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/99a594c51a374ac8ba914fdb101ea703-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/99a594c51a374ac8ba914fdb101ea703-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12053,7 +12059,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/03a4839fba964a56a24165e703bc227d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/03a4839fba964a56a24165e703bc227d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d53e3a0e1c2345b496406969fbd005d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d53e3a0e1c2345b496406969fbd005d8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12071,7 +12077,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b170bfabdcbb41bca49fc337184c195d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b170bfabdcbb41bca49fc337184c195d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/83da5168feae4212901a7c6b143c538b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/83da5168feae4212901a7c6b143c538b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12089,7 +12095,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2905095f28714181bbb4fb2f06f4cf22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2905095f28714181bbb4fb2f06f4cf22-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/44043a4f97af47159f86ca4f5d4fd1f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/44043a4f97af47159f86ca4f5d4fd1f5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12107,7 +12113,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/421e7a163882404ca77c3d85bc829b6c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/421e7a163882404ca77c3d85bc829b6c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1879502e3b74cdabfafbfbc96a08d71-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1879502e3b74cdabfafbfbc96a08d71-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12125,7 +12131,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a58f7991195b4b9c9ce0da0645e8c766-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a58f7991195b4b9c9ce0da0645e8c766-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d2b7367d27c84efcaa66c597751407f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d2b7367d27c84efcaa66c597751407f5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12143,7 +12149,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0451fae6e2234e2581b2d50cef745567-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0451fae6e2234e2581b2d50cef745567-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db01ff15ba4144c5965f6f075890d026-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db01ff15ba4144c5965f6f075890d026-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12161,7 +12167,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4b1151aa6cf74fb1930d726da3f00b03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4b1151aa6cf74fb1930d726da3f00b03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b54ae678d96432d96f75c80968b82e9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b54ae678d96432d96f75c80968b82e9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12179,7 +12185,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/91f4c9073e57400488bb28c33714909c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/91f4c9073e57400488bb28c33714909c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3ab62be0d64e4365ac5cc403788d410b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3ab62be0d64e4365ac5cc403788d410b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12197,7 +12203,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33eef064d4a54dd18d31ae84b031e70d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33eef064d4a54dd18d31ae84b031e70d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eadb438bdff34b189f443a4e81fb3c6e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eadb438bdff34b189f443a4e81fb3c6e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12215,7 +12221,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d98c0875d1a545238761741439284500-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d98c0875d1a545238761741439284500-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eeca472836454aa79511373d5c3806dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eeca472836454aa79511373d5c3806dd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12233,7 +12239,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0755a841747947cb82c4591b17b7a780-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0755a841747947cb82c4591b17b7a780-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3187200df80345ab9e97982d32f6dee0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3187200df80345ab9e97982d32f6dee0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12251,7 +12257,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/afacdc1f107c4d3bad7de4099f84b759-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/afacdc1f107c4d3bad7de4099f84b759-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e1a62b0124df44e0bf9ae96f52826e22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e1a62b0124df44e0bf9ae96f52826e22-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12269,7 +12275,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/155f9530f6af4fdbb094cf4e4e2dcba5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/155f9530f6af4fdbb094cf4e4e2dcba5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c238f9db4f9a41ddb30ee0250a6ea643-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c238f9db4f9a41ddb30ee0250a6ea643-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12287,7 +12293,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8bd81461dffd47ba958fc3cf9eb88c1e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8bd81461dffd47ba958fc3cf9eb88c1e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/443c60d71b88487eadb43cfb5e0e38cc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/443c60d71b88487eadb43cfb5e0e38cc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12305,7 +12311,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/edc328e820ab4e51a0b918366e73e9f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/edc328e820ab4e51a0b918366e73e9f3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8118ed10f3c2466d953c694d1e04ee2b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8118ed10f3c2466d953c694d1e04ee2b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12323,7 +12329,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/59ba85f781ef4e0397d4b308d234a311-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/59ba85f781ef4e0397d4b308d234a311-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/01f56c5fa5344efc960f7f2393b0c0be-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/01f56c5fa5344efc960f7f2393b0c0be-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12341,7 +12347,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/258b3c3baf564aae8dfd65a2c0454358-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/258b3c3baf564aae8dfd65a2c0454358-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/49e45067762c4d17a5c8f9a8033a0104-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/49e45067762c4d17a5c8f9a8033a0104-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12359,7 +12365,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/958f2e0a17024488aa3847034487c790-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/958f2e0a17024488aa3847034487c790-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32a93af4e2c348479d0fa3b2b12f7202-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32a93af4e2c348479d0fa3b2b12f7202-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12377,7 +12383,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9de226e46cb44115ad2e8fefd64e1b38-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9de226e46cb44115ad2e8fefd64e1b38-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ce8e9d10cf04eaaaf6bca047a57239d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ce8e9d10cf04eaaaf6bca047a57239d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12395,7 +12401,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/27feb9cfaa5946cfb21c78f101040100-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/27feb9cfaa5946cfb21c78f101040100-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6fde17fbe2c04f79b3b411ae966ec68d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6fde17fbe2c04f79b3b411ae966ec68d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12413,7 +12419,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7273f68370a04332ab60c6584aabecb1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7273f68370a04332ab60c6584aabecb1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4f4959789c8d44628e4a3031f5f6f43b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4f4959789c8d44628e4a3031f5f6f43b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12431,7 +12437,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5659818d410742d7b07261ef3d145c14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5659818d410742d7b07261ef3d145c14-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f5004071df074f0aa13ccda17d0af70b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f5004071df074f0aa13ccda17d0af70b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12449,7 +12455,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c11f914c424448e82fdbbc759021e8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c11f914c424448e82fdbbc759021e8c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/490e449310fb44e395dd84a969995b2a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/490e449310fb44e395dd84a969995b2a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12467,7 +12473,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/385d73dde12b478282b5e38a4c3b64eb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/385d73dde12b478282b5e38a4c3b64eb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c56844d15ed2418e9efb37f6a797a3ff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c56844d15ed2418e9efb37f6a797a3ff-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12485,7 +12491,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f940bfbb519d4c3aa01bde3f383770d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f940bfbb519d4c3aa01bde3f383770d9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9b4595610de4185852f31a9bf44884d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9b4595610de4185852f31a9bf44884d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12503,7 +12509,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b2b080d2ce8a496da759adeaa39cd71e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b2b080d2ce8a496da759adeaa39cd71e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/926717364eea40cc859b1aea48fb6b12-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/926717364eea40cc859b1aea48fb6b12-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12521,7 +12527,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d14af1347b6e4ce68e95f4e71a73b6c3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d14af1347b6e4ce68e95f4e71a73b6c3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/03e9d7e550084f85bf2ae17da50386e8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/03e9d7e550084f85bf2ae17da50386e8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12539,7 +12545,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f24c7a53a944922b327019eca0dbecb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f24c7a53a944922b327019eca0dbecb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/62f6374afa7746a9bfacec3d35616372-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/62f6374afa7746a9bfacec3d35616372-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12557,7 +12563,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/251542d4a1a54507bc715a9a16492a14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/251542d4a1a54507bc715a9a16492a14-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7cbd5d5716e34b02b8a909689474e7f7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7cbd5d5716e34b02b8a909689474e7f7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12575,7 +12581,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d68243cc1e9741de836b0256b5544e47-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d68243cc1e9741de836b0256b5544e47-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d1119de3b28c4e5e870cd164e9baa76d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d1119de3b28c4e5e870cd164e9baa76d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12593,7 +12599,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53510aaa111440999cb8a3dee1078b08-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53510aaa111440999cb8a3dee1078b08-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9f3c8477e69f42b4b79f36fdcad809cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9f3c8477e69f42b4b79f36fdcad809cd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12614,7 +12620,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c5c28704e9914df78742a850543d52fc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c5c28704e9914df78742a850543d52fc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/804f653c822246e693d999f792f5b45f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/804f653c822246e693d999f792f5b45f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12632,7 +12638,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dcaf19fd7ea84d5ca732bcac526e36bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dcaf19fd7ea84d5ca732bcac526e36bb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/41b4d11389fd4d9c8ffae553b6fc207c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/41b4d11389fd4d9c8ffae553b6fc207c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12650,7 +12656,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5f1f55f76ce543bea1dd6edb278d0ed0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5f1f55f76ce543bea1dd6edb278d0ed0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d2a2d6b95be54d4a82ae852c0878bf72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d2a2d6b95be54d4a82ae852c0878bf72-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12668,7 +12674,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f00f762558134e9ea6990bdba25fbb07-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f00f762558134e9ea6990bdba25fbb07-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d149392e6104a5b9611ee2021c166ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d149392e6104a5b9611ee2021c166ca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12686,7 +12692,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c7c7be69ff84b209fc1a0d625b4595a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c7c7be69ff84b209fc1a0d625b4595a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8b67eb6895cc4733888380b43667a9f8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8b67eb6895cc4733888380b43667a9f8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12704,7 +12710,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fed439bc88a046eaa5aff7f18b7c685b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fed439bc88a046eaa5aff7f18b7c685b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/082c35749e4544e985b913a2526ecf23-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/082c35749e4544e985b913a2526ecf23-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12722,7 +12728,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78399a0049834f06a2ac77dec3b72546-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78399a0049834f06a2ac77dec3b72546-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d4fcdd9560a4c85a7a9e66b24d694ce-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d4fcdd9560a4c85a7a9e66b24d694ce-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12740,7 +12746,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3ae77b1565c4d1dba706c18d0f56deb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3ae77b1565c4d1dba706c18d0f56deb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7818e6f19a814bcf90702182fce9d34b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7818e6f19a814bcf90702182fce9d34b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12758,7 +12764,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba2882d2c0b24aaf8b6116b665f508d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba2882d2c0b24aaf8b6116b665f508d9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2208a25dbd444b5e819ae9039d059966-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2208a25dbd444b5e819ae9039d059966-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12776,7 +12782,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6233d27fa67348f7856ee6261ba4c03c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6233d27fa67348f7856ee6261ba4c03c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8551fc06877f4b049f771295fa4d30c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8551fc06877f4b049f771295fa4d30c9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12794,7 +12800,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3ab44e4ff5a8464c9769c41c9476d965-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3ab44e4ff5a8464c9769c41c9476d965-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a27a292835549668fe09772ffa17a65-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a27a292835549668fe09772ffa17a65-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12812,7 +12818,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/85ec334c754e475f92afe8c14c945bb7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/85ec334c754e475f92afe8c14c945bb7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4e3d8d39dfee4ba5aca48e199631cfe9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4e3d8d39dfee4ba5aca48e199631cfe9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12830,7 +12836,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2844cd6ddb7948bfbdf56a0e8f61be4b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2844cd6ddb7948bfbdf56a0e8f61be4b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f268eebf7d14a739a6732b28672fede-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f268eebf7d14a739a6732b28672fede-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12848,7 +12854,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/165ea30c4f45414ab69ef486473fe706-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/165ea30c4f45414ab69ef486473fe706-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13f0b948429a45879ef59461fe761a73-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13f0b948429a45879ef59461fe761a73-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12866,7 +12872,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5342bdfbf92e4f0caa1fff70dd10c980-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5342bdfbf92e4f0caa1fff70dd10c980-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9afd65e7c06544768d05aac573d3d3e1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9afd65e7c06544768d05aac573d3d3e1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12884,7 +12890,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ceaf1af839f84b5f83f8b030fb97ce52-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ceaf1af839f84b5f83f8b030fb97ce52-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c35ebe16d7b747a5a36c662deea109a3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c35ebe16d7b747a5a36c662deea109a3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12902,7 +12908,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d8af28536394290bbdfb8b55113f717-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d8af28536394290bbdfb8b55113f717-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1cac426e23664aa289d906f6c7ec00c4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1cac426e23664aa289d906f6c7ec00c4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12920,7 +12926,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/46454fc9df3d445aa71f6bdec19bf04e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/46454fc9df3d445aa71f6bdec19bf04e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fa3c7f95a0184d35911a3d697daf471f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fa3c7f95a0184d35911a3d697daf471f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12938,7 +12944,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b6cbb2109b9744f3a897810e0813d7e3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b6cbb2109b9744f3a897810e0813d7e3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/07b413ddfe4a43c88753e9e81ce274a6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/07b413ddfe4a43c88753e9e81ce274a6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12956,7 +12962,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9638f56389064745afd530372051a806-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9638f56389064745afd530372051a806-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/baee24d9151b432a83a7bb22963ef5d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/baee24d9151b432a83a7bb22963ef5d0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12974,7 +12980,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8d885b028a37405585b36e4b3eae8cb2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8d885b028a37405585b36e4b3eae8cb2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/432a7ae5002945929c2b2546f1e74769-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/432a7ae5002945929c2b2546f1e74769-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -12992,7 +12998,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9e0e4c761bc64ab5a518e8f5d0e96451-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9e0e4c761bc64ab5a518e8f5d0e96451-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f501dc85c8b546b3ba20fd3f1cf096e5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f501dc85c8b546b3ba20fd3f1cf096e5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13010,7 +13016,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0115f56974494456bb2ac8207e7b7921-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0115f56974494456bb2ac8207e7b7921-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3ea487dca89349ea861163440961015a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3ea487dca89349ea861163440961015a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13028,7 +13034,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9840aed344ca4eb0a386e3b5155ed77c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9840aed344ca4eb0a386e3b5155ed77c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bcd4d9235eb44527a1e9022f35480b22-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bcd4d9235eb44527a1e9022f35480b22-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13046,7 +13052,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5ba2f3c397fa40079612ce69c2cc7a2c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5ba2f3c397fa40079612ce69c2cc7a2c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/48d3d1438256470caf7dbd50f5093ce1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/48d3d1438256470caf7dbd50f5093ce1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13064,7 +13070,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c25662db607c4f0e904040dd7c52ae19-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c25662db607c4f0e904040dd7c52ae19-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d208bd5edcd74d4194e114321cd000b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d208bd5edcd74d4194e114321cd000b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13082,7 +13088,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/de991ec9525a4df7b454fc244e371494-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/de991ec9525a4df7b454fc244e371494-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4eb93adb327748da9f5aff8a6a79e8ff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4eb93adb327748da9f5aff8a6a79e8ff-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13100,7 +13106,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/91aa72bca5c1436e89b8d1f8c3d19185-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/91aa72bca5c1436e89b8d1f8c3d19185-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8c6ca2f03fcb43799b8896bd2a6eb938-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8c6ca2f03fcb43799b8896bd2a6eb938-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13118,7 +13124,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/40517b8e997f4260899ff48204098896-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/40517b8e997f4260899ff48204098896-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e031f9e799541cc9530e8fedc8120c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e031f9e799541cc9530e8fedc8120c0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13136,7 +13142,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d71511b641e44d6e95e8d6f1ce1afd3a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d71511b641e44d6e95e8d6f1ce1afd3a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78d93c93a82845fba7430fe54989afb6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78d93c93a82845fba7430fe54989afb6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13154,7 +13160,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/afacdc565c504a7abb8da7f09d55382c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/afacdc565c504a7abb8da7f09d55382c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e9895a278cd43a9a6add95139050a5e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e9895a278cd43a9a6add95139050a5e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13172,7 +13178,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a6650d2b4ee94e9d9013cd24f1ac1686-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a6650d2b4ee94e9d9013cd24f1ac1686-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f6d425f2e604b35b8bd3d43a271b0c3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f6d425f2e604b35b8bd3d43a271b0c3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13190,7 +13196,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8292886c4b684845bfba23989b3d4bb6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8292886c4b684845bfba23989b3d4bb6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bf131c28ca164ab1beccbce245e6115a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bf131c28ca164ab1beccbce245e6115a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13208,7 +13214,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fdb32ed8235e44b0905a9432cf1909d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fdb32ed8235e44b0905a9432cf1909d3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/06ad1e4247ba4337aae026793b4540ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/06ad1e4247ba4337aae026793b4540ed-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13226,7 +13232,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/122cf1b5b374454089d18fc3f04a3b1c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/122cf1b5b374454089d18fc3f04a3b1c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b3c7352dcd1d4ab4a78dd977e03699f8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b3c7352dcd1d4ab4a78dd977e03699f8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13244,7 +13250,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68a54869430348109fa2588bbae73b2e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68a54869430348109fa2588bbae73b2e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/23191bb1bd4c497d9bac2ca1c52b1030-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/23191bb1bd4c497d9bac2ca1c52b1030-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13262,7 +13268,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d4def9af2d074a83aa57930f0379ed0f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d4def9af2d074a83aa57930f0379ed0f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/102d18ebd0d043828606545738a315e5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/102d18ebd0d043828606545738a315e5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13280,7 +13286,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/72b54888732a4ae9a8bda410c25e94dd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/72b54888732a4ae9a8bda410c25e94dd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fb84d458ef9049ac9f9849966850cfaf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fb84d458ef9049ac9f9849966850cfaf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13298,7 +13304,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c5ecc77407554653b39005cd1a9f1ce1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c5ecc77407554653b39005cd1a9f1ce1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3490bdab57fc443ca3aa460510730dfd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3490bdab57fc443ca3aa460510730dfd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13316,7 +13322,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e752a4a1c26b4aa0a328f316b3ce5478-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e752a4a1c26b4aa0a328f316b3ce5478-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2587e331bbc147b1b17b8037cb811226-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2587e331bbc147b1b17b8037cb811226-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13334,7 +13340,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8de95fe9c841447283be970f0504ed17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8de95fe9c841447283be970f0504ed17-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/98ae1e1ff8054a35b84a49c289a0fea7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/98ae1e1ff8054a35b84a49c289a0fea7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13352,7 +13358,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cbcdb2a7d2a9477eb600ef0faa309fd7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cbcdb2a7d2a9477eb600ef0faa309fd7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/92357968af884312bed720fd5c1b4886-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/92357968af884312bed720fd5c1b4886-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13370,7 +13376,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b39b388d96eb4e059d2520eb8c58def0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b39b388d96eb4e059d2520eb8c58def0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/401e65b7c8304c2ead6491dc0afa31cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/401e65b7c8304c2ead6491dc0afa31cd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13388,7 +13394,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/faabb1ac19454b1d85803d4eea7041bd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/faabb1ac19454b1d85803d4eea7041bd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/80924546c91b4c88be498e0e60580f71-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/80924546c91b4c88be498e0e60580f71-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13406,7 +13412,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64d88035912d4a7680741828ffcdf123-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64d88035912d4a7680741828ffcdf123-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d855333e0ee94bdfa52ce852a3bdd75b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d855333e0ee94bdfa52ce852a3bdd75b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13424,7 +13430,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/834729508e92470e8931d5e095cb3b81-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/834729508e92470e8931d5e095cb3b81-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee89433a10b041df85e1763c146bc156-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee89433a10b041df85e1763c146bc156-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13442,7 +13448,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75eca8c90b984eaf841ae07ca6ab0a7e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75eca8c90b984eaf841ae07ca6ab0a7e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a67228c606124c55974dd109c7167e5b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a67228c606124c55974dd109c7167e5b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13460,7 +13466,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a5b00a0b16984d3ca1d30475f7527c8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a5b00a0b16984d3ca1d30475f7527c8f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64cf2ec644dd46b88397434ac5cc5484-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64cf2ec644dd46b88397434ac5cc5484-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13478,7 +13484,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fa1710a646b04d3081ce23ef16b64b37-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fa1710a646b04d3081ce23ef16b64b37-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f95bf16495fd44cb905663dad97bfa8d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f95bf16495fd44cb905663dad97bfa8d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13496,7 +13502,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8bd171a70bdd4e1bbb249172e9cfa16e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8bd171a70bdd4e1bbb249172e9cfa16e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e59b57f378f543f59fea9b7b72217680-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e59b57f378f543f59fea9b7b72217680-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13514,7 +13520,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3b042ab9f4c4410190ea208489e94343-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3b042ab9f4c4410190ea208489e94343-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8405f91b55b04303a69b3718d27197ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8405f91b55b04303a69b3718d27197ef-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13532,7 +13538,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/93dde657b8dd488687ae5e4db87f35ce-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/93dde657b8dd488687ae5e4db87f35ce-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e9f1dcd27dfa4c3aa146c60da71456d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e9f1dcd27dfa4c3aa146c60da71456d8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13550,7 +13556,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9011819dbd0f427ba0d084e1f4500b4b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9011819dbd0f427ba0d084e1f4500b4b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c520177aa4c94880938941400b3d2bf5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c520177aa4c94880938941400b3d2bf5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13568,7 +13574,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ca63fd870dc042159d45f0f151297e8b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ca63fd870dc042159d45f0f151297e8b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3fcabff0c325435880759bb02a877e42-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3fcabff0c325435880759bb02a877e42-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13586,7 +13592,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b5c6e436fd994206899df8fd1ab70df7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b5c6e436fd994206899df8fd1ab70df7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1232ca85b5c848328018e77ac65ff539-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1232ca85b5c848328018e77ac65ff539-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13604,7 +13610,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/98fa4bfe7b664a7796678213be4b9a35-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/98fa4bfe7b664a7796678213be4b9a35-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9a3ede95eda249dd84a202d7ec85852f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9a3ede95eda249dd84a202d7ec85852f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13622,7 +13628,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d1de1255bfea4504a16fe37725836567-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d1de1255bfea4504a16fe37725836567-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1bd836472da748c5a755f3ec2fb3c60b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1bd836472da748c5a755f3ec2fb3c60b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13640,7 +13646,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5914fcbf1ba94f738de17669f953d41f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5914fcbf1ba94f738de17669f953d41f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3dc4d9fcbabf4c988fd0e6ee0c87babf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3dc4d9fcbabf4c988fd0e6ee0c87babf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13658,7 +13664,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8db2317e1254512848874d38e15811d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8db2317e1254512848874d38e15811d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/aaf4368fd5a642ad8d33a071a944e577-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/aaf4368fd5a642ad8d33a071a944e577-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13676,7 +13682,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6dde0156dbfd45e2b91ebe6184fe1768-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6dde0156dbfd45e2b91ebe6184fe1768-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a9385a961548424fa5aedf074efd5f74-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a9385a961548424fa5aedf074efd5f74-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13694,7 +13700,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b109f803b7ed4991ba758a6971fc96aa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b109f803b7ed4991ba758a6971fc96aa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1a529496dde44b7b90a2283a4c61349f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1a529496dde44b7b90a2283a4c61349f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13712,7 +13718,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8d09606f1671425f93d0480bdb49a6a9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8d09606f1671425f93d0480bdb49a6a9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8aa5eebc9c8243eba9e19c930e3efe71-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8aa5eebc9c8243eba9e19c930e3efe71-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13730,7 +13736,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/610334e0892345c3bb77c19f929bb950-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/610334e0892345c3bb77c19f929bb950-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cbebc6dbdb84d7f9e1d729600e2e083-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cbebc6dbdb84d7f9e1d729600e2e083-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13748,7 +13754,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d6500c1ac75b47eb8f4c7181052fac0e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d6500c1ac75b47eb8f4c7181052fac0e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c8f964fbecd41dea79cd6fd4424d67c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c8f964fbecd41dea79cd6fd4424d67c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13768,7 +13774,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4b6155696e58406fac42df42168d6c42-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4b6155696e58406fac42df42168d6c42-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2013fd805d3491a8715b95a82313eea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2013fd805d3491a8715b95a82313eea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13852,7 +13858,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b7ca7ea2fbd24262857104a2d0bf0b9b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b7ca7ea2fbd24262857104a2d0bf0b9b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a21f5d5997d545499d6a3c8d4a6e052d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a21f5d5997d545499d6a3c8d4a6e052d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13901,7 +13907,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5c73b50e6b8349779b2a55a8edb40b3f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5c73b50e6b8349779b2a55a8edb40b3f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/56a7525ad7e24fe7af576c1055da502b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/56a7525ad7e24fe7af576c1055da502b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13950,7 +13956,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5b6d8c2ec2654301bf4b58d37293b89d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5b6d8c2ec2654301bf4b58d37293b89d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c82fad999fc4a209e5e157ab74251d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c82fad999fc4a209e5e157ab74251d8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13972,7 +13978,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b81d0404c630490888f9cd3adb95a0fb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b81d0404c630490888f9cd3adb95a0fb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d251c171a6a14cffb8bd5e5b77f4b014-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d251c171a6a14cffb8bd5e5b77f4b014-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -13993,7 +13999,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d0227a3dbfb44fd9e0fb861f6386006-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d0227a3dbfb44fd9e0fb861f6386006-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6af282f970fe4bfcbf1314459e4d70e4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6af282f970fe4bfcbf1314459e4d70e4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14011,7 +14017,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e2ceadfa71ff4876b5d1577d1de4680c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e2ceadfa71ff4876b5d1577d1de4680c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9477a0769c524492be7afb1ce7e99efa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9477a0769c524492be7afb1ce7e99efa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14029,7 +14035,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/49f23abcebda437ca6459ad0e39f7db0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/49f23abcebda437ca6459ad0e39f7db0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e003e5699b2944dd9fc21c287439d560-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e003e5699b2944dd9fc21c287439d560-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14047,7 +14053,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/11a85464ac544b3fa95bf6869c28d462-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/11a85464ac544b3fa95bf6869c28d462-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/06c39d6b698f4d4582f4518588b26543-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/06c39d6b698f4d4582f4518588b26543-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14065,7 +14071,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b29f60e5b41d41bb85355066164d9144-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b29f60e5b41d41bb85355066164d9144-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc8d126f332440ab91da7290c51b9fca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc8d126f332440ab91da7290c51b9fca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14083,7 +14089,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8459e6126681412f82bd3e91e4b08d91-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8459e6126681412f82bd3e91e4b08d91-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/259eb4d4424440749205cfc8a9e9ba6a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/259eb4d4424440749205cfc8a9e9ba6a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14101,7 +14107,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d62039fee0c4605a87b89eca093f608-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d62039fee0c4605a87b89eca093f608-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/58456739b7df4c9a8ab3bcd68d063e0f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/58456739b7df4c9a8ab3bcd68d063e0f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14185,7 +14191,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/90690c15de7647dbb4210af339d3edc2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/90690c15de7647dbb4210af339d3edc2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0861ba5f682548339e12a915a599968b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0861ba5f682548339e12a915a599968b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14208,7 +14214,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5cc7773e099f4edd98876ee43ce6744d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5cc7773e099f4edd98876ee43ce6744d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/12a0639f07374c9f9bc2e78934e848ca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/12a0639f07374c9f9bc2e78934e848ca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14231,7 +14237,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b9aabee0f3764c6d8498325ee087f645-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b9aabee0f3764c6d8498325ee087f645-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/74b570369d6c497495217ddce2e72855-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/74b570369d6c497495217ddce2e72855-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14252,7 +14258,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb3a4b3e07b44fec8ccd184df50c4c3e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb3a4b3e07b44fec8ccd184df50c4c3e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/413b9f87e4cd4659b125250738c60ce4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/413b9f87e4cd4659b125250738c60ce4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14270,7 +14276,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/10363b71453c4067983bef2dcd8957be-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/10363b71453c4067983bef2dcd8957be-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dab02360a5cf40c8893fe5db75a2d1a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dab02360a5cf40c8893fe5db75a2d1a4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14288,7 +14294,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae13813cf9494112adf6166665d21ad4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae13813cf9494112adf6166665d21ad4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dc5a2beb1d114e148a881966fec28ed1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dc5a2beb1d114e148a881966fec28ed1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14306,7 +14312,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b6cc84328a04371be94c32c99e72494-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b6cc84328a04371be94c32c99e72494-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/676a7629f0ca454ba2b6858e39db1c7f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/676a7629f0ca454ba2b6858e39db1c7f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14324,7 +14330,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5fefb2844328470da05f53c0ea783bca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5fefb2844328470da05f53c0ea783bca-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/df5e3e8b803e42a1b83dcac661e7dc54-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/df5e3e8b803e42a1b83dcac661e7dc54-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14342,7 +14348,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dfd90e3279f540d5890363e9de23d4da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dfd90e3279f540d5890363e9de23d4da-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/978c0a91fcdd46f4acbc2ed5ba1efed0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/978c0a91fcdd46f4acbc2ed5ba1efed0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14360,7 +14366,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bbc3199e7b794815ac82d6d00b4f2a86-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bbc3199e7b794815ac82d6d00b4f2a86-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c92f9d4707274aef9ecc0eb173220e9c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c92f9d4707274aef9ecc0eb173220e9c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14378,7 +14384,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1007cd96b1e94679aa95fa6036fcc606-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1007cd96b1e94679aa95fa6036fcc606-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f4bda8c3a6b74adf887fe0a9aeee4f8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f4bda8c3a6b74adf887fe0a9aeee4f8c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14427,7 +14433,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24cfe2c1906a4a87a5344423863ca859-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24cfe2c1906a4a87a5344423863ca859-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/014323372ec446cf971b0e39be4d640c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/014323372ec446cf971b0e39be4d640c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14475,7 +14481,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3a28df6eb3e2485799a6a069421682a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3a28df6eb3e2485799a6a069421682a8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/faaadf1426204ad4a2c4444c0d10ae79-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/faaadf1426204ad4a2c4444c0d10ae79-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14493,7 +14499,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6141a35f007a468dbf31b4fe70de2b58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6141a35f007a468dbf31b4fe70de2b58-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ecf5b113dff446dcaf89c382620a160f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ecf5b113dff446dcaf89c382620a160f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14511,7 +14517,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc16e9264e6f46ffbb007de453ba6343-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc16e9264e6f46ffbb007de453ba6343-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a33691c55aed49f1954af6afa18a2677-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a33691c55aed49f1954af6afa18a2677-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14529,7 +14535,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bdd763ccb164499285fe86f241ca1147-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bdd763ccb164499285fe86f241ca1147-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3424a89b31c14e408665a442d2571c76-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3424a89b31c14e408665a442d2571c76-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14547,7 +14553,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/48cb57312cdf455e9f848aacb020a92e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/48cb57312cdf455e9f848aacb020a92e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ecfe1d4abf6143278c57f8b35015b191-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ecfe1d4abf6143278c57f8b35015b191-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14565,7 +14571,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/71b9b49269804760a8edc57177645bd1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/71b9b49269804760a8edc57177645bd1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5684be54424546ab81dd5098249773a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5684be54424546ab81dd5098249773a4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14583,7 +14589,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bf3781b095c544958bdb2e395dd30a8a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bf3781b095c544958bdb2e395dd30a8a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b1c3080c9bfd4f75b84607fb01c2852a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b1c3080c9bfd4f75b84607fb01c2852a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14601,7 +14607,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/15087c3bda4f4682830cb463c584ceb9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/15087c3bda4f4682830cb463c584ceb9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ecc5b6b940464aa49bcba3826cb886da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ecc5b6b940464aa49bcba3826cb886da-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14619,7 +14625,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ceb44239de5a430f902e9a329bc46740-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ceb44239de5a430f902e9a329bc46740-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/83ce0c891f8146e9a9fdb93455651b5e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/83ce0c891f8146e9a9fdb93455651b5e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14637,7 +14643,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/070a3cca3c7346f6975002368e66dcbc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/070a3cca3c7346f6975002368e66dcbc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/392153baf3364935b853d2582495c231-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/392153baf3364935b853d2582495c231-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14655,7 +14661,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d13ed497210743a092dd05b7f853bca6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d13ed497210743a092dd05b7f853bca6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0922cf70a34f4cf2b8887d7839a2eef9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0922cf70a34f4cf2b8887d7839a2eef9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14673,7 +14679,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6447463cdc0347689d4e6283102ade2b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6447463cdc0347689d4e6283102ade2b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/da4d070aab5947658e18a073effe7822-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/da4d070aab5947658e18a073effe7822-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14691,7 +14697,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1b26f1136e6d44018e0e8cab5c640d83-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1b26f1136e6d44018e0e8cab5c640d83-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/27456aee31c5411da91c1fedecf63992-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/27456aee31c5411da91c1fedecf63992-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14709,7 +14715,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7f00e722d729478197f3374519296759-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7f00e722d729478197f3374519296759-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cae0b0f438ea44a6bc6c4d2739e35b25-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cae0b0f438ea44a6bc6c4d2739e35b25-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14727,7 +14733,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/926ffd03ce2944e7922d622d1aee1022-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/926ffd03ce2944e7922d622d1aee1022-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d12cac46983643d6bcaa88d27929e3b0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d12cac46983643d6bcaa88d27929e3b0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14745,7 +14751,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cb4e6139f9ac489496d7323f080d5f31-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cb4e6139f9ac489496d7323f080d5f31-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e083e86e24494491909f628c5c7034a7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e083e86e24494491909f628c5c7034a7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14763,7 +14769,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5a5bde0d96f44771b357b24c0a1efe8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5a5bde0d96f44771b357b24c0a1efe8f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dea506fd3e2540c88742616690b92f59-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dea506fd3e2540c88742616690b92f59-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14781,7 +14787,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1097f48471f446a9b893ecc714755abd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1097f48471f446a9b893ecc714755abd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ad1d7c49eafa4011b622796bd4e7d938-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ad1d7c49eafa4011b622796bd4e7d938-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14799,7 +14805,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b64eeee294354261b06cc6e7f28043b3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b64eeee294354261b06cc6e7f28043b3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d01a4aa591f4aae87505287cffc02ba-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d01a4aa591f4aae87505287cffc02ba-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14817,7 +14823,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/edc898ab848446b4aefc087321de0c69-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/edc898ab848446b4aefc087321de0c69-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8c00dc9cd17e4e32807f112b7ee30f60-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8c00dc9cd17e4e32807f112b7ee30f60-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14835,7 +14841,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/146d3a1db0be439da7836a52cb9314c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/146d3a1db0be439da7836a52cb9314c9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8853514b831440c193703818e2ecf504-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8853514b831440c193703818e2ecf504-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14853,7 +14859,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c4de9da53e984cdba1fe76b08fe43e66-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c4de9da53e984cdba1fe76b08fe43e66-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e799993c695412d8dfff8abdda13005-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e799993c695412d8dfff8abdda13005-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14871,7 +14877,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/344320a518cb419087901df6e85538cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/344320a518cb419087901df6e85538cf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/762b409ac05e455f934f62aa6a3e5a58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/762b409ac05e455f934f62aa6a3e5a58-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14889,7 +14895,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1efe503399c8438c866f916efd9b4420-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1efe503399c8438c866f916efd9b4420-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/551f621d9fb84ee7bb2547129118cee0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/551f621d9fb84ee7bb2547129118cee0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14907,7 +14913,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c14df202dc04af7935e19c1313cb03d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c14df202dc04af7935e19c1313cb03d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/801ee522fbb84386920dfcca1157f6de-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/801ee522fbb84386920dfcca1157f6de-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14925,7 +14931,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89a4e2db01114f0699a90190b00ee577-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89a4e2db01114f0699a90190b00ee577-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4ee5ab7990a04858be61c8b4019b0a84-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4ee5ab7990a04858be61c8b4019b0a84-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14943,7 +14949,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2797df3b90194aaf96ebe0ee4671cd8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2797df3b90194aaf96ebe0ee4671cd8f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/78aef4b3dcc44deda082d3c71ab9b34d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/78aef4b3dcc44deda082d3c71ab9b34d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14961,7 +14967,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8ee9695a5f7404fb2f0b8ad03d61789-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8ee9695a5f7404fb2f0b8ad03d61789-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/987532e0a41644f1946aa2e2453b5ecc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/987532e0a41644f1946aa2e2453b5ecc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14979,7 +14985,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/95715f4ab2ef47ae9583eb8cdb24cf3b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/95715f4ab2ef47ae9583eb8cdb24cf3b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/17101464163f4a828ba25dcc334c1408-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/17101464163f4a828ba25dcc334c1408-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -14997,7 +15003,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ab2fe16951334b68aabfe12b0eab3600-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ab2fe16951334b68aabfe12b0eab3600-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c690ffc9965a4121bbd743a5a8caafa9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c690ffc9965a4121bbd743a5a8caafa9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15015,7 +15021,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89c779b1a2874919a9ef27c46415c9f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89c779b1a2874919a9ef27c46415c9f5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c535787dd3ae44529f583313a643316a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c535787dd3ae44529f583313a643316a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15033,7 +15039,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/87a0ee0b0a5744919e09e68239eaeb5c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/87a0ee0b0a5744919e09e68239eaeb5c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/833410c694d940f097b1cf0a2e326d32-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/833410c694d940f097b1cf0a2e326d32-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15117,7 +15123,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6f36b725ed4647beb1387083c83ef1f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6f36b725ed4647beb1387083c83ef1f5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/93f66efddf4b4425a94befb8ab06d692-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/93f66efddf4b4425a94befb8ab06d692-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15139,7 +15145,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/80be2bc236894104831f764dd9f0dcdf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/80be2bc236894104831f764dd9f0dcdf-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2f63b2b319d944be8b4fe8e4b4176e89-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2f63b2b319d944be8b4fe8e4b4176e89-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15189,7 +15195,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ceeba03e41a4038a094943dd6ee29f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ceeba03e41a4038a094943dd6ee29f5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/52d3441b1f20453d8c6a08bdc5b4cd96-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/52d3441b1f20453d8c6a08bdc5b4cd96-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15207,7 +15213,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e215f5d09cf6437d9c749a33ce78301c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e215f5d09cf6437d9c749a33ce78301c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/28d49f55692c44e5942af119c1ed4fa4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/28d49f55692c44e5942af119c1ed4fa4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15225,7 +15231,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1349f463584e42b49a5a8034adda011e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1349f463584e42b49a5a8034adda011e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4de2e155ccd24b65a95b4a350488e8e4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4de2e155ccd24b65a95b4a350488e8e4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15243,7 +15249,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3d038c7b72284c0696eadb81ef017eab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3d038c7b72284c0696eadb81ef017eab-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f79e005683a140cfb2c84ba6831592e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f79e005683a140cfb2c84ba6831592e7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15261,7 +15267,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d05ab52aa0504f328e6e0d325989f90a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d05ab52aa0504f328e6e0d325989f90a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75b81a9ff26040588749d3074872c51c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75b81a9ff26040588749d3074872c51c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15279,7 +15285,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c270592943b5483aa6a282ced0a68b7e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c270592943b5483aa6a282ced0a68b7e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8dc7cf7310c743c4927d7cf4cde6cd8e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8dc7cf7310c743c4927d7cf4cde6cd8e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15297,7 +15303,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0bc5d11e09d34dbab9b8a2dfc86614b9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0bc5d11e09d34dbab9b8a2dfc86614b9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/59797ff1e59c46ccba6aa02a592d8870-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/59797ff1e59c46ccba6aa02a592d8870-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15315,7 +15321,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1c2d3eb649154be59fd668252c20700e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1c2d3eb649154be59fd668252c20700e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a82cf352a6db4138b1197e6d19d5c13d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a82cf352a6db4138b1197e6d19d5c13d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15338,7 +15344,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd8e35477dd94a2fbc57c29a85cf02c4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd8e35477dd94a2fbc57c29a85cf02c4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89b7f54cc7e947baa127298eea236a1f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89b7f54cc7e947baa127298eea236a1f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15356,7 +15362,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e674d6a383cf4358b5459040b28e2cf7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e674d6a383cf4358b5459040b28e2cf7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4200dad078734f6c9b9a5903f8a03b0b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4200dad078734f6c9b9a5903f8a03b0b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15374,7 +15380,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c9b5114743344e0ba6693dbe80203f96-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c9b5114743344e0ba6693dbe80203f96-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b05c84bce5f346a1962c73bc2a181cae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b05c84bce5f346a1962c73bc2a181cae-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15392,7 +15398,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0c375c8522684f1881c5aa0f05e4db44-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0c375c8522684f1881c5aa0f05e4db44-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7357f21f3c0343d5986277fb3aa4f48b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7357f21f3c0343d5986277fb3aa4f48b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15410,7 +15416,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c4ed1c68fe8c4934a5420d7520fe508b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c4ed1c68fe8c4934a5420d7520fe508b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fec849cf9dea49ddb7f9afbb039c5dd0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fec849cf9dea49ddb7f9afbb039c5dd0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15428,7 +15434,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/104247e6594f4c11bae14436c333bc8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/104247e6594f4c11bae14436c333bc8c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e650eb9dcae74732882a66727aa48e65-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e650eb9dcae74732882a66727aa48e65-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15446,7 +15452,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1b3199f4aff84334ab07fadbb3b3b7e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1b3199f4aff84334ab07fadbb3b3b7e7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e72f0b6aff7f4af3977f75a31eda8513-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e72f0b6aff7f4af3977f75a31eda8513-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15464,7 +15470,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6e7dbb9b459e4e66883bf45092d8e54e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6e7dbb9b459e4e66883bf45092d8e54e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d1269ce1229c447ea78e9257dc19af61-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d1269ce1229c447ea78e9257dc19af61-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15482,7 +15488,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f4bd31966f3546c69cd8213c4e6836c9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f4bd31966f3546c69cd8213c4e6836c9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/77c48b7531c54f5488ee9338a7649322-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/77c48b7531c54f5488ee9338a7649322-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15500,7 +15506,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9b48b40ab7bc406b9adfd61f89697f28-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9b48b40ab7bc406b9adfd61f89697f28-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1e1c144bbcb941fea3fc4ff8275019c0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1e1c144bbcb941fea3fc4ff8275019c0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15518,7 +15524,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c48ba6ceb1ed40d1b87cb05cd4252b7f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c48ba6ceb1ed40d1b87cb05cd4252b7f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4ee1292bb31c4e5d924685052051cc17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4ee1292bb31c4e5d924685052051cc17-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15536,7 +15542,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/75d7c35f466f436a90a0066b3e8fe48f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/75d7c35f466f436a90a0066b3e8fe48f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64436ec36f3c427987a97bddad735383-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64436ec36f3c427987a97bddad735383-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15554,7 +15560,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6e5516fa324b47a79673d2d43becae0a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6e5516fa324b47a79673d2d43becae0a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2210747cd3a34035b68ec9081d834f7d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2210747cd3a34035b68ec9081d834f7d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15572,7 +15578,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/16a00288835c45fd8d9ff434d2069041-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/16a00288835c45fd8d9ff434d2069041-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/114a6831de174a06aa61d15ae837b010-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/114a6831de174a06aa61d15ae837b010-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15590,7 +15596,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ede680c1b36d40c7bde08e6b9aa1e203-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ede680c1b36d40c7bde08e6b9aa1e203-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24c35fd9105445af8cb6d829e99cc74c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24c35fd9105445af8cb6d829e99cc74c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15608,7 +15614,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c79c6ce4a75b45d6b48746426aac9985-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c79c6ce4a75b45d6b48746426aac9985-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2cc423286f934a5dbe2c9062b14a0cc1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2cc423286f934a5dbe2c9062b14a0cc1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15626,7 +15632,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a13cf87789e64c4cb2ef3dbdf50421b9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a13cf87789e64c4cb2ef3dbdf50421b9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f47bb3199d98437282788e13ea4f84f8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f47bb3199d98437282788e13ea4f84f8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15644,7 +15650,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/02ea11618c8a46409ae45e46621aa22e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/02ea11618c8a46409ae45e46621aa22e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1902bc46f68c423da127a777b5623139-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1902bc46f68c423da127a777b5623139-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15662,7 +15668,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e71ce811dae54212a168369d44d3795d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e71ce811dae54212a168369d44d3795d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5bf1f139199144799de7b23f280cc436-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5bf1f139199144799de7b23f280cc436-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15680,7 +15686,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ad76488995494bae9fdf554351373503-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ad76488995494bae9fdf554351373503-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2cde269795cc4f5a8c93e2974c1be7ec-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2cde269795cc4f5a8c93e2974c1be7ec-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15698,7 +15704,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/289a803df8f14f4b8f441bfdec43b2fd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/289a803df8f14f4b8f441bfdec43b2fd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1e54219c9355423aa3e2cb730df5c751-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1e54219c9355423aa3e2cb730df5c751-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15716,7 +15722,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7a399e3f0b6d47aba8326048884e0493-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7a399e3f0b6d47aba8326048884e0493-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e6633c05110944d3ae58d46783dbbd7c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e6633c05110944d3ae58d46783dbbd7c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15734,7 +15740,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1d649cdfcce54c9dacf107a549710e03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1d649cdfcce54c9dacf107a549710e03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/09861532951940f1bd1f315ed4f54b6e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/09861532951940f1bd1f315ed4f54b6e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15752,7 +15758,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dec96a1575e340378279ac1177dfb4d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dec96a1575e340378279ac1177dfb4d0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0a88538bcc8740279e32eab02e1e9d9f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0a88538bcc8740279e32eab02e1e9d9f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15770,7 +15776,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24c4050a754c4cb09bc9e4804b1e895e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24c4050a754c4cb09bc9e4804b1e895e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/603016c9335e4757ad2ea9bfb3f3698d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/603016c9335e4757ad2ea9bfb3f3698d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15788,7 +15794,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8a466eb9a9154bff95c69fc95a7cf432-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8a466eb9a9154bff95c69fc95a7cf432-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d7d3687ef1c1410581769106662a058f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d7d3687ef1c1410581769106662a058f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15806,7 +15812,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/406bcf4c0b7a4e208a317968b7db6abe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/406bcf4c0b7a4e208a317968b7db6abe-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/249983b188744844a70647c741a5d0d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/249983b188744844a70647c741a5d0d6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15824,7 +15830,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b563773e18c47438f7056e0f249e224-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b563773e18c47438f7056e0f249e224-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d3c35a7b6252475db02c07795c8bdaa7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d3c35a7b6252475db02c07795c8bdaa7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15842,7 +15848,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db397bc7364c4ab49d2dcff1aee1a60a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db397bc7364c4ab49d2dcff1aee1a60a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e0f56ca4a86a4818a38fca786e7189f7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e0f56ca4a86a4818a38fca786e7189f7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15860,7 +15866,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/04a7a8f58b6d4e318b5ba4a515199c02-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/04a7a8f58b6d4e318b5ba4a515199c02-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/103194c706654c07819df9a290b6becf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/103194c706654c07819df9a290b6becf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15878,7 +15884,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ada5d03c971c47d482c91a342de4651e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ada5d03c971c47d482c91a342de4651e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/869f2d4618344171ac27b1fdf3d14c50-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/869f2d4618344171ac27b1fdf3d14c50-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15896,7 +15902,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1d55ddc35b664cafa9c674d78858f3ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1d55ddc35b664cafa9c674d78858f3ed-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0027c69995c5461fa867c06f9369dad6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0027c69995c5461fa867c06f9369dad6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15914,7 +15920,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ab9b758aa4cb4b9489bd16c7ab2b8574-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ab9b758aa4cb4b9489bd16c7ab2b8574-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64de3f9b8379486a9e670a262251169a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64de3f9b8379486a9e670a262251169a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15932,7 +15938,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d85cd6fa45ba46c7b4f383035186f333-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d85cd6fa45ba46c7b4f383035186f333-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/31210e2e33fa4ce1b1714791496414bc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/31210e2e33fa4ce1b1714791496414bc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15950,7 +15956,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1cfd05631bfc4734b8f8e1381477f20c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1cfd05631bfc4734b8f8e1381477f20c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8fde06d8ea04680a5d361114258210c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8fde06d8ea04680a5d361114258210c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15968,7 +15974,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3395dad8b8e54bcbb60a392b4c88bfcd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3395dad8b8e54bcbb60a392b4c88bfcd-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d42b6a21e4a841fd952a64d5dd398338-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d42b6a21e4a841fd952a64d5dd398338-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -15986,7 +15992,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1cb136e3f5334a349016b4f2de813b79-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1cb136e3f5334a349016b4f2de813b79-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/baa5c9339cd5471c997cda7dcf6956ac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/baa5c9339cd5471c997cda7dcf6956ac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16004,7 +16010,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9f6436dfb7eb4c6abe1a2b2d9450fe72-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9f6436dfb7eb4c6abe1a2b2d9450fe72-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a656a8cd7085425bb6aafd96606606e0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a656a8cd7085425bb6aafd96606606e0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16022,7 +16028,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/71088b1ea3ff48b29d69bab1eeff4fa2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/71088b1ea3ff48b29d69bab1eeff4fa2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1b9f45be3d441c59c99bc4244d5f400-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1b9f45be3d441c59c99bc4244d5f400-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16040,7 +16046,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ac4777a9c0a24973a213b8e0e4082925-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ac4777a9c0a24973a213b8e0e4082925-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba4220cf657b4a2bb6ab243731be3e9d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba4220cf657b4a2bb6ab243731be3e9d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16058,7 +16064,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a41cd974cc724af3a633b9c31f7bf5af-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a41cd974cc724af3a633b9c31f7bf5af-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ccbb128c723047a5a655316bad597437-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ccbb128c723047a5a655316bad597437-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16076,7 +16082,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fbdffc89d7964d47a912740fb091354b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fbdffc89d7964d47a912740fb091354b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4afb1ef3fafd43cda4a38b9e88a92397-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4afb1ef3fafd43cda4a38b9e88a92397-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16094,7 +16100,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ddb5ecda5cfe4362a03d867f27833ad2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ddb5ecda5cfe4362a03d867f27833ad2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/27fc2ee155b54bc38f1fb13364eea40d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/27fc2ee155b54bc38f1fb13364eea40d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16112,7 +16118,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/503f79f9064f46d1933558f11b66f5a8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/503f79f9064f46d1933558f11b66f5a8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/afee7095f1774e71b24b7d61f2d2f085-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/afee7095f1774e71b24b7d61f2d2f085-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16130,7 +16136,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dbeb9b453613461199cff124c669c297-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dbeb9b453613461199cff124c669c297-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f19e8a2e5967404081a57e83bc2c3df4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f19e8a2e5967404081a57e83bc2c3df4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16148,7 +16154,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bd4bc6f7884b446089485f4e9393801d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bd4bc6f7884b446089485f4e9393801d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/25c0388b52e340f78627823301ca735c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/25c0388b52e340f78627823301ca735c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16166,7 +16172,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/39ea0259f5e64deea6d39397041d35b1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/39ea0259f5e64deea6d39397041d35b1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a850b999761f47c49ac65eaef45006fb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a850b999761f47c49ac65eaef45006fb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16184,7 +16190,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fe8726803db942d6b7ec59bffa4d5d50-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fe8726803db942d6b7ec59bffa4d5d50-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/abe48837a691474aa913840badfe4359-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/abe48837a691474aa913840badfe4359-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16202,7 +16208,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d0f14d7e05cd43ef8d97255e8fc270c8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d0f14d7e05cd43ef8d97255e8fc270c8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4280a9288d734f2e9c213502a1977022-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4280a9288d734f2e9c213502a1977022-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16220,7 +16226,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1b2b85c48a9d40f39cf4727d30083482-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1b2b85c48a9d40f39cf4727d30083482-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6811fa1e45f24d538dd99f83b2873a77-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6811fa1e45f24d538dd99f83b2873a77-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16238,7 +16244,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/781f5e58d0df485db7c64d35ba58d5a2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/781f5e58d0df485db7c64d35ba58d5a2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ebfe63e1df9646579d331a9d2dbb2ee8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ebfe63e1df9646579d331a9d2dbb2ee8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16256,7 +16262,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53b2fe47a76d4f08abc6f4739ac3da03-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53b2fe47a76d4f08abc6f4739ac3da03-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/192b400f8bdb476492d577ea693c63c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/192b400f8bdb476492d577ea693c63c6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16274,7 +16280,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3cdce5d5dbf64a74a6dbc70c3c831c33-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3cdce5d5dbf64a74a6dbc70c3c831c33-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b1a3877d5cf49c9aeba5bc30294512c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b1a3877d5cf49c9aeba5bc30294512c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16292,7 +16298,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8f7355f38894d2fb2511489a503b42a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8f7355f38894d2fb2511489a503b42a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0bdb81d2fcab40acbc64bc86ab4970f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0bdb81d2fcab40acbc64bc86ab4970f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16310,7 +16316,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8348c0d8a56442afb07b9195133479ae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8348c0d8a56442afb07b9195133479ae-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e136bd43c54946e98bf147e506c87afc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e136bd43c54946e98bf147e506c87afc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16328,7 +16334,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/81ccff339b624585b6e5229ddded6bc4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/81ccff339b624585b6e5229ddded6bc4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d47e8747a33d4349a7e3a3abb45da46f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d47e8747a33d4349a7e3a3abb45da46f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16346,7 +16352,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/84ded7a652ec4aa48c108a05e8b24383-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/84ded7a652ec4aa48c108a05e8b24383-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3f4e7067786d4c23a601fc7bd5990192-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3f4e7067786d4c23a601fc7bd5990192-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16364,7 +16370,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6fa795f331c24c8682d033744dfd510b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6fa795f331c24c8682d033744dfd510b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e22dada7eed4b7499f5e0f81e946d5e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e22dada7eed4b7499f5e0f81e946d5e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16382,7 +16388,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9ab25d94848a4927b0ebb48ca4e414d8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9ab25d94848a4927b0ebb48ca4e414d8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/04f40b8909da410ebaece930ded2112c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/04f40b8909da410ebaece930ded2112c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16400,7 +16406,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e242b243a863456798f23a9604879266-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e242b243a863456798f23a9604879266-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b9c1f7c7df0945e8a9a823de78598921-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b9c1f7c7df0945e8a9a823de78598921-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16418,7 +16424,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4864c86aff44475fa1593cf2bab8e492-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4864c86aff44475fa1593cf2bab8e492-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ccb7c77c359f4f0b9c6cae95b46a86b5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ccb7c77c359f4f0b9c6cae95b46a86b5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16436,7 +16442,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7dadf1a7ef82488c8e75529708f17671-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7dadf1a7ef82488c8e75529708f17671-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f9f2798471464308adba375191562ef2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f9f2798471464308adba375191562ef2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16448,13 +16454,13 @@
"Option for timeMode changed from cpu to elapsed\n",
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.01\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b2d737b15a65434d90e4f7d9cc36f633-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b2d737b15a65434d90e4f7d9cc36f633-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6a42c9b9467d4ba3adcb7e76e39fb1d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6a42c9b9467d4ba3adcb7e76e39fb1d0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16472,7 +16478,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/955dcaabc3d0407a9d4a4065eb168b44-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/955dcaabc3d0407a9d4a4065eb168b44-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3501d15c74e3450ba83d0a8f1033e7f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3501d15c74e3450ba83d0a8f1033e7f1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16490,7 +16496,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/723b87ff1466478daf127d3c0b727914-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/723b87ff1466478daf127d3c0b727914-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/547cd3d7972e46b9a085ecfc47e7e22d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/547cd3d7972e46b9a085ecfc47e7e22d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16508,7 +16514,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/efed4645d7694412a87e44f374df55e1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/efed4645d7694412a87e44f374df55e1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f035cd79666c4361b41ba75a3392ed6d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f035cd79666c4361b41ba75a3392ed6d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16526,7 +16532,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/98a087b3b9724f569ba44e4c71f65ccc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/98a087b3b9724f569ba44e4c71f65ccc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/853a38a249da40cbb83d17eb4c366460-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/853a38a249da40cbb83d17eb4c366460-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16544,7 +16550,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c25a1ff12dd9475ba076485907fa7208-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c25a1ff12dd9475ba076485907fa7208-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ad0d87f504cd43d78925d3fedf435267-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ad0d87f504cd43d78925d3fedf435267-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16562,7 +16568,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32dc2c03494e4068b6a5a3652a64e76c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32dc2c03494e4068b6a5a3652a64e76c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dd8ccb240dd841ab986834fa1d6eae3c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dd8ccb240dd841ab986834fa1d6eae3c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16580,7 +16586,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dac5b02868aa492980d8a2ec693b6f7c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dac5b02868aa492980d8a2ec693b6f7c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a23efb6884c4407c8c1c013788819679-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a23efb6884c4407c8c1c013788819679-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16598,7 +16604,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/23923074fbbe49e0b61e5c31b457b905-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/23923074fbbe49e0b61e5c31b457b905-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f40b03dc7df9434bbe8882cb09127c9c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f40b03dc7df9434bbe8882cb09127c9c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16616,7 +16622,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef75b270a72241c59ac744afc697c658-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef75b270a72241c59ac744afc697c658-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/013c62cfe2d1415c9a234b4261afedb5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/013c62cfe2d1415c9a234b4261afedb5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16634,7 +16640,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/972b92b7691248a986e79ba444f04b17-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/972b92b7691248a986e79ba444f04b17-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/154bf0774fbf4689a7aab046bfc753d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/154bf0774fbf4689a7aab046bfc753d2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16652,7 +16658,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/246cd69bd9e841d196aad349573460a2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/246cd69bd9e841d196aad349573460a2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f8ad9a8f07a8463abc4de76f1dbf4b5d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f8ad9a8f07a8463abc4de76f1dbf4b5d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16670,7 +16676,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c4fd187a4cb54930b2b771f94f1eec1f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c4fd187a4cb54930b2b771f94f1eec1f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cbc150e76da446719a4284d885d8e298-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cbc150e76da446719a4284d885d8e298-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16688,7 +16694,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4afbd13492414ac094ce4bbc464ba872-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4afbd13492414ac094ce4bbc464ba872-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc7df48e9b03498d8f7de1ff56a56823-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc7df48e9b03498d8f7de1ff56a56823-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16706,7 +16712,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bd43e65fce00482dbc9f314f8ec18cef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bd43e65fce00482dbc9f314f8ec18cef-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f0a2cd67daf14d8fafbb83d3cc16913a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f0a2cd67daf14d8fafbb83d3cc16913a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16724,7 +16730,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f321f05b417e43abbb7bb22d97e67174-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f321f05b417e43abbb7bb22d97e67174-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7659b7fdbf1840f284077a9b592a6305-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7659b7fdbf1840f284077a9b592a6305-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16742,7 +16748,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7651eb0f0f2a410397ecbac693910604-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7651eb0f0f2a410397ecbac693910604-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d5e10bc8e23347d692be4456d8d56c54-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d5e10bc8e23347d692be4456d8d56c54-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16760,7 +16766,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ca1d249019f14f0cb8f3634b9873eb3b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ca1d249019f14f0cb8f3634b9873eb3b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba5270b05dbb4d0391f05c9ec11cf1c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba5270b05dbb4d0391f05c9ec11cf1c5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16778,7 +16784,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4dc3af56b2194bbab2166cfb2d19d7a0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4dc3af56b2194bbab2166cfb2d19d7a0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/311d34e4dfda42fe9628d9f4c377931d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/311d34e4dfda42fe9628d9f4c377931d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16796,7 +16802,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f407d16f3ff94ea0a66c606155f83b58-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f407d16f3ff94ea0a66c606155f83b58-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b23621a41462456cafb4b3412357a37b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b23621a41462456cafb4b3412357a37b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16814,7 +16820,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/33618bf9951441ac8cd0004249df8d57-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/33618bf9951441ac8cd0004249df8d57-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7e0060bffb9d4e758be022f7d82c7117-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7e0060bffb9d4e758be022f7d82c7117-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16832,7 +16838,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/24e5cb9789444201916961f2cb51cfff-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/24e5cb9789444201916961f2cb51cfff-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f77b335f420b4880a9582a14e0d720cd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f77b335f420b4880a9582a14e0d720cd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16852,7 +16858,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/65c53000282d40d4b07f7fad511fc934-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/65c53000282d40d4b07f7fad511fc934-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/60cf92e3eb104d77809c068129de1732-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/60cf92e3eb104d77809c068129de1732-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16870,7 +16876,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2ad3e4f234f440fab0fc015813e1a71b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2ad3e4f234f440fab0fc015813e1a71b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73692049cff6405cb641b38a24134d27-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73692049cff6405cb641b38a24134d27-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16888,7 +16894,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a11d0c93ce1c4407967102b5402f65c2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a11d0c93ce1c4407967102b5402f65c2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/468755c9b414435886545cda25ff0c0e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/468755c9b414435886545cda25ff0c0e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16906,7 +16912,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4acb6dd3750c4e1b8e290df0d01b95a1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4acb6dd3750c4e1b8e290df0d01b95a1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dff843b2f5b04f1ea68ee7045289cc61-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dff843b2f5b04f1ea68ee7045289cc61-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16924,7 +16930,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c105518ab53749aaa8e3c475b5264c2a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c105518ab53749aaa8e3c475b5264c2a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8450f947138b41f8b7220d248f699d61-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8450f947138b41f8b7220d248f699d61-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16942,7 +16948,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b1d4589382f49aab6f912f3ded8e610-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b1d4589382f49aab6f912f3ded8e610-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/44df967bba2c45db806b4449f5203bcb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/44df967bba2c45db806b4449f5203bcb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16960,7 +16966,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b450efebbfbd4bec953510f310141f41-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b450efebbfbd4bec953510f310141f41-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9c7c8191c798435489664e0d610ec4f3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9c7c8191c798435489664e0d610ec4f3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16978,7 +16984,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a90e5d5af84d43f198f4e0c3a2752c39-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a90e5d5af84d43f198f4e0c3a2752c39-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/63a51b853a5a4ec088a0cda3c14c04f6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/63a51b853a5a4ec088a0cda3c14c04f6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -16996,7 +17002,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9f784e0d01db4a9a9dc185aee229d711-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9f784e0d01db4a9a9dc185aee229d711-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ba2d2df195074a3b9d9f68615bc56d31-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ba2d2df195074a3b9d9f68615bc56d31-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17014,7 +17020,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5df0a0b35b8746c3b0662ab961afabb5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5df0a0b35b8746c3b0662ab961afabb5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5d64451bfc42411c8ee32c37689cabfc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5d64451bfc42411c8ee32c37689cabfc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17032,7 +17038,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6fe2d1da94fa4251a6d6160b2869e030-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6fe2d1da94fa4251a6d6160b2869e030-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f880ec6badf64ea087c3db5c96a0b11f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f880ec6badf64ea087c3db5c96a0b11f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17050,7 +17056,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bfb294de6fa74285bd3091f6ad8095a3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bfb294de6fa74285bd3091f6ad8095a3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1620791b508449daa9215a454862dd56-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1620791b508449daa9215a454862dd56-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17068,7 +17074,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/395e8b3d99f548a99948f29268053519-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/395e8b3d99f548a99948f29268053519-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/513c86bdec80417582132033293fca8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/513c86bdec80417582132033293fca8c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17086,7 +17092,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c26a5545213d40a18dcf293d3741db53-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c26a5545213d40a18dcf293d3741db53-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e5418bf13e174b28bfae7bd37c43c037-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e5418bf13e174b28bfae7bd37c43c037-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17104,7 +17110,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c258f10896cb4b59a606132cbfd4f56b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c258f10896cb4b59a606132cbfd4f56b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/194e22d3802d4653a5cb79af0c5a5f2f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/194e22d3802d4653a5cb79af0c5a5f2f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17122,7 +17128,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5343591d6d7e4f30962961c0cc44aa75-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5343591d6d7e4f30962961c0cc44aa75-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8434b5836954411ab55a72b48430f6ea-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8434b5836954411ab55a72b48430f6ea-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17140,7 +17146,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f443d5c8759944eb9d05594d0ba5bc3a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f443d5c8759944eb9d05594d0ba5bc3a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c3b1cc2b89374b21a2d70fe05193e40c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c3b1cc2b89374b21a2d70fe05193e40c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17158,7 +17164,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7a9f679cc7094cf5b1eab228003006d9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7a9f679cc7094cf5b1eab228003006d9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b75bc81277dc4f52ae0ad166e6ae3c94-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b75bc81277dc4f52ae0ad166e6ae3c94-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17176,7 +17182,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3de19c928df74b6da3a3b5b521761505-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3de19c928df74b6da3a3b5b521761505-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f526bb999944407b547e2fff21d5a95-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f526bb999944407b547e2fff21d5a95-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17194,7 +17200,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3bf27b84925b4ea5959334026dc9fe71-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3bf27b84925b4ea5959334026dc9fe71-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6075445919744fee9e10e602cb171b92-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6075445919744fee9e10e602cb171b92-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17212,7 +17218,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/69442bd1ee5c4957ad4fb4a422ac963c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/69442bd1ee5c4957ad4fb4a422ac963c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e6866d4cbca74d66ba1644271fd79d9a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e6866d4cbca74d66ba1644271fd79d9a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17230,7 +17236,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6aac48d05a0440bcad24cd33b7d3c820-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6aac48d05a0440bcad24cd33b7d3c820-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/89123a74727e4a0b8fc24d3e1f8411f2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/89123a74727e4a0b8fc24d3e1f8411f2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17248,7 +17254,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f12eb6f17256456bb62cfdc9138409ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f12eb6f17256456bb62cfdc9138409ed-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d37f3c9ca824c0f8fd8fc1407ff220c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d37f3c9ca824c0f8fd8fc1407ff220c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17266,7 +17272,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f953de4cd94740aba7bac4cc93cef0de-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f953de4cd94740aba7bac4cc93cef0de-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b49920aa6664a77bcef36e1a493434d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b49920aa6664a77bcef36e1a493434d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17284,7 +17290,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f385674b34ad4b4aacca8f327124abe3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f385674b34ad4b4aacca8f327124abe3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b09bec97d5264179929279b0705e1bf4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b09bec97d5264179929279b0705e1bf4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17302,7 +17308,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/763ff33c84e644e499ae73bc7c3765f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/763ff33c84e644e499ae73bc7c3765f1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a1c4f39862784908bc0b41ca72f190cf-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a1c4f39862784908bc0b41ca72f190cf-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17320,7 +17326,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/17fd0f67ab81463eacda89a8623469f7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/17fd0f67ab81463eacda89a8623469f7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57d36e247d824145a5c2f497bf8088e2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57d36e247d824145a5c2f497bf8088e2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17338,7 +17344,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1260f42a4f5043c3a5d302ff3fa9785c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1260f42a4f5043c3a5d302ff3fa9785c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32db34be8dbe43d2ab9540d6d6fd1ac7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32db34be8dbe43d2ab9540d6d6fd1ac7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17356,7 +17362,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d00020eaacbb4306acd577ac2593706c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d00020eaacbb4306acd577ac2593706c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/85d80e0583394fa4af97c47a679f168f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/85d80e0583394fa4af97c47a679f168f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17374,7 +17380,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/36f007a40844468fb005ac2439f222b9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/36f007a40844468fb005ac2439f222b9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e54a046588f40c897cd78e8cc67549c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e54a046588f40c897cd78e8cc67549c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17392,7 +17398,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/64776df3844c4d57bd9addabc72fedfc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/64776df3844c4d57bd9addabc72fedfc-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e1bc59b38e4342b6a20ac2c0bda949f6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e1bc59b38e4342b6a20ac2c0bda949f6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17410,7 +17416,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5ca88f7fcb9d451298c2f87f81c3520f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5ca88f7fcb9d451298c2f87f81c3520f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/773a953dda58433aa261ecb7d59d336b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/773a953dda58433aa261ecb7d59d336b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17428,7 +17434,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/79712257ecb646f3a4b682ec8c9fdf14-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/79712257ecb646f3a4b682ec8c9fdf14-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a8b8226be4e6466f9553798ed756a416-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a8b8226be4e6466f9553798ed756a416-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17438,15 +17444,15 @@
"Problem MODEL has 6 rows, 11 columns and 40 elements\n",
"Coin0008I MODEL read with 0 errors\n",
"Option for timeMode changed from cpu to elapsed\n",
- "Problem is infeasible - 0.00 seconds\n",
+ "Problem is infeasible - 0.01 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/58ad8aa427a94ef98cb9e88ae16e3051-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/58ad8aa427a94ef98cb9e88ae16e3051-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/45824060c51343b48f32b2703d00eb53-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/45824060c51343b48f32b2703d00eb53-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17456,15 +17462,15 @@
"Problem MODEL has 6 rows, 11 columns and 40 elements\n",
"Coin0008I MODEL read with 0 errors\n",
"Option for timeMode changed from cpu to elapsed\n",
- "Problem is infeasible - 0.00 seconds\n",
+ "Problem is infeasible - 0.01 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/be8e57f0b09a464695f227d27526e3f1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/be8e57f0b09a464695f227d27526e3f1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6eb5b2e0603b47708c437068255f2b8c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6eb5b2e0603b47708c437068255f2b8c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17482,7 +17488,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5c82410b74d7436b830e6cf0002be063-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5c82410b74d7436b830e6cf0002be063-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c468314eed740b09f48b66d2e037661-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c468314eed740b09f48b66d2e037661-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17500,7 +17506,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a3647fdc81414911a596bd1c3a4dc527-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a3647fdc81414911a596bd1c3a4dc527-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/50278a8a5431464198e232e740368203-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/50278a8a5431464198e232e740368203-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17518,7 +17524,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/840bcb647cd1463a9e3fafb9b21e5077-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/840bcb647cd1463a9e3fafb9b21e5077-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9935ffd5c2fe4ba59c034d291cafb7d2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9935ffd5c2fe4ba59c034d291cafb7d2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17536,7 +17542,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/408f8f92807f4cc6b7c06ecf59cbc849-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/408f8f92807f4cc6b7c06ecf59cbc849-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8a31e2ff858d4cf38cea4ce79c857d0c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8a31e2ff858d4cf38cea4ce79c857d0c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17554,7 +17560,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e8092fa4cf834cd4994c97781d600b27-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e8092fa4cf834cd4994c97781d600b27-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9bda512a742646ed8604437a833b307b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9bda512a742646ed8604437a833b307b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17572,7 +17578,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/347ffd782cb44922bf44c599a0c09a3a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/347ffd782cb44922bf44c599a0c09a3a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7c524c62ed6a43dc9cead8b7b129874b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7c524c62ed6a43dc9cead8b7b129874b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17590,7 +17596,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2453387d8b54437b898d1d8bae52ddb8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2453387d8b54437b898d1d8bae52ddb8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/012da964533b412497c8b5b20b0848b6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/012da964533b412497c8b5b20b0848b6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17608,7 +17614,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/373222932f63400a99c937eecaaba549-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/373222932f63400a99c937eecaaba549-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9ae7f4f2d7264affa0ba1f23ab6fadb9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9ae7f4f2d7264affa0ba1f23ab6fadb9-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17626,7 +17632,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fb15987c05a345afb1a51b98a0543f6d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fb15987c05a345afb1a51b98a0543f6d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0209ca7dc1d44e9db76ac0a19784009e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0209ca7dc1d44e9db76ac0a19784009e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17644,7 +17650,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2507f6afe3ab431c8f469c23ac3c5779-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2507f6afe3ab431c8f469c23ac3c5779-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/070f3379961f44d1b573370bd46ca86b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/070f3379961f44d1b573370bd46ca86b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17662,7 +17668,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e79f0607c2cb4020bc857776f4249196-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e79f0607c2cb4020bc857776f4249196-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/810086ec45d146f9967c945889cdccfc-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/810086ec45d146f9967c945889cdccfc-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17680,7 +17686,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/68834f69a39345e4b76a51ae6e4bf732-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/68834f69a39345e4b76a51ae6e4bf732-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/040dc1f7b12b4dde99b4e0adc2ef9ce0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/040dc1f7b12b4dde99b4e0adc2ef9ce0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17698,7 +17704,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cd201bfb3e27418582786dc968cb730c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cd201bfb3e27418582786dc968cb730c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3326b80320074949b4a311cbc104355d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3326b80320074949b4a311cbc104355d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17716,7 +17722,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/13ce369ea98d40c98d08be83dc31c47b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/13ce369ea98d40c98d08be83dc31c47b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00779548df364b078465781fb0b63dcd-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00779548df364b078465781fb0b63dcd-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17734,7 +17740,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a260f49abf8d4c4797f5337381549f19-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a260f49abf8d4c4797f5337381549f19-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f34e432887eb49e1a086a6f0412e2f36-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f34e432887eb49e1a086a6f0412e2f36-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17752,7 +17758,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/14ec0a6f2f594ffca8b078cd79e3550e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/14ec0a6f2f594ffca8b078cd79e3550e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ecc794cf67cb4f5e9c40a9a8e32651ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ecc794cf67cb4f5e9c40a9a8e32651ee-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17770,7 +17776,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae9eabcec4024d008ff9ebf2ff8c1c48-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae9eabcec4024d008ff9ebf2ff8c1c48-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ef004e4d3cb54823958f900720949f6c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ef004e4d3cb54823958f900720949f6c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17788,7 +17794,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9d3cd612c33c4b279ade7f3f5dc9506f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9d3cd612c33c4b279ade7f3f5dc9506f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ab54737dd9ff44148b348ead5c6e4764-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ab54737dd9ff44148b348ead5c6e4764-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17806,7 +17812,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e4335a35c4944eb8b1b4b37f90f41c33-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e4335a35c4944eb8b1b4b37f90f41c33-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/812e0a7ede3449808c12a9f0079faa0b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/812e0a7ede3449808c12a9f0079faa0b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17824,7 +17830,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/30df3ab81f954c4fa1489a33fe38c54e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/30df3ab81f954c4fa1489a33fe38c54e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7acac09389e24c42b3540ec4b44f4b09-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7acac09389e24c42b3540ec4b44f4b09-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17842,7 +17848,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1d77c229b16b4420962687d50e2bd9d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1d77c229b16b4420962687d50e2bd9d3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8c66cf8bdb4945438fdb9a402fc99382-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8c66cf8bdb4945438fdb9a402fc99382-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17860,7 +17866,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/370212dd7ab54206a62f83ed7fa9c619-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/370212dd7ab54206a62f83ed7fa9c619-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cd536713856b4ed7ab7dd162b53d027c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cd536713856b4ed7ab7dd162b53d027c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17878,7 +17884,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e395fd8d55494951a2eaa9306ec6bc0d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e395fd8d55494951a2eaa9306ec6bc0d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e07133a0e2a740c2853783141b4a0fb1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e07133a0e2a740c2853783141b4a0fb1-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17896,7 +17902,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c75c0954550c49a692770ce33cf9fe34-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c75c0954550c49a692770ce33cf9fe34-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bcb9668d92aa49e593b6f907e89f866d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bcb9668d92aa49e593b6f907e89f866d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17914,7 +17920,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/72d3505e1a6f400d98f72631b0fea265-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/72d3505e1a6f400d98f72631b0fea265-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/61538e3bdcb745a7923b70de5cd4e98c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/61538e3bdcb745a7923b70de5cd4e98c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17932,7 +17938,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/91d659c184914440aea03e22763237f8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/91d659c184914440aea03e22763237f8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ff805ed15ae54773bbcb2cdfa9d92fdb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ff805ed15ae54773bbcb2cdfa9d92fdb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17950,7 +17956,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2fbca6e3a8584dd8b7d409e44f473faa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2fbca6e3a8584dd8b7d409e44f473faa-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5bb48e44c7d4478d841ceb2aa539862a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5bb48e44c7d4478d841ceb2aa539862a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17968,7 +17974,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d15769a507e24558b26af165d07e0d81-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d15769a507e24558b26af165d07e0d81-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5413e271e5dc4bc1bba659adab7471bb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5413e271e5dc4bc1bba659adab7471bb-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -17986,7 +17992,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/abc11db3cde44ebdad521d78a6ef66a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/abc11db3cde44ebdad521d78a6ef66a4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c918a3b609be4910a2958a68193f8e1a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c918a3b609be4910a2958a68193f8e1a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18004,7 +18010,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b5b28090dc34c2ebbe4f2f489770f90-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b5b28090dc34c2ebbe4f2f489770f90-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f46209c726bd4b0d90cfedbfe46bd1c7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f46209c726bd4b0d90cfedbfe46bd1c7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18022,7 +18028,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4f6cec88218f4de3a9d8cf75a751ff99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4f6cec88218f4de3a9d8cf75a751ff99-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ff417b8b7a44f3daa578299aceb8bbe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ff417b8b7a44f3daa578299aceb8bbe-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18040,7 +18046,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae1bcee4c8d94b8d9e1d09b5a02a5f35-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae1bcee4c8d94b8d9e1d09b5a02a5f35-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ba20e6f78214294a13c019aaa2f913e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ba20e6f78214294a13c019aaa2f913e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18058,7 +18064,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cdb024265a5049e69d41e39450e9253d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cdb024265a5049e69d41e39450e9253d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b852a8e705d242d8ad8e37410499b92d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b852a8e705d242d8ad8e37410499b92d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18076,7 +18082,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cefe84b0f7ce434289ad6e258245748a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cefe84b0f7ce434289ad6e258245748a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32423aeb8de8476a9ddd6c4961e28a64-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32423aeb8de8476a9ddd6c4961e28a64-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18094,7 +18100,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00405506154c4b13a4bda2b27b6aa3e3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00405506154c4b13a4bda2b27b6aa3e3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6be115330a9e4ca29fecad44a30e495e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6be115330a9e4ca29fecad44a30e495e-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18112,7 +18118,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/be863a9ccaa44edcad9cdab429667915-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/be863a9ccaa44edcad9cdab429667915-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/232ac9a5bb064a688e21dc9e863c6e3b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/232ac9a5bb064a688e21dc9e863c6e3b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18130,7 +18136,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/56b4466fcff7405db14673d85d317dc9-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/56b4466fcff7405db14673d85d317dc9-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fb55c463c6274e349ee42d3f38faff12-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fb55c463c6274e349ee42d3f38faff12-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18148,7 +18154,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cd7287e81f514ee0b3db3e78808dd138-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cd7287e81f514ee0b3db3e78808dd138-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/73228163de4b4b69b6ac988c7ec6c665-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/73228163de4b4b69b6ac988c7ec6c665-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18166,7 +18172,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/129af26b8d6f4fe5a75e5d40bc7301c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/129af26b8d6f4fe5a75e5d40bc7301c6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7ecd2937ecf647cf874c170ea00711f5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7ecd2937ecf647cf874c170ea00711f5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18184,7 +18190,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8e82eed9433f42599fdf5084f849b21d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8e82eed9433f42599fdf5084f849b21d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2d0bd041751c44448d36bc28fc4fd776-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2d0bd041751c44448d36bc28fc4fd776-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18202,7 +18208,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8f0cfa007fd0448fb33e62ae99e9a49c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8f0cfa007fd0448fb33e62ae99e9a49c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c1f2bdbe2f8f4db49f1612232155611d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c1f2bdbe2f8f4db49f1612232155611d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18220,7 +18226,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/545fcbdfc2c24b63a215e438eb6bf15a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/545fcbdfc2c24b63a215e438eb6bf15a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f6b619f52933455bbb9be02e270d750b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f6b619f52933455bbb9be02e270d750b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18238,7 +18244,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a18b9a8e62c74031a49d2c253578d375-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a18b9a8e62c74031a49d2c253578d375-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0914c4d3b4d34f14b83f89f90514d7d3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0914c4d3b4d34f14b83f89f90514d7d3-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18256,7 +18262,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/20edf09633a348769e5fbe4b991c7497-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/20edf09633a348769e5fbe4b991c7497-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4215f4635a294104a0ce96446ec93afe-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4215f4635a294104a0ce96446ec93afe-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18274,7 +18280,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/500ec4f433644c3087349ac450795b2d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/500ec4f433644c3087349ac450795b2d-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2c37c089c14244df95ee144d7d59eb33-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2c37c089c14244df95ee144d7d59eb33-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18292,7 +18298,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4f4a410470414301aa0a9e4fd2d61ea1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4f4a410470414301aa0a9e4fd2d61ea1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e0a0da111af248ae8f4487dd68759b0b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e0a0da111af248ae8f4487dd68759b0b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18310,7 +18316,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a816ef3b2dfb4b8094f4f88c16116719-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a816ef3b2dfb4b8094f4f88c16116719-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b42314d9dac442d79138fac23f8ab270-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b42314d9dac442d79138fac23f8ab270-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18328,7 +18334,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1d8eddc05ece48a995ffda92a3e83c79-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1d8eddc05ece48a995ffda92a3e83c79-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7d2f353b16c24e259d6298abdff22679-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7d2f353b16c24e259d6298abdff22679-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18346,7 +18352,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2404585b911d456cbb303241502f45a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2404585b911d456cbb303241502f45a4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cf493651442745e7b4d8f408093ec38f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cf493651442745e7b4d8f408093ec38f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18364,7 +18370,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cc31a622f085435c9aed78dc2c1a14a7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cc31a622f085435c9aed78dc2c1a14a7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c8e803658534ca39f09e61b74b17c32-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c8e803658534ca39f09e61b74b17c32-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18376,13 +18382,13 @@
"Option for timeMode changed from cpu to elapsed\n",
"Problem is infeasible - 0.00 seconds\n",
"Option for printingOptions changed from normal to all\n",
- "Total time (CPU seconds): 0.01 (Wallclock seconds): 0.01\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
"\n",
"Welcome to the CBC MILP Solver \n",
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1e6b668f8a9c4f468482f041b7ddc9ee-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1e6b668f8a9c4f468482f041b7ddc9ee-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dbec9f826616447ab9c19df4221f1c11-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dbec9f826616447ab9c19df4221f1c11-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18400,7 +18406,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/709057086a3d4de58f073dd0758dccab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/709057086a3d4de58f073dd0758dccab-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c59ad4aef2647ae9659407193ed7bb8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c59ad4aef2647ae9659407193ed7bb8-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18418,7 +18424,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f81872b21530495fbf89fdce89c16898-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f81872b21530495fbf89fdce89c16898-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f843b1f45b954a148b97cefa4c8161ec-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f843b1f45b954a148b97cefa4c8161ec-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18436,7 +18442,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d6da9d29515646e0b5349290fd0902b7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d6da9d29515646e0b5349290fd0902b7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c02079685de94149a9753ea4a5407d7a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c02079685de94149a9753ea4a5407d7a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18454,7 +18460,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ab27049021144c649c470805d3045501-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ab27049021144c649c470805d3045501-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3026586a80634144b5800f048c8e7614-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3026586a80634144b5800f048c8e7614-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18472,7 +18478,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b090d6e630234725aef0124919dd0c4b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b090d6e630234725aef0124919dd0c4b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d8107b60913c4c29bc5cbcd168068245-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d8107b60913c4c29bc5cbcd168068245-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18490,7 +18496,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/45b48418ac394d43a24dad44f1ca8162-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/45b48418ac394d43a24dad44f1ca8162-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a9d99fcc29754815975bd0b2f1b48e20-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a9d99fcc29754815975bd0b2f1b48e20-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18508,7 +18514,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e7f982da1c724bd39a4002242de024b3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e7f982da1c724bd39a4002242de024b3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5fdea6ebd7574f4581216cc79cad1f6c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5fdea6ebd7574f4581216cc79cad1f6c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18526,7 +18532,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/20386972fe0049eba213ba596aae9a87-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/20386972fe0049eba213ba596aae9a87-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f6680890449043969001f7eb647481a6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f6680890449043969001f7eb647481a6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18544,7 +18550,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6b6cf788eff64b8192c73487ee13f42b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6b6cf788eff64b8192c73487ee13f42b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fd5d38ace7e8415dac5533d97fb691d0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fd5d38ace7e8415dac5533d97fb691d0-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18562,7 +18568,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f3c5eda2f75f4efe9db2ab693f9a1229-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f3c5eda2f75f4efe9db2ab693f9a1229-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/600a00f0eae245b4b19be4c7004f31af-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/600a00f0eae245b4b19be4c7004f31af-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18580,7 +18586,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/7dd0b38f9d824f2dbf1729de99c82ee6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/7dd0b38f9d824f2dbf1729de99c82ee6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/43d998a5323a4ac8a7346cd5954e295d-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/43d998a5323a4ac8a7346cd5954e295d-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18598,7 +18604,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/77f1275a35334f0d899a1e6e76ee39f7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/77f1275a35334f0d899a1e6e76ee39f7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8016bce60333455f9ca47c21e9b3dea7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8016bce60333455f9ca47c21e9b3dea7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18616,7 +18622,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f7a262fdb9b04e7c98f6e4437bc7f7cb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f7a262fdb9b04e7c98f6e4437bc7f7cb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3136e9a9e77949b691cd1b7cf548d225-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3136e9a9e77949b691cd1b7cf548d225-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18634,7 +18640,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ab99e626f7484608b058226a6476fcd4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ab99e626f7484608b058226a6476fcd4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/50aeb7ab7420497487de6908a0e54eca-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/50aeb7ab7420497487de6908a0e54eca-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18652,7 +18658,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8dd71462b5b14da291684b91774f9ff7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8dd71462b5b14da291684b91774f9ff7-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c462b95ef2944184af931e72854b227b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c462b95ef2944184af931e72854b227b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18670,7 +18676,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fa92ee860d63440e8bbeca80bf576241-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fa92ee860d63440e8bbeca80bf576241-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8ab21969a93e4b5bb03b0e825d1f76ef-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8ab21969a93e4b5bb03b0e825d1f76ef-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18688,7 +18694,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1d8866724bac45daa1e124b495a5db4b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1d8866724bac45daa1e124b495a5db4b-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/21dae29c98b94c459f038776c77bc941-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/21dae29c98b94c459f038776c77bc941-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18706,7 +18712,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/39f150ab23404df08bcf30da932a61da-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/39f150ab23404df08bcf30da932a61da-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/796667d21d2f427398c16f3481acec99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/796667d21d2f427398c16f3481acec99-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18724,7 +18730,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8961189138f9437881c16068b3fa21e3-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8961189138f9437881c16068b3fa21e3-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/79d30f6c12ff48f59a00c227bb08ea12-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/79d30f6c12ff48f59a00c227bb08ea12-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18742,7 +18748,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/32f1e41eb98d49659b95ae052deb502c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/32f1e41eb98d49659b95ae052deb502c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/029d9f421b5541bc99cc6a346b186ed5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/029d9f421b5541bc99cc6a346b186ed5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18760,7 +18766,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0e96aefc36834548825822e7863c04b1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0e96aefc36834548825822e7863c04b1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/eae29e476a974143a7f7c6d4be1a3609-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/eae29e476a974143a7f7c6d4be1a3609-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18778,7 +18784,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/54f1c8ecb2bb49f2ab64ae3b3ff18f08-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/54f1c8ecb2bb49f2ab64ae3b3ff18f08-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e0911683e440433fa8334cf55a18b7c5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e0911683e440433fa8334cf55a18b7c5-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18796,7 +18802,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2ecc2deacaba40f398275166f8f87c5e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2ecc2deacaba40f398275166f8f87c5e-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0dcd281e376047d6b50c9792bb8e2657-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0dcd281e376047d6b50c9792bb8e2657-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18814,7 +18820,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/4047d114d1244cf88771a5dbdf9d3115-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/4047d114d1244cf88771a5dbdf9d3115-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2b04df5f36314eb5b86470a566a4c887-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2b04df5f36314eb5b86470a566a4c887-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18832,7 +18838,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/836bbe10e2c247b081cbbe63845cdbd5-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/836bbe10e2c247b081cbbe63845cdbd5-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e28189a287c640ad94796db33335544b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e28189a287c640ad94796db33335544b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18850,7 +18856,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c44647beea064db590b8f48ce1d77feb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c44647beea064db590b8f48ce1d77feb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0f6f1136bca445fd8eb6ac3a7982405c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0f6f1136bca445fd8eb6ac3a7982405c-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18868,7 +18874,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5be9b3890ab34761933ff0b7182effcb-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5be9b3890ab34761933ff0b7182effcb-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/072ee0798b8c45fcad77a52aa712daab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/072ee0798b8c45fcad77a52aa712daab-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18886,7 +18892,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ee1ea0337bda494f876da5f75ee03ef4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ee1ea0337bda494f876da5f75ee03ef4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e4df141553df49d9a51ef9443e1a1e0a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e4df141553df49d9a51ef9443e1a1e0a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18904,7 +18910,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1cae732553cc4f27877f991ad24cd244-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1cae732553cc4f27877f991ad24cd244-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dab36f1ad36142098b371a5538051170-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dab36f1ad36142098b371a5538051170-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18922,7 +18928,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6ab2b8813e0f48f3b830275620cd8f66-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6ab2b8813e0f48f3b830275620cd8f66-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ae6663b8b18a4c13a07344ce6269d15f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ae6663b8b18a4c13a07344ce6269d15f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18940,7 +18946,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2e7cc11b50e147fc83011d31f6133e6f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2e7cc11b50e147fc83011d31f6133e6f-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/a760dd896fa342c6bb52c20f875c773a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/a760dd896fa342c6bb52c20f875c773a-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18958,7 +18964,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1872ff1ae4a541df821ec8e3a2f81888-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1872ff1ae4a541df821ec8e3a2f81888-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b188b84f538c4aaca8e1fab064d46df6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b188b84f538c4aaca8e1fab064d46df6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18976,7 +18982,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bce9494c45034fe5a7e7ea7dc5f65aa1-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bce9494c45034fe5a7e7ea7dc5f65aa1-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/264851f02e2e4f349dc78aa9f058a5e7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/264851f02e2e4f349dc78aa9f058a5e7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -18994,7 +19000,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1177137dca7840eca704fba9db363a23-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1177137dca7840eca704fba9db363a23-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/57d2190f850a469b96cc374b688c28ed-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/57d2190f850a469b96cc374b688c28ed-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19012,7 +19018,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0ecdde0f25f3453e8bd8857570481de2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0ecdde0f25f3453e8bd8857570481de2-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ffe2b672fbf84fa9b4937823039f2340-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ffe2b672fbf84fa9b4937823039f2340-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19030,7 +19036,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/502f16f218734e199a2550f36f3dd528-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/502f16f218734e199a2550f36f3dd528-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ffa7ab38abdb453398366461c743a870-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ffa7ab38abdb453398366461c743a870-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19048,7 +19054,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/030b47e670c54842a00583c46b087a99-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/030b47e670c54842a00583c46b087a99-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6d28bebf4cb74f0a85ee4280e8573f23-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6d28bebf4cb74f0a85ee4280e8573f23-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19066,7 +19072,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/ea641d9885d44936ba5bb8fac3b60ef0-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/ea641d9885d44936ba5bb8fac3b60ef0-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d74c697a163e439b8f683accde90ea18-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d74c697a163e439b8f683accde90ea18-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19084,7 +19090,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1ad010d265794b3eb0dc70796f3daa2c-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1ad010d265794b3eb0dc70796f3daa2c-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/dba383a585cf4f9486b65b30a014bd8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/dba383a585cf4f9486b65b30a014bd8f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19102,7 +19108,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc6423e409654bdc9656630be17c9796-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc6423e409654bdc9656630be17c9796-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/850e0e0da43e4101bd707bc7238c66c6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/850e0e0da43e4101bd707bc7238c66c6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19120,7 +19126,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/2711d69a600d4a8295bcec285acafb97-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/2711d69a600d4a8295bcec285acafb97-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0d150b0109a948c4baa8ec56839573a4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/0d150b0109a948c4baa8ec56839573a4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19138,7 +19144,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/00b7f6cdc0e342f0980e186a07c96fad-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/00b7f6cdc0e342f0980e186a07c96fad-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/29cba231ec8e4b55a4c91fdbe15b64d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/29cba231ec8e4b55a4c91fdbe15b64d6-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19156,7 +19162,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/cf510f4c7e2d4f3780ea6caec9895c16-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/cf510f4c7e2d4f3780ea6caec9895c16-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c9e84acb769403e8ef057f10237deb7-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c9e84acb769403e8ef057f10237deb7-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19174,7 +19180,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9c2dce21f8494000b6efadbd5e860ce4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9c2dce21f8494000b6efadbd5e860ce4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d77939a21457455f9be35bfd44016307-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d77939a21457455f9be35bfd44016307-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19192,7 +19198,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/d80b90f5b2b746c09e20e727bf783ca8-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/d80b90f5b2b746c09e20e727bf783ca8-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/6ddaaeb87dea496893c002a84582f897-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/6ddaaeb87dea496893c002a84582f897-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19210,7 +19216,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/654ea3c66a064618b31a2a4dae755ca4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/654ea3c66a064618b31a2a4dae755ca4-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/068956a72ec3446cb9a6801260a59bac-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/068956a72ec3446cb9a6801260a59bac-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19228,7 +19234,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/97afdc224e1b4396873dc19ca2530259-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/97afdc224e1b4396873dc19ca2530259-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/8211c9dc26004a548c56adabcee7e7aa-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/8211c9dc26004a548c56adabcee7e7aa-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19246,7 +19252,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/5972f849c3e240a5be6cfd4982239564-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/5972f849c3e240a5be6cfd4982239564-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8d0c6fe33bb497292522bcc090fe652-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8d0c6fe33bb497292522bcc090fe652-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19264,7 +19270,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/b8ebefe52dbd48aca8f76d371f73f432-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/b8ebefe52dbd48aca8f76d371f73f432-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e5d94f8891394b38b2b2d0b75812c437-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e5d94f8891394b38b2b2d0b75812c437-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19282,7 +19288,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/9c1dad9b78414db795c9086000273693-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/9c1dad9b78414db795c9086000273693-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/fc8d1624a4374eec9ee5cf801307173f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/fc8d1624a4374eec9ee5cf801307173f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19300,7 +19306,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/3c50eaad901d4044b932b32454a37619-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/3c50eaad901d4044b932b32454a37619-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/53eaa766625941788baa0892548ffb6f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/53eaa766625941788baa0892548ffb6f-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19318,7 +19324,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/666b60fb80584ca0acb464ce1e1c026a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/666b60fb80584ca0acb464ce1e1c026a-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/09bdd6000096436684aeb199695e14c4-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/09bdd6000096436684aeb199695e14c4-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19336,7 +19342,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/1ce5df4e992e47ad99eb3ed171d4e7d6-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/1ce5df4e992e47ad99eb3ed171d4e7d6-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/af3ea97053a048898cf55c0f9c856f8b-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/af3ea97053a048898cf55c0f9c856f8b-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19354,7 +19360,7 @@
"Version: 2.10.3 \n",
"Build Date: Dec 15 2019 \n",
"\n",
- "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/66c9494d612a4e1d9afa347d17b8d839-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/66c9494d612a4e1d9afa347d17b8d839-pulp.sol (default strategy 1)\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/db9586d7ac6f447fa5af3301badf06e2-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/db9586d7ac6f447fa5af3301badf06e2-pulp.sol (default strategy 1)\n",
"At line 2 NAME MODEL\n",
"At line 3 ROWS\n",
"At line 11 COLUMNS\n",
@@ -19377,7 +19383,7 @@
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 7,
"metadata": {},
"outputs": [
{
@@ -19386,7 +19392,7 @@
"True"
]
},
- "execution_count": 6,
+ "execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
@@ -19411,9 +19417,21 @@
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 8,
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "ename": "ImportError",
+ "evalue": "cannot import name 'check_1_dataset_known_folds_mos_scores' from 'mlscorecheck.check' (/home/gykovacs/workspaces/mlscorecheck/mlscorecheck/check/__init__.py)",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)",
+ "\u001b[1;32m/home/gykovacs/workspaces/mlscorecheck/notebooks/illustration/bundles/07_tpehg.ipynb Cell 11\u001b[0m line \u001b[0;36m2\n\u001b[1;32m 1\u001b[0m \u001b[39mfrom\u001b[39;00m \u001b[39mmlscorecheck\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39maggregated\u001b[39;00m \u001b[39mimport\u001b[39;00m fold_partitioning_generator\n\u001b[0;32m----> 2\u001b[0m \u001b[39mfrom\u001b[39;00m \u001b[39mmlscorecheck\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39mcheck\u001b[39;00m \u001b[39mimport\u001b[39;00m check_1_dataset_known_folds_mos_scores, check_1_dataset_unknown_folds_mos_scores\n",
+ "\u001b[0;31mImportError\u001b[0m: cannot import name 'check_1_dataset_known_folds_mos_scores' from 'mlscorecheck.check' (/home/gykovacs/workspaces/mlscorecheck/mlscorecheck/check/__init__.py)"
+ ]
+ }
+ ],
"source": [
"from mlscorecheck.aggregated import fold_partitioning_generator\n",
"from mlscorecheck.check import check_1_dataset_known_folds_mos_scores, check_1_dataset_unknown_folds_mos_scores"
@@ -19421,7 +19439,7 @@
},
{
"cell_type": "code",
- "execution_count": 8,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@@ -19432,7 +19450,7 @@
},
{
"cell_type": "code",
- "execution_count": 9,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@@ -19446,7 +19464,7 @@
},
{
"cell_type": "code",
- "execution_count": 10,
+ "execution_count": null,
"metadata": {},
"outputs": [
{
@@ -19466,7 +19484,7 @@
},
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": null,
"metadata": {},
"outputs": [
{
@@ -19486,7 +19504,7 @@
},
{
"cell_type": "code",
- "execution_count": 12,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@@ -19499,7 +19517,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": null,
"metadata": {},
"outputs": [
{
@@ -19516,7 +19534,7 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@@ -19533,7 +19551,7 @@
},
{
"cell_type": "code",
- "execution_count": 15,
+ "execution_count": null,
"metadata": {},
"outputs": [
{
@@ -19553,7 +19571,7 @@
},
{
"cell_type": "code",
- "execution_count": 16,
+ "execution_count": null,
"metadata": {},
"outputs": [
{
diff --git a/notebooks/illustration/28_isic2016.ipynb b/notebooks/illustration/bundles/08_isic2016.ipynb
similarity index 56%
rename from notebooks/illustration/28_isic2016.ipynb
rename to notebooks/illustration/bundles/08_isic2016.ipynb
index 400d3f0..a58cf67 100644
--- a/notebooks/illustration/28_isic2016.ipynb
+++ b/notebooks/illustration/bundles/08_isic2016.ipynb
@@ -13,7 +13,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.skinlesion import check_isic2016"
+ "from mlscorecheck.check.bundles.skinlesion import check_isic2016"
]
},
{
@@ -34,30 +34,30 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:44:09,399:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:44:09,401:INFO:calling the score check with scores {'acc': 0.7916, 'sens': 0.2933, 'spec': 0.9145}, uncertainty 0.0001, p 75 and n 304\n",
- "2023-10-26 23:44:09,402:INFO:checking the scores {'acc': 0.7916, 'sens': 0.2933, 'spec': 0.9145}\n",
- "2023-10-26 23:44:09,404:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:44:09,404:INFO:intervals before: (0, 75), (0, 304)\n",
- "2023-10-26 23:44:09,405:INFO:the tp solutions: (21.98985, 22.00515)\n",
- "2023-10-26 23:44:09,407:INFO:the tn solutions: (277.97259199999996, 278.06520800000004)\n",
- "2023-10-26 23:44:09,409:INFO:intervals after: [(22, 22)], [(278, 278)]\n",
- "2023-10-26 23:44:09,410:INFO:evaluating the tp and tn solution for acc and spec\n",
- "2023-10-26 23:44:09,410:INFO:intervals before: [(22, 22)], [(278, 278)]\n",
- "2023-10-26 23:44:09,411:INFO:the tp solutions: (21.938733999999954, 22.078066000000035)\n",
- "2023-10-26 23:44:09,412:INFO:the tn solutions: (277.976992, 278.039008)\n",
- "2023-10-26 23:44:09,413:INFO:intervals after: [(22, 22)], [(278, 278)]\n",
- "2023-10-26 23:44:09,414:INFO:evaluating the tp and tn solution for sens and spec\n",
- "2023-10-26 23:44:09,415:INFO:intervals before: [(22, 22)], [(278, 278)]\n",
- "2023-10-26 23:44:09,415:INFO:the tp solutions: (21.98985, 22.00515)\n",
- "2023-10-26 23:44:09,416:INFO:the tn solutions: (277.976992, 278.039008)\n",
- "2023-10-26 23:44:09,417:INFO:intervals after: [(22, 22)], [(278, 278)]\n",
- "2023-10-26 23:44:09,419:INFO:testing acc, feasible tptn pairs: 1\n",
- "2023-10-26 23:44:09,421:INFO:testing sens, feasible tptn pairs: 1\n",
- "2023-10-26 23:44:09,421:INFO:testing spec, feasible tptn pairs: 1\n",
- "2023-10-26 23:44:09,422:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:44:09,423:INFO:final number of intervals: 1\n",
- "2023-10-26 23:44:09,424:INFO:final number of pairs: 1\n"
+ "2023-11-03 08:52:30,546:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:52:30,547:INFO:calling the score check with scores {'acc': 0.7916, 'sens': 0.2933, 'spec': 0.9145}, uncertainty 0.0001, p 75 and n 304\n",
+ "2023-11-03 08:52:30,549:INFO:checking the scores {'acc': 0.7916, 'sens': 0.2933, 'spec': 0.9145}\n",
+ "2023-11-03 08:52:30,551:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:52:30,552:INFO:intervals before: (0, 75), (0, 304)\n",
+ "2023-11-03 08:52:30,554:INFO:the tp solutions: (21.98985, 22.00515)\n",
+ "2023-11-03 08:52:30,555:INFO:the tn solutions: (277.97259199999996, 278.06520800000004)\n",
+ "2023-11-03 08:52:30,557:INFO:intervals after: [(22, 22)], [(278, 278)]\n",
+ "2023-11-03 08:52:30,558:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-03 08:52:30,559:INFO:intervals before: [(22, 22)], [(278, 278)]\n",
+ "2023-11-03 08:52:30,560:INFO:the tp solutions: (21.938733999999954, 22.078066000000035)\n",
+ "2023-11-03 08:52:30,561:INFO:the tn solutions: (277.976992, 278.039008)\n",
+ "2023-11-03 08:52:30,561:INFO:intervals after: [(22, 22)], [(278, 278)]\n",
+ "2023-11-03 08:52:30,562:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-03 08:52:30,565:INFO:intervals before: [(22, 22)], [(278, 278)]\n",
+ "2023-11-03 08:52:30,566:INFO:the tp solutions: (21.98985, 22.00515)\n",
+ "2023-11-03 08:52:30,567:INFO:the tn solutions: (277.976992, 278.039008)\n",
+ "2023-11-03 08:52:30,567:INFO:intervals after: [(22, 22)], [(278, 278)]\n",
+ "2023-11-03 08:52:30,568:INFO:testing acc, feasible tptn pairs: 1\n",
+ "2023-11-03 08:52:30,569:INFO:testing sens, feasible tptn pairs: 1\n",
+ "2023-11-03 08:52:30,570:INFO:testing spec, feasible tptn pairs: 1\n",
+ "2023-11-03 08:52:30,571:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:52:30,572:INFO:final number of intervals: 1\n",
+ "2023-11-03 08:52:30,574:INFO:final number of pairs: 1\n"
]
},
{
diff --git a/notebooks/illustration/29_isic2017.ipynb b/notebooks/illustration/bundles/09_isic2017.ipynb
similarity index 50%
rename from notebooks/illustration/29_isic2017.ipynb
rename to notebooks/illustration/bundles/09_isic2017.ipynb
index 72f4202..43aee2d 100644
--- a/notebooks/illustration/29_isic2017.ipynb
+++ b/notebooks/illustration/bundles/09_isic2017.ipynb
@@ -13,7 +13,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from mlscorecheck.bundles.skinlesion import check_isic2017"
+ "from mlscorecheck.check.bundles.skinlesion import check_isic2017"
]
},
{
@@ -34,46 +34,46 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "2023-10-26 23:44:15,476:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
- "2023-10-26 23:44:15,478:INFO:calling the score check with scores {'acc': 0.6183, 'sens': 0.4957, 'ppv': 0.2544, 'f1p': 0.3362}, uncertainty 0.0001, p 117 and n 483\n",
- "2023-10-26 23:44:15,478:INFO:checking the scores {'acc': 0.6183, 'sens': 0.4957, 'ppv': 0.2544, 'f1p': 0.3362}\n",
- "2023-10-26 23:44:15,479:INFO:evaluating the tp and tn solution for acc and sens\n",
- "2023-10-26 23:44:15,480:INFO:intervals before: (0, 117), (0, 483)\n",
- "2023-10-26 23:44:15,480:INFO:the tp solutions: (57.984966, 58.00883399999999)\n",
- "2023-10-26 23:44:15,481:INFO:the tn solutions: (312.90996599999994, 313.056234)\n",
- "2023-10-26 23:44:15,482:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,483:INFO:evaluating the tp and tn solution for acc and ppv\n",
- "2023-10-26 23:44:15,484:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,485:INFO:the tp solutions: (57.93786563072341, 58.095971377363604)\n",
- "2023-10-26 23:44:15,486:INFO:the tn solutions: (312.4997080154006, 313.4269057132848)\n",
- "2023-10-26 23:44:15,488:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,489:INFO:evaluating the tp and tn solution for acc and f1p\n",
- "2023-10-26 23:44:15,490:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,491:INFO:the tp solutions: (57.95478456338436, 58.03879605061339)\n",
- "2023-10-26 23:44:15,492:INFO:the tn solutions: (312.7528628906073, 313.2136402080464)\n",
- "2023-10-26 23:44:15,492:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,494:INFO:evaluating the tp and tn solution for sens and ppv\n",
- "2023-10-26 23:44:15,495:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,496:INFO:the tp solutions: (57.984966, 58.00883399999999)\n",
- "2023-10-26 23:44:15,496:INFO:the tn solutions: (312.6205722700333, 313.4230873646981)\n",
- "2023-10-26 23:44:15,497:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,498:INFO:evaluating the tp and tn solution for sens and f1p\n",
- "2023-10-26 23:44:15,499:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,500:INFO:the tp solutions: (57.984966, 58.00883399999999)\n",
- "2023-10-26 23:44:15,501:INFO:the tn solutions: (312.605227154962, 313.3602547229319)\n",
- "2023-10-26 23:44:15,503:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,504:INFO:evaluating the tp and tn solution for ppv and f1p\n",
- "2023-10-26 23:44:15,506:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,507:INFO:the tp solutions: (57.83419173925718, 58.121438922237544)\n",
- "2023-10-26 23:44:15,507:INFO:the tn solutions: (311.5597304712854, 314.6024074992048)\n",
- "2023-10-26 23:44:15,508:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
- "2023-10-26 23:44:15,510:INFO:testing acc, feasible tptn pairs: 1\n",
- "2023-10-26 23:44:15,511:INFO:testing sens, feasible tptn pairs: 1\n",
- "2023-10-26 23:44:15,512:INFO:testing ppv, feasible tptn pairs: 1\n",
- "2023-10-26 23:44:15,513:INFO:testing f1p, feasible tptn pairs: 1\n",
- "2023-10-26 23:44:15,514:INFO:constructing final tp, tn pair set\n",
- "2023-10-26 23:44:15,515:INFO:final number of intervals: 1\n",
- "2023-10-26 23:44:15,516:INFO:final number of pairs: 1\n"
+ "2023-11-03 08:52:48,649:INFO:Use this function if the scores originate from the tp and tn statistics calculated on one test set with no aggregation of any kind.\n",
+ "2023-11-03 08:52:48,650:INFO:calling the score check with scores {'acc': 0.6183, 'sens': 0.4957, 'ppv': 0.2544, 'f1p': 0.3362}, uncertainty 0.0001, p 117 and n 483\n",
+ "2023-11-03 08:52:48,651:INFO:checking the scores {'acc': 0.6183, 'sens': 0.4957, 'ppv': 0.2544, 'f1p': 0.3362}\n",
+ "2023-11-03 08:52:48,652:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-03 08:52:48,653:INFO:intervals before: (0, 117), (0, 483)\n",
+ "2023-11-03 08:52:48,654:INFO:the tp solutions: (57.984966, 58.00883399999999)\n",
+ "2023-11-03 08:52:48,656:INFO:the tn solutions: (312.90996599999994, 313.056234)\n",
+ "2023-11-03 08:52:48,657:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,658:INFO:evaluating the tp and tn solution for acc and ppv\n",
+ "2023-11-03 08:52:48,659:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,660:INFO:the tp solutions: (57.93786563072341, 58.095971377363604)\n",
+ "2023-11-03 08:52:48,660:INFO:the tn solutions: (312.4997080154006, 313.4269057132848)\n",
+ "2023-11-03 08:52:48,662:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,663:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-03 08:52:48,664:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,665:INFO:the tp solutions: (57.95478456338436, 58.03879605061339)\n",
+ "2023-11-03 08:52:48,666:INFO:the tn solutions: (312.7528628906073, 313.2136402080464)\n",
+ "2023-11-03 08:52:48,666:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,667:INFO:evaluating the tp and tn solution for sens and ppv\n",
+ "2023-11-03 08:52:48,668:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,669:INFO:the tp solutions: (57.984966, 58.00883399999999)\n",
+ "2023-11-03 08:52:48,669:INFO:the tn solutions: (312.6205722700333, 313.4230873646981)\n",
+ "2023-11-03 08:52:48,670:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,674:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-03 08:52:48,675:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,676:INFO:the tp solutions: (57.984966, 58.00883399999999)\n",
+ "2023-11-03 08:52:48,676:INFO:the tn solutions: (312.605227154962, 313.3602547229319)\n",
+ "2023-11-03 08:52:48,677:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,678:INFO:evaluating the tp and tn solution for ppv and f1p\n",
+ "2023-11-03 08:52:48,679:INFO:intervals before: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,680:INFO:the tp solutions: (57.83419173925718, 58.121438922237544)\n",
+ "2023-11-03 08:52:48,681:INFO:the tn solutions: (311.5597304712854, 314.6024074992048)\n",
+ "2023-11-03 08:52:48,682:INFO:intervals after: [(58, 58)], [(313, 313)]\n",
+ "2023-11-03 08:52:48,683:INFO:testing acc, feasible tptn pairs: 1\n",
+ "2023-11-03 08:52:48,685:INFO:testing sens, feasible tptn pairs: 1\n",
+ "2023-11-03 08:52:48,686:INFO:testing ppv, feasible tptn pairs: 1\n",
+ "2023-11-03 08:52:48,687:INFO:testing f1p, feasible tptn pairs: 1\n",
+ "2023-11-03 08:52:48,688:INFO:constructing final tp, tn pair set\n",
+ "2023-11-03 08:52:48,689:INFO:final number of intervals: 1\n",
+ "2023-11-03 08:52:48,691:INFO:final number of pairs: 1\n"
]
},
{
diff --git a/notebooks/illustration/multiclass/00_check_1_testset_no_kfold_macro.ipynb b/notebooks/illustration/multiclass/00_check_1_testset_no_kfold_macro.ipynb
new file mode 100644
index 0000000..665262e
--- /dev/null
+++ b/notebooks/illustration/multiclass/00_check_1_testset_no_kfold_macro.ipynb
@@ -0,0 +1,244 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.multiclass import check_1_testset_no_kfold_macro"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "testset = {0: 10, 1: 100, 2: 80}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = {'acc': 0.6, 'sens': 0.3417, 'spec': 0.6928, 'f1p': 0.3308}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/62d7116b71bc46a88a616d361418de8f-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/62d7116b71bc46a88a616d361418de8f-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 49 RHS\n",
+ "At line 56 BOUNDS\n",
+ "At line 64 ENDATA\n",
+ "Problem MODEL has 6 rows, 7 columns and 24 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0004I processed model has 3 rows, 6 columns (6 integer (0 of which binary)) and 12 elements\n",
+ "Cbc0031I 2 added rows had average density of 6\n",
+ "Cbc0013I At root node, 2 cuts changed objective from 0 to 0 in 39 passes\n",
+ "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 86 row cuts average 5.7 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 8 row cuts average 2.5 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 6 (TwoMirCuts) - 10 row cuts average 3.1 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.01 seconds)\n",
+ "Cbc0012I Integer solution of 0 found by DiveCoefficient after 81 iterations and 3 nodes (0.02 seconds)\n",
+ "Cbc0001I Search completed - best objective 0, took 81 iterations and 3 nodes (0.02 seconds)\n",
+ "Cbc0032I Strong branching done 20 times (30 iterations), fathomed 0 nodes and fixed 4 variables\n",
+ "Cbc0035I Maximum depth 3, 0 variables fixed on reduced cost\n",
+ "Cuts at root node changed objective from 0 to 0\n",
+ "Probing was tried 39 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Gomory was tried 39 times and created 86 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Knapsack was tried 39 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Clique was tried 39 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 39 times and created 8 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 39 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 39 times and created 10 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "\n",
+ "Result - Optimal solution found\n",
+ "\n",
+ "Objective value: 0.00000000\n",
+ "Enumerated nodes: 3\n",
+ "Total iterations: 81\n",
+ "Time (CPU seconds): 0.02\n",
+ "Time (Wallclock seconds): 0.02\n",
+ "\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.02 (Wallclock seconds): 0.02\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_1_testset_no_kfold_macro(scores=scores, testset=testset, eps=1e-4)\n",
+ "results['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'acc': 0.6, 'sens': 0.3417, 'spec': 0.6928, 'f1p': 0.3308}"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['acc'] = 0.6020"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/f1cf50867bd542b1987d654e01fbe5ab-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/f1cf50867bd542b1987d654e01fbe5ab-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 49 RHS\n",
+ "At line 56 BOUNDS\n",
+ "At line 64 ENDATA\n",
+ "Problem MODEL has 6 rows, 7 columns and 24 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0004I processed model has 3 rows, 6 columns (6 integer (0 of which binary)) and 12 elements\n",
+ "Cbc0031I 2 added rows had average density of 6\n",
+ "Cbc0013I At root node, 2 cuts changed objective from 0 to 0 in 37 passes\n",
+ "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 90 row cuts average 5.8 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 3 row cuts average 2.3 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 6 (TwoMirCuts) - 8 row cuts average 3.5 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.01 seconds)\n",
+ "Cbc0010I After 1000 nodes, 2 on tree, 1e+50 best solution, best possible 0 (0.29 seconds)\n",
+ "Cbc0001I Search completed - best objective 1e+50, took 69036 iterations and 45721 nodes (0.32 seconds)\n",
+ "Cbc0032I Strong branching done 2622 times (4495 iterations), fathomed 179 nodes and fixed 418 variables\n",
+ "Cbc0041I Maximum depth 29, 0 variables fixed on reduced cost (complete fathoming 141 times, 44653 nodes taking 60327 iterations)\n",
+ "Cuts at root node changed objective from 0 to 0\n",
+ "Probing was tried 37 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Gomory was tried 37 times and created 90 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Knapsack was tried 37 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Clique was tried 37 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 37 times and created 3 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 37 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 37 times and created 8 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "\n",
+ "Result - Problem proven infeasible\n",
+ "\n",
+ "No feasible solution found\n",
+ "Enumerated nodes: 45721\n",
+ "Total iterations: 69036\n",
+ "Time (CPU seconds): 0.32\n",
+ "Time (Wallclock seconds): 0.32\n",
+ "\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.32 (Wallclock seconds): 0.32\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 25,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_1_testset_no_kfold_macro(scores=scores, testset=testset, eps=1e-4)\n",
+ "results['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/multiclass/01_check_1_testset_no_kfold_micro.ipynb b/notebooks/illustration/multiclass/01_check_1_testset_no_kfold_micro.ipynb
new file mode 100644
index 0000000..91eeb3f
--- /dev/null
+++ b/notebooks/illustration/multiclass/01_check_1_testset_no_kfold_micro.ipynb
@@ -0,0 +1,318 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.multiclass import check_1_testset_no_kfold_micro"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "testset = {0: 10, 1: 100, 2: 80}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = {'acc': 0.5158, 'sens': 0.2737, 'spec': 0.6368,\n",
+ " 'bacc': 0.4553, 'ppv': 0.2737, 'npv': 0.6368}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-04 11:34:20,704:INFO:checking the scores {'acc': 0.5158, 'sens': 0.2737, 'spec': 0.6368, 'bacc': 0.4553, 'ppv': 0.2737, 'npv': 0.6368}\n",
+ "2023-11-04 11:34:20,705:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-04 11:34:20,706:INFO:intervals before: (0, 190), (0, 380)\n",
+ "2023-11-04 11:34:20,707:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,709:INFO:the tn solutions: (241.92548, 242.08052000000006)\n",
+ "2023-11-04 11:34:20,709:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,710:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-04 11:34:20,711:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,712:INFO:the tp solutions: (51.92509999999996, 52.11890000000008)\n",
+ "2023-11-04 11:34:20,713:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,714:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,716:INFO:evaluating the tp and tn solution for acc and bacc\n",
+ "2023-11-04 11:34:20,717:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,718:INFO:the tp solutions: (51.886339999999905, 52.157659999999964)\n",
+ "2023-11-04 11:34:20,719:INFO:the tn solutions: (241.79020000000003, 242.17780000000016)\n",
+ "2023-11-04 11:34:20,719:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,720:INFO:evaluating the tp and tn solution for acc and ppv\n",
+ "2023-11-04 11:34:20,721:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,722:INFO:the tp solutions: (51.92507006183688, 52.081026437634286)\n",
+ "2023-11-04 11:34:20,722:INFO:the tn solutions: (241.57858662847494, 242.42776991777123)\n",
+ "2023-11-04 11:34:20,723:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,724:INFO:evaluating the tp and tn solution for acc and npv\n",
+ "2023-11-04 11:34:20,725:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,726:INFO:the tp solutions: (51.36701642883198, 52.50098353406811)\n",
+ "2023-11-04 11:34:20,727:INFO:the tn solutions: (241.71814351243938, 242.4273174965254)\n",
+ "2023-11-04 11:34:20,729:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,730:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-04 11:34:20,730:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,731:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,732:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,733:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,733:INFO:evaluating the tp and tn solution for sens and bacc\n",
+ "2023-11-04 11:34:20,734:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,735:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,736:INFO:the tn solutions: (241.90572, 242.13827999999998)\n",
+ "2023-11-04 11:34:20,737:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,738:INFO:evaluating the tp and tn solution for sens and ppv\n",
+ "2023-11-04 11:34:20,738:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,740:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,741:INFO:the tn solutions: (241.66176457717626, 242.3445043047098)\n",
+ "2023-11-04 11:34:20,742:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,742:INFO:evaluating the tp and tn solution for sens and npv\n",
+ "2023-11-04 11:34:20,743:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,745:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,747:INFO:the tn solutions: (241.81004976234647, 242.0914145898904)\n",
+ "2023-11-04 11:34:20,751:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,751:INFO:evaluating the tp and tn solution for spec and bacc\n",
+ "2023-11-04 11:34:20,753:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,753:INFO:the tp solutions: (51.96385999999998, 52.08013999999999)\n",
+ "2023-11-04 11:34:20,754:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,755:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,756:INFO:evaluating the tp and tn solution for spec and ppv\n",
+ "2023-11-04 11:34:20,756:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,758:INFO:the tp solutions: (51.96887798976324, 52.0514644732153)\n",
+ "2023-11-04 11:34:20,759:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,759:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,760:INFO:evaluating the tp and tn solution for spec and npv\n",
+ "2023-11-04 11:34:20,761:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,762:INFO:the tp solutions: (51.806887743357564, 52.161181422778235)\n",
+ "2023-11-04 11:34:20,763:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,764:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,765:INFO:evaluating the tp and tn solution for bacc and ppv\n",
+ "2023-11-04 11:34:20,766:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,767:INFO:the tp solutions: (51.74749695345026, 52.201231144607334)\n",
+ "2023-11-04 11:34:20,768:INFO:the tn solutions: (240.48623602424024, 243.67941013113506)\n",
+ "2023-11-04 11:34:20,769:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,769:INFO:evaluating the tp and tn solution for bacc and npv\n",
+ "2023-11-04 11:34:20,770:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,771:INFO:the tp solutions: (50.094566557959965, 54.50446037762892)\n",
+ "2023-11-04 11:34:20,771:INFO:the tn solutions: (240.03456529085898, 242.86298504983597)\n",
+ "2023-11-04 11:34:20,772:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,773:INFO:evaluating the tp and tn solution for ppv and npv\n",
+ "2023-11-04 11:34:20,773:INFO:intervals before: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,774:INFO:the tp solutions: (51.74602430527055, 52.37775596084947)\n",
+ "2023-11-04 11:34:20,775:INFO:the tn solutions: (240.84749951261927, 242.85483520292078)\n",
+ "2023-11-04 11:34:20,776:INFO:intervals after: [(52, 52)], [(242, 242)]\n",
+ "2023-11-04 11:34:20,777:INFO:testing acc, feasible tptn pairs: 1\n",
+ "2023-11-04 11:34:20,778:INFO:testing sens, feasible tptn pairs: 1\n",
+ "2023-11-04 11:34:20,779:INFO:testing spec, feasible tptn pairs: 1\n",
+ "2023-11-04 11:34:20,781:INFO:testing bacc, feasible tptn pairs: 1\n",
+ "2023-11-04 11:34:20,782:INFO:testing npv, feasible tptn pairs: 1\n",
+ "2023-11-04 11:34:20,784:INFO:testing ppv, feasible tptn pairs: 1\n",
+ "2023-11-04 11:34:20,785:INFO:constructing final tp, tn pair set\n",
+ "2023-11-04 11:34:20,786:INFO:final number of intervals: 1\n",
+ "2023-11-04 11:34:20,787:INFO:final number of pairs: 1\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_1_testset_no_kfold_micro(testset=testset,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "results['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'acc': 0.5158,\n",
+ " 'sens': 0.2737,\n",
+ " 'spec': 0.6368,\n",
+ " 'bacc': 0.4553,\n",
+ " 'ppv': 0.2737,\n",
+ " 'npv': 0.6368}"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['acc'] = 0.5258"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-04 11:34:20,856:INFO:checking the scores {'acc': 0.5258, 'sens': 0.2737, 'spec': 0.6368, 'bacc': 0.4553, 'ppv': 0.2737, 'npv': 0.6368}\n",
+ "2023-11-04 11:34:20,857:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-04 11:34:20,858:INFO:intervals before: (0, 190), (0, 380)\n",
+ "2023-11-04 11:34:20,859:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,860:INFO:the tn solutions: (247.62548000000004, 247.78052000000005)\n",
+ "2023-11-04 11:34:20,862:INFO:intervals after: [(52, 52)], [(1, 0)]\n",
+ "2023-11-04 11:34:20,863:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-04 11:34:20,864:INFO:intervals before: [(52, 52)], [(1, 0)]\n",
+ "2023-11-04 11:34:20,865:INFO:the tp solutions: (57.6251, 57.81890000000007)\n",
+ "2023-11-04 11:34:20,866:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,867:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,867:INFO:evaluating the tp and tn solution for acc and bacc\n",
+ "2023-11-04 11:34:20,868:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,869:INFO:the tp solutions: (46.18633999999993, 46.45765999999995)\n",
+ "2023-11-04 11:34:20,870:INFO:the tn solutions: (253.19020000000012, 253.57780000000014)\n",
+ "2023-11-04 11:34:20,871:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,871:INFO:evaluating the tp and tn solution for acc and ppv\n",
+ "2023-11-04 11:34:20,872:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,873:INFO:the tp solutions: (48.48095605224332, 48.631235988558664)\n",
+ "2023-11-04 11:34:20,879:INFO:the tn solutions: (250.7201326395527, 251.58013068134997)\n",
+ "2023-11-04 11:34:20,880:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,881:INFO:evaluating the tp and tn solution for acc and npv\n",
+ "2023-11-04 11:34:20,883:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,884:INFO:the tp solutions: (43.80386395479962, 44.93079743039423)\n",
+ "2023-11-04 11:34:20,884:INFO:the tn solutions: (254.97280231946934, 255.70600994264748)\n",
+ "2023-11-04 11:34:20,885:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,885:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-04 11:34:20,886:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,887:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,887:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,888:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,889:INFO:evaluating the tp and tn solution for sens and bacc\n",
+ "2023-11-04 11:34:20,890:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,891:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,892:INFO:the tn solutions: (241.90572, 242.13827999999998)\n",
+ "2023-11-04 11:34:20,893:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,894:INFO:evaluating the tp and tn solution for sens and ppv\n",
+ "2023-11-04 11:34:20,896:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,897:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,898:INFO:the tn solutions: (241.66176457717626, 242.3445043047098)\n",
+ "2023-11-04 11:34:20,899:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,899:INFO:evaluating the tp and tn solution for sens and npv\n",
+ "2023-11-04 11:34:20,900:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,901:INFO:the tp solutions: (51.98362, 52.02238)\n",
+ "2023-11-04 11:34:20,902:INFO:the tn solutions: (241.81004976234647, 242.0914145898904)\n",
+ "2023-11-04 11:34:20,902:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,903:INFO:evaluating the tp and tn solution for spec and bacc\n",
+ "2023-11-04 11:34:20,904:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,904:INFO:the tp solutions: (51.96385999999998, 52.08013999999999)\n",
+ "2023-11-04 11:34:20,905:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,905:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,906:INFO:evaluating the tp and tn solution for spec and ppv\n",
+ "2023-11-04 11:34:20,906:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,910:INFO:the tp solutions: (51.96887798976324, 52.0514644732153)\n",
+ "2023-11-04 11:34:20,912:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,913:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,914:INFO:evaluating the tp and tn solution for spec and npv\n",
+ "2023-11-04 11:34:20,915:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,915:INFO:the tp solutions: (51.806887743357564, 52.161181422778235)\n",
+ "2023-11-04 11:34:20,916:INFO:the tn solutions: (241.94523999999998, 242.02276000000003)\n",
+ "2023-11-04 11:34:20,916:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,917:INFO:evaluating the tp and tn solution for bacc and ppv\n",
+ "2023-11-04 11:34:20,918:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,919:INFO:the tp solutions: (51.74749695345026, 52.201231144607334)\n",
+ "2023-11-04 11:34:20,925:INFO:the tn solutions: (240.48623602424024, 243.67941013113506)\n",
+ "2023-11-04 11:34:20,925:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,926:INFO:evaluating the tp and tn solution for bacc and npv\n",
+ "2023-11-04 11:34:20,928:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,928:INFO:the tp solutions: (50.094566557959965, 54.50446037762892)\n",
+ "2023-11-04 11:34:20,929:INFO:the tn solutions: (240.03456529085898, 242.86298504983597)\n",
+ "2023-11-04 11:34:20,930:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,931:INFO:evaluating the tp and tn solution for ppv and npv\n",
+ "2023-11-04 11:34:20,932:INFO:intervals before: [], []\n",
+ "2023-11-04 11:34:20,933:INFO:the tp solutions: (51.74602430527055, 52.37775596084947)\n",
+ "2023-11-04 11:34:20,934:INFO:the tn solutions: (240.84749951261927, 242.85483520292078)\n",
+ "2023-11-04 11:34:20,935:INFO:intervals after: [], []\n",
+ "2023-11-04 11:34:20,939:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-04 11:34:20,940:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-04 11:34:20,940:INFO:constructing final tp, tn pair set\n",
+ "2023-11-04 11:34:20,941:INFO:final number of intervals: 0\n",
+ "2023-11-04 11:34:20,942:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "results = check_1_testset_no_kfold_micro(testset=testset,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "results['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/multiclass/02_check_1_dataset_known_folds_mos_macro.ipynb b/notebooks/illustration/multiclass/02_check_1_dataset_known_folds_mos_macro.ipynb
new file mode 100644
index 0000000..bae0883
--- /dev/null
+++ b/notebooks/illustration/multiclass/02_check_1_dataset_known_folds_mos_macro.ipynb
@@ -0,0 +1,207 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_macro\n",
+ "from mlscorecheck.aggregated import generate_dataset_folding_multiclass"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "({0: 71, 1: 38, 2: 21},\n",
+ " {'folds': [{0: 24, 1: 13, 2: 7}, {0: 24, 1: 12, 2: 7}, {0: 23, 1: 13, 2: 7}]},\n",
+ " {'acc': 0.5536, 'sens': 0.3247, 'spec': 0.6544, 'f1p': 0.3063})"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset, folding, scores = generate_dataset_folding_multiclass(random_state=None,\n",
+ " average='macro',\n",
+ " aggregation='mos',\n",
+ " rounding_decimals=4,\n",
+ " subset=['acc', 'sens', 'spec', 'f1p'])\n",
+ "dataset, folding, scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dataset = {0: 149, 1: 118, 2: 83, 3: 154}\n",
+ "folding = {'n_folds': 4, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}\n",
+ "scores = {'acc': 0.626, 'sens': 0.2483, 'spec': 0.7509, 'f1p': 0.2469}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/e602a5cac2494b4b83930f267d731f2e-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/e602a5cac2494b4b83930f267d731f2e-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 397 RHS\n",
+ "At line 404 BOUNDS\n",
+ "At line 470 ENDATA\n",
+ "Problem MODEL has 6 rows, 65 columns and 256 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0004I processed model has 3 rows, 14 columns (14 integer (0 of which binary)) and 28 elements\n",
+ "Cbc0012I Integer solution of 0 found by DiveCoefficient after 0 iterations and 0 nodes (0.00 seconds)\n",
+ "Cbc0001I Search completed - best objective 0, took 0 iterations and 0 nodes (0.00 seconds)\n",
+ "Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
+ "Cuts at root node changed objective from 0 to 0\n",
+ "Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "\n",
+ "Result - Optimal solution found\n",
+ "\n",
+ "Objective value: 0.00000000\n",
+ "Enumerated nodes: 0\n",
+ "Total iterations: 0\n",
+ "Time (CPU seconds): 0.00\n",
+ "Time (Wallclock seconds): 0.00\n",
+ "\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_mos_macro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['acc'] = 0.656"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/870a6e1de01f441c8178286af5968aae-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/870a6e1de01f441c8178286af5968aae-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 397 RHS\n",
+ "At line 404 BOUNDS\n",
+ "At line 470 ENDATA\n",
+ "Problem MODEL has 6 rows, 65 columns and 256 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Problem is infeasible - 0.00 seconds\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_mos_macro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/multiclass/03_check_1_dataset_known_folds_mos_micro.ipynb b/notebooks/illustration/multiclass/03_check_1_dataset_known_folds_mos_micro.ipynb
new file mode 100644
index 0000000..600b92b
--- /dev/null
+++ b/notebooks/illustration/multiclass/03_check_1_dataset_known_folds_mos_micro.ipynb
@@ -0,0 +1,248 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_micro\n",
+ "from mlscorecheck.aggregated import generate_dataset_folding_multiclass"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "({0: 99, 1: 29, 2: 119},\n",
+ " {'folds': [{0: 50, 1: 14, 2: 60}, {0: 49, 1: 15, 2: 59}]},\n",
+ " {'acc': 0.579, 'sens': 0.3685, 'spec': 0.6843, 'f1p': 0.3685})"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset, folding, scores = generate_dataset_folding_multiclass(random_state=None,\n",
+ " average='micro',\n",
+ " aggregation='mos',\n",
+ " rounding_decimals=4,\n",
+ " subset=['acc', 'sens', 'spec', 'f1p'])\n",
+ "dataset, folding, scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dataset = {0: 66, 1: 178, 2: 151}\n",
+ "folding = {'folds': [{0: 33, 1: 89, 2: 76}, {0: 33, 1: 89, 2: 75}]}\n",
+ "scores = {'acc': 0.5646, 'sens': 0.3469, 'spec': 0.6734, 'f1p': 0.3469}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/794c36f09f26415db85a540793d0d103-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/794c36f09f26415db85a540793d0d103-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 85 RHS\n",
+ "At line 92 BOUNDS\n",
+ "At line 106 ENDATA\n",
+ "Problem MODEL has 6 rows, 13 columns and 48 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0003I 0 fixed, 11 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 9 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 14 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 14 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 14 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 14 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 14 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 14 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 14 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 10 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0003I 0 fixed, 4 tightened bounds, 0 strengthened rows, 0 substitutions\n",
+ "Cgl0004I processed model has 3 rows, 4 columns (4 integer (0 of which binary)) and 8 elements\n",
+ "Cbc0012I Integer solution of 0 found by DiveCoefficient after 18 iterations and 0 nodes (0.01 seconds)\n",
+ "Cbc0031I 2 added rows had average density of 2\n",
+ "Cbc0013I At root node, 2 cuts changed objective from 0 to 0 in 33 passes\n",
+ "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 75 column cuts (75 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 19 row cuts average 2.4 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 6 (TwoMirCuts) - 15 row cuts average 2.1 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0001I Search completed - best objective 0, took 18 iterations and 0 nodes (0.01 seconds)\n",
+ "Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
+ "Cuts at root node changed objective from 0 to 0\n",
+ "Probing was tried 33 times and created 75 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Gomory was tried 33 times and created 19 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Knapsack was tried 33 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "Clique was tried 33 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 33 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "FlowCover was tried 33 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "TwoMirCuts was tried 33 times and created 15 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "\n",
+ "Result - Optimal solution found\n",
+ "\n",
+ "Objective value: 0.00000000\n",
+ "Enumerated nodes: 0\n",
+ "Total iterations: 18\n",
+ "Time (CPU seconds): 0.00\n",
+ "Time (Wallclock seconds): 0.01\n",
+ "\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.01\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_mos_micro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['acc'] = 0.5746"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/c519d509ea944541b893c7ab86b8178a-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/c519d509ea944541b893c7ab86b8178a-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 85 RHS\n",
+ "At line 92 BOUNDS\n",
+ "At line 106 ENDATA\n",
+ "Problem MODEL has 6 rows, 13 columns and 48 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Problem is infeasible - 0.00 seconds\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_mos_micro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/multiclass/04_check_1_dataset_known_folds_som_macro.ipynb b/notebooks/illustration/multiclass/04_check_1_dataset_known_folds_som_macro.ipynb
new file mode 100644
index 0000000..031e6f1
--- /dev/null
+++ b/notebooks/illustration/multiclass/04_check_1_dataset_known_folds_som_macro.ipynb
@@ -0,0 +1,223 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_macro\n",
+ "from mlscorecheck.aggregated import generate_dataset_folding_multiclass"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "({0: 26, 1: 167, 2: 72, 3: 156},\n",
+ " {'n_folds': 3, 'n_repeats': 1, 'strategy': 'stratified_sklearn'},\n",
+ " {'acc': 0.6211,\n",
+ " 'sens': 0.2695,\n",
+ " 'spec': 0.7509,\n",
+ " 'f1p': 0.2272,\n",
+ " 'beta_positive': 2,\n",
+ " 'beta_negative': 2})"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset, folding, scores = generate_dataset_folding_multiclass(random_state=None,\n",
+ " average='macro',\n",
+ " aggregation='som',\n",
+ " rounding_decimals=4,\n",
+ " subset=['acc', 'sens', 'spec', 'f1p'])\n",
+ "dataset, folding, scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dataset = {0: 129, 1: 81, 2: 135}\n",
+ "folding = {'n_folds': 2, 'n_repeats': 2, 'strategy': 'stratified_sklearn'}\n",
+ "scores = {'acc': 0.5662, 'sens': 0.3577, 'spec': 0.6767, 'f1p': 0.3481}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/791f13693e8f4b3399949d625fbc3460-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/791f13693e8f4b3399949d625fbc3460-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 49 RHS\n",
+ "At line 56 BOUNDS\n",
+ "At line 64 ENDATA\n",
+ "Problem MODEL has 6 rows, 7 columns and 24 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Continuous objective value is 0 - 0.00 seconds\n",
+ "Cgl0004I processed model has 3 rows, 6 columns (6 integer (0 of which binary)) and 12 elements\n",
+ "Cbc0031I 4 added rows had average density of 4.5\n",
+ "Cbc0013I At root node, 4 cuts changed objective from 0 to 0 in 50 passes\n",
+ "Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 1 (Gomory) - 84 row cuts average 5.2 elements, 0 column cuts (0 active) in 0.002 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.000 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 1 row cuts average 2.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0014I Cut generator 6 (TwoMirCuts) - 19 row cuts average 3.5 elements, 0 column cuts (0 active) in 0.001 seconds - new frequency is -100\n",
+ "Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 0 (0.02 seconds)\n",
+ "Cbc0012I Integer solution of 0 found by DiveCoefficient after 68 iterations and 1 nodes (0.02 seconds)\n",
+ "Cbc0001I Search completed - best objective 0, took 68 iterations and 1 nodes (0.02 seconds)\n",
+ "Cbc0032I Strong branching done 6 times (7 iterations), fathomed 0 nodes and fixed 0 variables\n",
+ "Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost\n",
+ "Cuts at root node changed objective from 0 to 0\n",
+ "Probing was tried 50 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Gomory was tried 50 times and created 84 cuts of which 0 were active after adding rounds of cuts (0.002 seconds)\n",
+ "Knapsack was tried 50 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "Clique was tried 50 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "MixedIntegerRounding2 was tried 50 times and created 1 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "FlowCover was tried 50 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "TwoMirCuts was tried 50 times and created 19 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)\n",
+ "ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
+ "\n",
+ "Result - Optimal solution found\n",
+ "\n",
+ "Objective value: 0.00000000\n",
+ "Enumerated nodes: 1\n",
+ "Total iterations: 68\n",
+ "Time (CPU seconds): 0.02\n",
+ "Time (Wallclock seconds): 0.02\n",
+ "\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.02 (Wallclock seconds): 0.02\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_som_macro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['acc'] = 0.6762"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Welcome to the CBC MILP Solver \n",
+ "Version: 2.10.3 \n",
+ "Build Date: Dec 15 2019 \n",
+ "\n",
+ "command line - /home/gykovacs/anaconda3/envs/mlscorecheck/lib/python3.10/site-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/bc4f50ab17ad4604b3b040ce2c6f2a15-pulp.mps timeMode elapsed branch printingOptions all solution /tmp/bc4f50ab17ad4604b3b040ce2c6f2a15-pulp.sol (default strategy 1)\n",
+ "At line 2 NAME MODEL\n",
+ "At line 3 ROWS\n",
+ "At line 11 COLUMNS\n",
+ "At line 49 RHS\n",
+ "At line 56 BOUNDS\n",
+ "At line 64 ENDATA\n",
+ "Problem MODEL has 6 rows, 7 columns and 24 elements\n",
+ "Coin0008I MODEL read with 0 errors\n",
+ "Option for timeMode changed from cpu to elapsed\n",
+ "Problem is infeasible - 0.00 seconds\n",
+ "Option for printingOptions changed from normal to all\n",
+ "Total time (CPU seconds): 0.00 (Wallclock seconds): 0.00\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_som_macro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/multiclass/05_check_1_dataset_known_folds_som_micro.ipynb b/notebooks/illustration/multiclass/05_check_1_dataset_known_folds_som_micro.ipynb
new file mode 100644
index 0000000..293e41a
--- /dev/null
+++ b/notebooks/illustration/multiclass/05_check_1_dataset_known_folds_som_micro.ipynb
@@ -0,0 +1,248 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_micro\n",
+ "from mlscorecheck.aggregated import generate_dataset_folding_multiclass"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "({0: 62, 1: 181, 2: 107, 3: 66},\n",
+ " {'n_folds': 3, 'n_repeats': 2, 'strategy': 'stratified_sklearn'},\n",
+ " {'acc': 0.6304,\n",
+ " 'sens': 0.2608,\n",
+ " 'spec': 0.7536,\n",
+ " 'f1p': 0.2608,\n",
+ " 'beta_positive': 2,\n",
+ " 'beta_negative': 2})"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataset, folding, scores = generate_dataset_folding_multiclass(random_state=None,\n",
+ " average='micro',\n",
+ " aggregation='som',\n",
+ " rounding_decimals=4,\n",
+ " subset=['acc', 'sens', 'spec', 'f1p'])\n",
+ "dataset, folding, scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dataset = {0: 86, 1: 96, 2: 59, 3: 105}\n",
+ "folding = {'folds': [{0: 43, 1: 48, 2: 30, 3: 52}, {0: 43, 1: 48, 2: 29, 3: 53}]}\n",
+ "scores = {'acc': 0.6272, 'sens': 0.2543, 'spec': 0.7514, 'f1p': 0.2543}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-04 11:32:09,641:INFO:checking the scores {'acc': 0.6272, 'sens': 0.2543, 'spec': 0.7514, 'f1p': 0.2543}\n",
+ "2023-11-04 11:32:09,644:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-04 11:32:09,645:INFO:intervals before: (0, 346), (0, 1038)\n",
+ "2023-11-04 11:32:09,646:INFO:the tp solutions: (87.95250800000001, 88.023092)\n",
+ "2023-11-04 11:32:09,647:INFO:the tn solutions: (779.8805399999999, 780.23346)\n",
+ "2023-11-04 11:32:09,648:INFO:intervals after: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,649:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-04 11:32:09,653:INFO:intervals before: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,654:INFO:the tp solutions: (87.8445559999999, 88.33864400000004)\n",
+ "2023-11-04 11:32:09,656:INFO:the tn solutions: (779.847324, 780.059076)\n",
+ "2023-11-04 11:32:09,656:INFO:intervals after: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,657:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-04 11:32:09,658:INFO:intervals before: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,659:INFO:the tp solutions: (87.90462837746212, 88.04741174998863)\n",
+ "2023-11-04 11:32:09,660:INFO:the tn solutions: (779.594751267001, 780.5429578357666)\n",
+ "2023-11-04 11:32:09,661:INFO:intervals after: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,664:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-04 11:32:09,665:INFO:intervals before: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,666:INFO:the tp solutions: (87.95250800000001, 88.023092)\n",
+ "2023-11-04 11:32:09,666:INFO:the tn solutions: (779.847324, 780.059076)\n",
+ "2023-11-04 11:32:09,667:INFO:intervals after: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,668:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-04 11:32:09,670:INFO:intervals before: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,672:INFO:the tp solutions: (87.95250800000001, 88.023092)\n",
+ "2023-11-04 11:32:09,673:INFO:the tn solutions: (778.7721780040409, 781.2044258844836)\n",
+ "2023-11-04 11:32:09,674:INFO:intervals after: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,675:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-04 11:32:09,677:INFO:intervals before: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,677:INFO:the tp solutions: (87.93699113585161, 88.04870828206266)\n",
+ "2023-11-04 11:32:09,680:INFO:the tn solutions: (779.847324, 780.059076)\n",
+ "2023-11-04 11:32:09,680:INFO:intervals after: [(88, 88)], [(780, 780)]\n",
+ "2023-11-04 11:32:09,682:INFO:testing acc, feasible tptn pairs: 1\n",
+ "2023-11-04 11:32:09,682:INFO:testing sens, feasible tptn pairs: 1\n",
+ "2023-11-04 11:32:09,684:INFO:testing spec, feasible tptn pairs: 1\n",
+ "2023-11-04 11:32:09,685:INFO:testing f1p, feasible tptn pairs: 1\n",
+ "2023-11-04 11:32:09,686:INFO:constructing final tp, tn pair set\n",
+ "2023-11-04 11:32:09,689:INFO:final number of intervals: 1\n",
+ "2023-11-04 11:32:09,690:INFO:final number of pairs: 1\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_som_micro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'acc': 0.6272, 'sens': 0.2543, 'spec': 0.7514, 'f1p': 0.2543}"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['sens'] = 0.2553"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-11-04 11:32:09,740:INFO:checking the scores {'acc': 0.6272, 'sens': 0.2553, 'spec': 0.7514, 'f1p': 0.2543}\n",
+ "2023-11-04 11:32:09,742:INFO:evaluating the tp and tn solution for acc and sens\n",
+ "2023-11-04 11:32:09,744:INFO:intervals before: (0, 346), (0, 1038)\n",
+ "2023-11-04 11:32:09,744:INFO:the tp solutions: (88.29850800000001, 88.36909200000001)\n",
+ "2023-11-04 11:32:09,745:INFO:the tn solutions: (779.5345399999999, 779.88746)\n",
+ "2023-11-04 11:32:09,746:INFO:intervals after: [(1, 0)], [(1, 0)]\n",
+ "2023-11-04 11:32:09,750:INFO:evaluating the tp and tn solution for acc and spec\n",
+ "2023-11-04 11:32:09,751:INFO:intervals before: [(1, 0)], [(1, 0)]\n",
+ "2023-11-04 11:32:09,752:INFO:the tp solutions: (87.8445559999999, 88.33864400000004)\n",
+ "2023-11-04 11:32:09,754:INFO:the tn solutions: (779.847324, 780.059076)\n",
+ "2023-11-04 11:32:09,755:INFO:intervals after: [], []\n",
+ "2023-11-04 11:32:09,756:INFO:evaluating the tp and tn solution for acc and f1p\n",
+ "2023-11-04 11:32:09,757:INFO:intervals before: [], []\n",
+ "2023-11-04 11:32:09,758:INFO:the tp solutions: (87.90462837746212, 88.04741174998863)\n",
+ "2023-11-04 11:32:09,759:INFO:the tn solutions: (779.594751267001, 780.5429578357666)\n",
+ "2023-11-04 11:32:09,760:INFO:intervals after: [], []\n",
+ "2023-11-04 11:32:09,761:INFO:evaluating the tp and tn solution for sens and spec\n",
+ "2023-11-04 11:32:09,762:INFO:intervals before: [], []\n",
+ "2023-11-04 11:32:09,763:INFO:the tp solutions: (88.29850800000001, 88.36909200000001)\n",
+ "2023-11-04 11:32:09,765:INFO:the tn solutions: (779.847324, 780.059076)\n",
+ "2023-11-04 11:32:09,765:INFO:intervals after: [], []\n",
+ "2023-11-04 11:32:09,767:INFO:evaluating the tp and tn solution for sens and f1p\n",
+ "2023-11-04 11:32:09,768:INFO:intervals before: [], []\n",
+ "2023-11-04 11:32:09,769:INFO:the tp solutions: (88.29850800000001, 88.36909200000001)\n",
+ "2023-11-04 11:32:09,769:INFO:the tn solutions: (776.397796151697, 778.8284162069881)\n",
+ "2023-11-04 11:32:09,771:INFO:intervals after: [], []\n",
+ "2023-11-04 11:32:09,772:INFO:evaluating the tp and tn solution for spec and f1p\n",
+ "2023-11-04 11:32:09,773:INFO:intervals before: [], []\n",
+ "2023-11-04 11:32:09,774:INFO:the tp solutions: (87.93699113585161, 88.04870828206266)\n",
+ "2023-11-04 11:32:09,775:INFO:the tn solutions: (779.847324, 780.059076)\n",
+ "2023-11-04 11:32:09,776:INFO:intervals after: [], []\n",
+ "2023-11-04 11:32:09,777:INFO:testing acc, feasible tptn pairs: 0\n",
+ "2023-11-04 11:32:09,777:INFO:no more feasible tp,tn pairs left\n",
+ "2023-11-04 11:32:09,778:INFO:constructing final tp, tn pair set\n",
+ "2023-11-04 11:32:09,779:INFO:final number of intervals: 0\n",
+ "2023-11-04 11:32:09,780:INFO:final number of pairs: 0\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_dataset_known_folds_som_micro(dataset=dataset,\n",
+ " folding=folding,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/notebooks/illustration/regression/00_1_testset_no_kfold.ipynb b/notebooks/illustration/regression/00_1_testset_no_kfold.ipynb
new file mode 100644
index 0000000..c3235bf
--- /dev/null
+++ b/notebooks/illustration/regression/00_1_testset_no_kfold.ipynb
@@ -0,0 +1,123 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from mlscorecheck.check.regression import (generate_regression_problem_and_scores,\n",
+ " check_1_testset_no_kfold,\n",
+ " score_formulas)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "var, n_samples, scores = generate_regression_problem_and_scores(random_state=None,\n",
+ " rounding_decimals=4,\n",
+ " subset=None)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "var = 0.08316192579267838\n",
+ "n_samples = 100\n",
+ "scores = {'mae': 0.0254, 'r2': 0.9897}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "False"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_testset_no_kfold(var=var,\n",
+ " n_samples=n_samples,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores['mae'] = 0.03"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "result = check_1_testset_no_kfold(var=var,\n",
+ " n_samples=n_samples,\n",
+ " scores=scores,\n",
+ " eps=1e-4)\n",
+ "result['inconsistency']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "mlscorecheck",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/tests/aggregated/_evaluate_lp.py b/tests/aggregated/_evaluate_lp.py
index 0f7f690..7ab183e 100644
--- a/tests/aggregated/_evaluate_lp.py
+++ b/tests/aggregated/_evaluate_lp.py
@@ -8,13 +8,12 @@
from mlscorecheck.aggregated import compare_scores, Experiment
-__all__ = ['evaluate_timeout']
+__all__ = ["evaluate_timeout"]
-def evaluate_timeout(result: pl.LpProblem,
- problem: Experiment,
- scores: dict,
- eps,
- score_subset: list):
+
+def evaluate_timeout(
+ result: pl.LpProblem, problem: Experiment, scores: dict, eps, score_subset: list
+):
"""
Evaluate the stopped or succeeded tests
@@ -29,6 +28,6 @@ def evaluate_timeout(result: pl.LpProblem,
populated = problem.populate(result)
assert compare_scores(scores, populated.calculate_scores(), eps, score_subset)
- assert populated.check_bounds()['bounds_flag'] is True
+ assert populated.check_bounds()["bounds_flag"] is True
elif result.status == 0:
- warnings.warn('test timed out')
+ warnings.warn("test timed out")
diff --git a/tests/aggregated/test_check_aggregated_scores.py b/tests/aggregated/test_check_aggregated_scores.py
index 9871764..7e50610 100644
--- a/tests/aggregated/test_check_aggregated_scores.py
+++ b/tests/aggregated/test_check_aggregated_scores.py
@@ -4,13 +4,16 @@
import pytest
-from mlscorecheck.aggregated import (check_aggregated_scores,
- generate_experiment,
- round_scores,
- Experiment)
+from mlscorecheck.aggregated import (
+ check_aggregated_scores,
+ generate_experiment,
+ round_scores,
+ Experiment,
+)
random_seeds = list(range(20))
+
def check_timeout(details):
"""
Checks if timeout happened
@@ -18,34 +21,48 @@ def check_timeout(details):
Args:
details (dict): the result of the test
"""
- if details['lp_status'] != 'timeout':
- return (details['lp_configuration_scores_match']
- and details['lp_configuration_bounds_match']
- and details['lp_configuration'] is not None)
+ if details["lp_status"] != "timeout":
+ return (
+ details["lp_configuration_scores_match"]
+ and details["lp_configuration_bounds_match"]
+ and details["lp_configuration"] is not None
+ )
return True
+
def test_check_timeout():
"""
Testing the check_timeout function
"""
- assert check_timeout({'lp_status': 'dummy',
- 'lp_configuration_scores_match': True,
- 'lp_configuration_bounds_match': True,
- 'lp_configuration': 'dummy'})
-
- assert not check_timeout({'lp_status': 'dummy',
- 'lp_configuration_scores_match': True,
- 'lp_configuration_bounds_match': True,
- 'lp_configuration': None})
-
- assert check_timeout({'lp_status': 'timeout'})
-
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('rounding_decimals', [2, 3, 4])
-@pytest.mark.parametrize('aggregations', [('som', 'som'), ('mos', 'mos'), ('mos', 'som')])
-def test_check_aggregated_scores_feasible(random_seed: int,
- rounding_decimals: int,
- aggregations: tuple):
+ assert check_timeout(
+ {
+ "lp_status": "dummy",
+ "lp_configuration_scores_match": True,
+ "lp_configuration_bounds_match": True,
+ "lp_configuration": "dummy",
+ }
+ )
+
+ assert not check_timeout(
+ {
+ "lp_status": "dummy",
+ "lp_configuration_scores_match": True,
+ "lp_configuration_bounds_match": True,
+ "lp_configuration": None,
+ }
+ )
+
+ assert check_timeout({"lp_status": "timeout"})
+
+
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("rounding_decimals", [2, 3, 4])
+@pytest.mark.parametrize(
+ "aggregations", [("som", "som"), ("mos", "mos"), ("mos", "som")]
+)
+def test_check_aggregated_scores_feasible(
+ random_seed: int, rounding_decimals: int, aggregations: tuple
+):
"""
Testing the top level aggregated check function with a feasible problem
@@ -54,25 +71,31 @@ def test_check_aggregated_scores_feasible(random_seed: int,
rounding_decimals (int): the number of decimals to round to
aggregations (tuple(str,str)): the aggregations to use
"""
- experiment, scores = generate_experiment(random_state=random_seed,
- return_scores=True,
- aggregation=aggregations[0],
- evaluation_params={'aggregation': aggregations[1]})
+ experiment, scores = generate_experiment(
+ random_state=random_seed,
+ return_scores=True,
+ aggregation=aggregations[0],
+ evaluation_params={"aggregation": aggregations[1]},
+ )
scores = round_scores(scores, rounding_decimals)
- details = check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=10**(-rounding_decimals),
- timeout=1)
+ details = check_aggregated_scores(
+ experiment=experiment, scores=scores, eps=10 ** (-rounding_decimals), timeout=1
+ )
- assert not details['inconsistency']
- assert details['lp_status'] in {'feasible', 'timeout'}
+ assert not details["inconsistency"]
+ assert details["lp_status"] in {"feasible", "timeout"}
assert check_timeout(details)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregations', [('som', 'som'), ('mos', 'mos'), ('mos', 'som')])
-def test_check_aggregated_scores_feasible_custom_solver(random_seed: int, aggregations: tuple):
+
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize(
+ "aggregations", [("som", "som"), ("mos", "mos"), ("mos", "som")]
+)
+def test_check_aggregated_scores_feasible_custom_solver(
+ random_seed: int, aggregations: tuple
+):
"""
Testing the top level aggregated check function with a feasible problem
with custom solver
@@ -81,24 +104,27 @@ def test_check_aggregated_scores_feasible_custom_solver(random_seed: int, aggreg
random_seed (int): the random seed to use
aggregations (tuple(str,str)): the aggregations to use
"""
- experiment, scores = generate_experiment(random_state=random_seed,
- return_scores=True,
- aggregation=aggregations[0],
- evaluation_params={'aggregation': aggregations[1]})
+ experiment, scores = generate_experiment(
+ random_state=random_seed,
+ return_scores=True,
+ aggregation=aggregations[0],
+ evaluation_params={"aggregation": aggregations[1]},
+ )
scores = round_scores(scores, 4)
- details = check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=1e-3,
- solver_name='dummy',
- timeout=1)
+ details = check_aggregated_scores(
+ experiment=experiment, scores=scores, eps=1e-3, solver_name="dummy", timeout=1
+ )
- assert not details['inconsistency']
- assert details['lp_status'] in {'feasible', 'timeout'}
+ assert not details["inconsistency"]
+ assert details["lp_status"] in {"feasible", "timeout"}
assert check_timeout(details)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregations', [('som', 'som'), ('mos', 'mos'), ('mos', 'som')])
+
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize(
+ "aggregations", [("som", "som"), ("mos", "mos"), ("mos", "som")]
+)
def test_check_aggregated_scores_infeasible(random_seed: int, aggregations: tuple):
"""
Testing the top level aggregated check function with an infeasible problem
@@ -108,20 +134,21 @@ def test_check_aggregated_scores_infeasible(random_seed: int, aggregations: tupl
random_seed (int): the random seed to use
aggregations (tuple(str,str)): the aggregations to use
"""
- experiment, scores = generate_experiment(random_state=random_seed,
- return_scores=True,
- aggregation=aggregations[0],
- evaluation_params={'aggregation': aggregations[1]})
+ experiment, scores = generate_experiment(
+ random_state=random_seed,
+ return_scores=True,
+ aggregation=aggregations[0],
+ evaluation_params={"aggregation": aggregations[1]},
+ )
- scores = {'acc': 0.1, 'sens': 0.1, 'spec': 0.1, 'bacc': 0.4}
+ scores = {"acc": 0.1, "sens": 0.1, "spec": 0.1, "bacc": 0.4}
- details = check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=1e-4)
+ details = check_aggregated_scores(experiment=experiment, scores=scores, eps=1e-4)
+
+ assert details["inconsistency"]
+ assert details["lp_status"] == "infeasible"
+ assert details["lp_configuration"] is not None
- assert details['inconsistency']
- assert details['lp_status'] == 'infeasible'
- assert details['lp_configuration'] is not None
def test_check_aggregated_scores_timeout():
"""
@@ -129,203 +156,200 @@ def test_check_aggregated_scores_timeout():
Eventually this test can fail, due to the unpredictability of solvers timing out
"""
- experiment, scores = generate_experiment(max_evaluations=20,
- evaluation_params={'max_folds': 20,
- 'max_repeats': 20,
- 'aggregation': 'mos',
- 'feasible_fold_score_bounds': True},
- random_state=5,
- return_scores=True,
- aggregation='mos',
- feasible_dataset_score_bounds=True)
+ experiment, scores = generate_experiment(
+ max_evaluations=20,
+ evaluation_params={
+ "max_folds": 20,
+ "max_repeats": 20,
+ "aggregation": "mos",
+ "feasible_fold_score_bounds": True,
+ },
+ random_state=5,
+ return_scores=True,
+ aggregation="mos",
+ feasible_dataset_score_bounds=True,
+ )
scores = round_scores(scores, 7)
- details = check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=1e-7,
- timeout=0.001,
- numerical_tolerance=1e-9)
+ details = check_aggregated_scores(
+ experiment=experiment,
+ scores=scores,
+ eps=1e-7,
+ timeout=0.001,
+ numerical_tolerance=1e-9,
+ )
+
+ assert not details["inconsistency"]
+ assert details["lp_status"] == "timeout"
+ assert details["lp_configuration"] is not None
- assert not details['inconsistency']
- assert details['lp_status'] == 'timeout'
- assert details['lp_configuration'] is not None
def test_no_suitable_score():
"""
Testing the case when no score is suitable
"""
- details = check_aggregated_scores(experiment=None,
- scores={'f1': 0.5},
- eps=1e-7,
- timeout=0.001,
- numerical_tolerance=1e-9)
+ details = check_aggregated_scores(
+ experiment=None,
+ scores={"f1": 0.5},
+ eps=1e-7,
+ timeout=0.001,
+ numerical_tolerance=1e-9,
+ )
+
+ assert not details["inconsistency"]
- assert not details['inconsistency']
def test_others():
"""
Testing other functionalities
"""
- experiment, scores = generate_experiment(aggregation='som',
- evaluation_params={'aggregation': 'mos'},
- return_scores=True)
+ experiment, scores = generate_experiment(
+ aggregation="som", evaluation_params={"aggregation": "mos"}, return_scores=True
+ )
with pytest.raises(ValueError):
- check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=1e-4)
+ check_aggregated_scores(experiment=experiment, scores=scores, eps=1e-4)
+
def test_aggregated_success_with_p_zero():
"""
Testing a valid configuration with one p in a fold zero
"""
- dataset = {'p': 10, 'n': 20}
- folds = [{'p': 0, 'n': 5}, {'p': 5, 'n': 10}, {'p': 5, 'n': 5}]
- evaluation = {'dataset': dataset,
- 'folding': {'folds': folds},
- 'aggregation': 'mos'}
- experiment = {'evaluations': [evaluation],
- 'aggregation': 'mos'}
+ dataset = {"p": 10, "n": 20}
+ folds = [{"p": 0, "n": 5}, {"p": 5, "n": 10}, {"p": 5, "n": 5}]
+ evaluation = {"dataset": dataset, "folding": {"folds": folds}, "aggregation": "mos"}
+ experiment = {"evaluations": [evaluation], "aggregation": "mos"}
experiment = Experiment(**experiment)
- experiment.sample_figures(score_subset=['acc', 'spec'])
+ experiment.sample_figures(score_subset=["acc", "spec"])
- scores = experiment.calculate_scores(score_subset=['acc', 'spec'],
- rounding_decimals=4)
+ scores = experiment.calculate_scores(
+ score_subset=["acc", "spec"], rounding_decimals=4
+ )
- details = check_aggregated_scores(experiment=experiment.to_dict(),
- scores=scores,
- eps=1e-4,
- solver_name='dummy',
- timeout=1)
+ details = check_aggregated_scores(
+ experiment=experiment.to_dict(),
+ scores=scores,
+ eps=1e-4,
+ solver_name="dummy",
+ timeout=1,
+ )
- assert not details['inconsistency']
- assert details['lp_status'] in {'feasible', 'timeout'}
+ assert not details["inconsistency"]
+ assert details["lp_status"] in {"feasible", "timeout"}
assert check_timeout(details)
+
def test_aggregated_failure_with_p_zero():
"""
Testing an invalid configuration with one p in a fold zero
"""
- dataset = {'p': 10, 'n': 20}
- folds = [{'p': 0, 'n': 5}, {'p': 5, 'n': 10}, {'p': 5, 'n': 5}]
- evaluation = {'dataset': dataset,
- 'folding': {'folds': folds},
- 'aggregation': 'mos'}
- experiment = {'evaluations': [evaluation],
- 'aggregation': 'mos'}
+ dataset = {"p": 10, "n": 20}
+ folds = [{"p": 0, "n": 5}, {"p": 5, "n": 10}, {"p": 5, "n": 5}]
+ evaluation = {"dataset": dataset, "folding": {"folds": folds}, "aggregation": "mos"}
+ experiment = {"evaluations": [evaluation], "aggregation": "mos"}
+
+ scores = {"acc": 0.9584, "spec": 0.9576}
- scores = {'acc': 0.9584, 'spec': 0.9576}
+ details = check_aggregated_scores(
+ experiment=experiment, scores=scores, eps=1e-4, solver_name="dummy", timeout=1
+ )
- details = check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=1e-4,
- solver_name='dummy',
- timeout=1)
+ assert details["inconsistency"]
- assert details['inconsistency']
def test_aggregated_success_with_n_zero():
"""
Testing a valid configuration with one n in a fold zero
"""
- dataset = {'p': 10, 'n': 20}
- folds = [{'p': 2, 'n': 0}, {'p': 3, 'n': 15}, {'p': 5, 'n': 5}]
- evaluation = {'dataset': dataset,
- 'folding': {'folds': folds},
- 'aggregation': 'mos'}
- experiment = {'evaluations': [evaluation],
- 'aggregation': 'mos'}
+ dataset = {"p": 10, "n": 20}
+ folds = [{"p": 2, "n": 0}, {"p": 3, "n": 15}, {"p": 5, "n": 5}]
+ evaluation = {"dataset": dataset, "folding": {"folds": folds}, "aggregation": "mos"}
+ experiment = {"evaluations": [evaluation], "aggregation": "mos"}
experiment = Experiment(**experiment)
- experiment.sample_figures(score_subset=['acc', 'sens'])
+ experiment.sample_figures(score_subset=["acc", "sens"])
- scores = experiment.calculate_scores(score_subset=['acc', 'sens'],
- rounding_decimals=4)
+ scores = experiment.calculate_scores(
+ score_subset=["acc", "sens"], rounding_decimals=4
+ )
- details = check_aggregated_scores(experiment=experiment.to_dict(),
- scores=scores,
- eps=1e-4,
- solver_name='dummy',
- timeout=1)
+ details = check_aggregated_scores(
+ experiment=experiment.to_dict(),
+ scores=scores,
+ eps=1e-4,
+ solver_name="dummy",
+ timeout=1,
+ )
- assert not details['inconsistency']
- assert details['lp_status'] in {'feasible', 'timeout'}
+ assert not details["inconsistency"]
+ assert details["lp_status"] in {"feasible", "timeout"}
assert check_timeout(details)
+
def test_aggregated_failure_with_n_zero():
"""
Testing an invalid configuration with one n in a fold zero
"""
- dataset = {'p': 10, 'n': 20}
- folds = [{'p': 2, 'n': 0}, {'p': 3, 'n': 15}, {'p': 5, 'n': 5}]
- evaluation = {'dataset': dataset,
- 'folding': {'folds': folds},
- 'aggregation': 'mos'}
- experiment = {'evaluations': [evaluation],
- 'aggregation': 'mos'}
+ dataset = {"p": 10, "n": 20}
+ folds = [{"p": 2, "n": 0}, {"p": 3, "n": 15}, {"p": 5, "n": 5}]
+ evaluation = {"dataset": dataset, "folding": {"folds": folds}, "aggregation": "mos"}
+ experiment = {"evaluations": [evaluation], "aggregation": "mos"}
+
+ scores = {"acc": 0.9584, "sens": 0.9576}
- scores = {'acc': 0.9584, 'sens': 0.9576}
+ details = check_aggregated_scores(
+ experiment=experiment, scores=scores, eps=1e-4, solver_name="dummy", timeout=1
+ )
- details = check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=1e-4,
- solver_name='dummy',
- timeout=1)
+ assert details["inconsistency"]
- assert details['inconsistency']
def test_aggregated_success_with_n_or_p_zero():
"""
Testing a valid configuration with one p and one n in some fold zero
"""
- dataset = {'p': 10, 'n': 20}
- folds = [{'p': 5, 'n': 0}, {'p': 0, 'n': 15}, {'p': 5, 'n': 5}]
- evaluation = {'dataset': dataset,
- 'folding': {'folds': folds},
- 'aggregation': 'mos'}
- experiment = {'evaluations': [evaluation],
- 'aggregation': 'mos'}
+ dataset = {"p": 10, "n": 20}
+ folds = [{"p": 5, "n": 0}, {"p": 0, "n": 15}, {"p": 5, "n": 5}]
+ evaluation = {"dataset": dataset, "folding": {"folds": folds}, "aggregation": "mos"}
+ experiment = {"evaluations": [evaluation], "aggregation": "mos"}
experiment = Experiment(**experiment)
- experiment.sample_figures(score_subset=['acc'])
+ experiment.sample_figures(score_subset=["acc"])
- scores = experiment.calculate_scores(score_subset=['acc'],
- rounding_decimals=4)
+ scores = experiment.calculate_scores(score_subset=["acc"], rounding_decimals=4)
- details = check_aggregated_scores(experiment=experiment.to_dict(),
- scores=scores,
- eps=1e-4,
- solver_name='dummy',
- timeout=1)
+ details = check_aggregated_scores(
+ experiment=experiment.to_dict(),
+ scores=scores,
+ eps=1e-4,
+ solver_name="dummy",
+ timeout=1,
+ )
- assert not details['inconsistency']
- assert details['lp_status'] in {'feasible', 'timeout'}
+ assert not details["inconsistency"]
+ assert details["lp_status"] in {"feasible", "timeout"}
assert check_timeout(details)
+
def test_aggregated_failure_with_n_or_p_zero():
"""
Testing an invalid configuration with one p and one n in some fold zero
"""
- dataset = {'p': 10, 'n': 20}
- folds = [{'p': 5, 'n': 0}, {'p': 0, 'n': 15}, {'p': 5, 'n': 5}]
- evaluation = {'dataset': dataset,
- 'folding': {'folds': folds},
- 'aggregation': 'mos'}
- experiment = {'evaluations': [evaluation],
- 'aggregation': 'mos'}
-
- scores = {'acc': 0.9584}
-
- details = check_aggregated_scores(experiment=experiment,
- scores=scores,
- eps=1e-4,
- solver_name='dummy',
- timeout=1)
-
- assert details['inconsistency']
+ dataset = {"p": 10, "n": 20}
+ folds = [{"p": 5, "n": 0}, {"p": 0, "n": 15}, {"p": 5, "n": 5}]
+ evaluation = {"dataset": dataset, "folding": {"folds": folds}, "aggregation": "mos"}
+ experiment = {"evaluations": [evaluation], "aggregation": "mos"}
+
+ scores = {"acc": 0.9584}
+
+ details = check_aggregated_scores(
+ experiment=experiment, scores=scores, eps=1e-4, solver_name="dummy", timeout=1
+ )
+
+ assert details["inconsistency"]
diff --git a/tests/aggregated/test_dataset.py b/tests/aggregated/test_dataset.py
index 954af89..11cbc82 100644
--- a/tests/aggregated/test_dataset.py
+++ b/tests/aggregated/test_dataset.py
@@ -13,11 +13,11 @@
import pytest
-from mlscorecheck.aggregated import (Dataset,
- generate_dataset)
+from mlscorecheck.aggregated import Dataset, generate_dataset
random_seeds = list(range(20))
+
def test_dataset_instantiation():
"""
Testing the instantiation of a dataset
@@ -30,9 +30,9 @@ def test_dataset_instantiation():
Dataset()
with pytest.raises(ValueError):
- Dataset(p=5, n=10, dataset_name='common_datasets.ADA')
+ Dataset(p=5, n=10, dataset_name="common_datasets.ADA")
- dataset = Dataset(dataset_name='common_datasets.ADA')
+ dataset = Dataset(dataset_name="common_datasets.ADA")
assert dataset.p > 0 and dataset.n > 0
@@ -40,7 +40,8 @@ def test_dataset_instantiation():
assert dataset2.p == dataset.p and dataset2.n == dataset.n
-@pytest.mark.parametrize('random_seed', random_seeds)
+
+@pytest.mark.parametrize("random_seed", random_seeds)
def test_dataset_generation(random_seed: int):
"""
Testing the dataset generation
diff --git a/tests/aggregated/test_evaluation.py b/tests/aggregated/test_evaluation.py
index b8e282a..64be9cc 100644
--- a/tests/aggregated/test_evaluation.py
+++ b/tests/aggregated/test_evaluation.py
@@ -8,39 +8,53 @@
import pulp as pl
-from mlscorecheck.aggregated import (Evaluation,
- generate_dataset,
- generate_folding,
- generate_evaluation,
- solve,
- compare_scores,
- get_fold_score_bounds)
+from mlscorecheck.aggregated import (
+ Evaluation,
+ generate_dataset,
+ generate_folding,
+ generate_evaluation,
+ solve,
+ compare_scores,
+ get_fold_score_bounds,
+)
from ._evaluate_lp import evaluate_timeout
-PREFERRED_SOLVER = 'PULP_CBC_CMD'
+PREFERRED_SOLVER = "PULP_CBC_CMD"
solvers = pl.listSolvers(onlyAvailable=True)
PREFERRED_SOLVER = PREFERRED_SOLVER if PREFERRED_SOLVER in solvers else solvers[0]
solver = pl.getSolver(PREFERRED_SOLVER)
solver_timeout = pl.getSolver(PREFERRED_SOLVER, timeLimit=5)
-two_combs = [['acc', 'sens'], ['acc', 'spec'], ['acc', 'bacc'],
- ['sens', 'spec'], ['sens', 'bacc'], ['spec', 'bacc']]
-three_combs = [['acc', 'sens', 'spec'], ['acc', 'sens', 'bacc'],
- ['acc', 'spec', 'bacc'], ['sens', 'spec', 'bacc']]
-four_combs = [['acc', 'sens', 'spec', 'bacc']]
+two_combs = [
+ ["acc", "sens"],
+ ["acc", "spec"],
+ ["acc", "bacc"],
+ ["sens", "spec"],
+ ["sens", "bacc"],
+ ["spec", "bacc"],
+]
+three_combs = [
+ ["acc", "sens", "spec"],
+ ["acc", "sens", "bacc"],
+ ["acc", "spec", "bacc"],
+ ["sens", "spec", "bacc"],
+]
+four_combs = [["acc", "sens", "spec", "bacc"]]
random_seeds = list(range(5))
+
def test_evaluate_timeout():
"""
Testing the evaluate_timeout function
"""
- class Mock: # pylint: disable=too-few-public-methods
+ class Mock: # pylint: disable=too-few-public-methods
"""
Mock lp_problem class
"""
+
def __init__(self):
"""
Constructor of the mock class
@@ -53,8 +67,9 @@ def __init__(self):
evaluate_timeout(mock, None, None, None, None)
assert len(warn) == 1
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
+
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
def test_instantiation(random_seed: int, aggregation: str):
"""
Testing the instantiation of evaluations
@@ -73,11 +88,14 @@ def test_instantiation(random_seed: int, aggregation: str):
evaluation2 = Evaluation(**evaluation.to_dict())
- assert evaluation.figures['p'] == evaluation2.figures['p']\
- and evaluation.figures['n'] == evaluation2.figures['n']
+ assert (
+ evaluation.figures["p"] == evaluation2.figures["p"]
+ and evaluation.figures["n"] == evaluation2.figures["n"]
+ )
+
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
def test_sample_figures(random_seed: int, aggregation: str):
"""
Testing the sampling of figures
@@ -94,16 +112,16 @@ def test_sample_figures(random_seed: int, aggregation: str):
evaluation.sample_figures(random_state=random_seed).calculate_scores()
- assert evaluation.figures['tp'] >= 0 and evaluation.figures['tn'] >= 0
+ assert evaluation.figures["tp"] >= 0 and evaluation.figures["tn"] >= 0
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
-@pytest.mark.parametrize('rounding_decimals', [2, 3, 4])
-def test_linear_programming_success(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
+@pytest.mark.parametrize("rounding_decimals", [2, 3, 4])
+def test_linear_programming_success(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities
@@ -125,27 +143,29 @@ def test_linear_programming_success(subset: list,
skeleton = Evaluation(dataset, folding, aggregation=aggregation)
- lp_program = solve(skeleton, scores, eps=10**(-rounding_decimals))
+ lp_program = solve(skeleton, scores, eps=10 ** (-rounding_decimals))
assert lp_program.status == 1
skeleton.populate(lp_program)
- assert compare_scores(scores,
- skeleton.calculate_scores(),
- eps=10**(-rounding_decimals),
- tolerance=1e-6)
-
- assert skeleton.check_bounds()['bounds_flag']
-
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
-@pytest.mark.parametrize('rounding_decimals', [2, 3, 4])
-def test_linear_programming_evaluation_generation_success(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+ assert compare_scores(
+ scores,
+ skeleton.calculate_scores(),
+ eps=10 ** (-rounding_decimals),
+ tolerance=1e-6,
+ )
+
+ assert skeleton.check_bounds()["bounds_flag"]
+
+
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
+@pytest.mark.parametrize("rounding_decimals", [2, 3, 4])
+def test_linear_programming_evaluation_generation_success(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities by generating the evaluation
@@ -156,8 +176,7 @@ def test_linear_programming_evaluation_generation_success(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- evaluation = generate_evaluation(random_state=random_seed,
- aggregation=aggregation)
+ evaluation = generate_evaluation(random_state=random_seed, aggregation=aggregation)
evaluation = Evaluation(**evaluation)
@@ -167,21 +186,25 @@ def test_linear_programming_evaluation_generation_success(subset: list,
skeleton = Evaluation(**evaluation.to_dict())
- lp_program = solve(skeleton, scores, eps=10**(-rounding_decimals))
+ lp_program = solve(skeleton, scores, eps=10 ** (-rounding_decimals))
assert lp_program.status == 1
skeleton.populate(lp_program)
- assert compare_scores(scores,
- skeleton.calculate_scores(),
- eps=10**(-rounding_decimals),
- tolerance=1e-6)
+ assert compare_scores(
+ scores,
+ skeleton.calculate_scores(),
+ eps=10 ** (-rounding_decimals),
+ tolerance=1e-6,
+ )
+
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
-def test_linear_programming_evaluation_generation_failure(random_seed: int,
- aggregation: str):
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
+def test_linear_programming_evaluation_generation_failure(
+ random_seed: int, aggregation: str
+):
"""
Testing the linear programming functionalities by generating the evaluation
@@ -190,14 +213,13 @@ def test_linear_programming_evaluation_generation_failure(random_seed: int,
aggregation (str): the aggregation to use ('mos'/'som')
"""
- evaluation = generate_evaluation(random_state=random_seed,
- aggregation=aggregation)
+ evaluation = generate_evaluation(random_state=random_seed, aggregation=aggregation)
evaluation = Evaluation(**evaluation)
evaluation.sample_figures(random_state=random_seed)
- scores = {'acc': 0.5, 'sens': 0.6, 'spec': 0.6}
+ scores = {"acc": 0.5, "sens": 0.6, "spec": 0.6}
skeleton = Evaluation(**evaluation.to_dict())
@@ -205,8 +227,9 @@ def test_linear_programming_evaluation_generation_failure(random_seed: int,
assert lp_program.status == -1
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
+
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
def test_get_fold_score_bounds(random_seed: int, aggregation: str):
"""
Testing the extraction of fold score bounds
@@ -216,8 +239,7 @@ def test_get_fold_score_bounds(random_seed: int, aggregation: str):
aggregation (str): the aggregation to use ('mos'/'som')
"""
- evaluation = generate_evaluation(random_state=random_seed,
- aggregation=aggregation)
+ evaluation = generate_evaluation(random_state=random_seed, aggregation=aggregation)
evaluation = Evaluation(**evaluation)
evaluation.sample_figures().calculate_scores()
@@ -228,14 +250,14 @@ def test_get_fold_score_bounds(random_seed: int, aggregation: str):
for key in score_bounds:
assert score_bounds[key][0] <= fold.scores[key] <= score_bounds[key][1]
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos'])
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_linear_programming_success_bounds(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos"])
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_linear_programming_success_bounds(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities by generating the evaluation
with bounds
@@ -247,31 +269,35 @@ def test_linear_programming_success_bounds(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- evaluation, scores = generate_evaluation(random_state=random_seed,
- aggregation=aggregation,
- feasible_fold_score_bounds=True,
- rounding_decimals=rounding_decimals,
- return_scores=True,
- score_subset=subset)
+ evaluation, scores = generate_evaluation(
+ random_state=random_seed,
+ aggregation=aggregation,
+ feasible_fold_score_bounds=True,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ score_subset=subset,
+ )
evaluation = Evaluation(**evaluation)
skeleton = Evaluation(**evaluation.to_dict())
- lp_program = solve(skeleton, scores, eps=10**(-rounding_decimals), solver=solver_timeout)
+ lp_program = solve(
+ skeleton, scores, eps=10 ** (-rounding_decimals), solver=solver_timeout
+ )
assert lp_program.status in (0, 1)
- evaluate_timeout(lp_program, skeleton, scores, 10**(-rounding_decimals), subset)
+ evaluate_timeout(lp_program, skeleton, scores, 10 ** (-rounding_decimals), subset)
+
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos'])
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_linear_programming_failure_bounds(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos"])
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_linear_programming_failure_bounds(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities by generating the evaluation
with bounds
@@ -283,29 +309,33 @@ def test_linear_programming_failure_bounds(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- evaluation, scores = generate_evaluation(random_state=random_seed,
- aggregation=aggregation,
- feasible_fold_score_bounds=False,
- rounding_decimals=rounding_decimals,
- return_scores=True,
- score_subset=subset)
+ evaluation, scores = generate_evaluation(
+ random_state=random_seed,
+ aggregation=aggregation,
+ feasible_fold_score_bounds=False,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ score_subset=subset,
+ )
evaluation = Evaluation(**evaluation)
skeleton = Evaluation(**evaluation.to_dict())
- lp_program = solve(skeleton, scores, eps=10**(-rounding_decimals), solver=solver_timeout)
+ lp_program = solve(
+ skeleton, scores, eps=10 ** (-rounding_decimals), solver=solver_timeout
+ )
assert lp_program.status in (-1, 0)
- evaluate_timeout(lp_program, skeleton, scores, 10**(-rounding_decimals), subset)
+ evaluate_timeout(lp_program, skeleton, scores, 10 ** (-rounding_decimals), subset)
+
def test_others():
"""
Testing other functionalities
"""
- evaluation = generate_evaluation(aggregation='som',
- feasible_fold_score_bounds=True)
+ evaluation = generate_evaluation(aggregation="som", feasible_fold_score_bounds=True)
with pytest.raises(ValueError):
Evaluation(**evaluation)
diff --git a/tests/aggregated/test_experiment.py b/tests/aggregated/test_experiment.py
index f551cb9..f61ead9 100644
--- a/tests/aggregated/test_experiment.py
+++ b/tests/aggregated/test_experiment.py
@@ -17,29 +17,42 @@
import numpy as np
-from mlscorecheck.aggregated import (Experiment,
- solve,
- generate_experiment,
- compare_scores,
- get_dataset_score_bounds)
+from mlscorecheck.aggregated import (
+ Experiment,
+ solve,
+ generate_experiment,
+ compare_scores,
+ get_dataset_score_bounds,
+)
from ._evaluate_lp import evaluate_timeout
-PREFERRED_SOLVER = 'PULP_CBC_CMD'
+PREFERRED_SOLVER = "PULP_CBC_CMD"
solvers = pl.listSolvers(onlyAvailable=True)
PREFERRED_SOLVER = PREFERRED_SOLVER if PREFERRED_SOLVER in solvers else solvers[0]
solver = pl.getSolver(PREFERRED_SOLVER)
solver_timeout = pl.getSolver(PREFERRED_SOLVER, timeLimit=5)
-two_combs = [['acc', 'sens'], ['acc', 'spec'], ['acc', 'bacc'],
- ['sens', 'spec'], ['sens', 'bacc'], ['spec', 'bacc']]
-three_combs = [['acc', 'sens', 'spec'], ['acc', 'sens', 'bacc'],
- ['acc', 'spec', 'bacc'], ['sens', 'spec', 'bacc']]
-four_combs = [['acc', 'sens', 'spec', 'bacc']]
+two_combs = [
+ ["acc", "sens"],
+ ["acc", "spec"],
+ ["acc", "bacc"],
+ ["sens", "spec"],
+ ["sens", "bacc"],
+ ["spec", "bacc"],
+]
+three_combs = [
+ ["acc", "sens", "spec"],
+ ["acc", "sens", "bacc"],
+ ["acc", "spec", "bacc"],
+ ["sens", "spec", "bacc"],
+]
+four_combs = [["acc", "sens", "spec", "bacc"]]
random_seeds = list(range(5))
-@pytest.mark.parametrize('random_seed', random_seeds)
+
+@pytest.mark.parametrize("random_seed", random_seeds)
def test_experiment_instantiation(random_seed: int):
"""
Testing the creation of Experiment objects
@@ -55,10 +68,13 @@ def test_experiment_instantiation(random_seed: int):
experiment2 = Experiment(**experiment.to_dict())
- assert experiment.figures['p'] == experiment2.figures['p']\
- and experiment.figures['n'] == experiment2.figures['n']
+ assert (
+ experiment.figures["p"] == experiment2.figures["p"]
+ and experiment.figures["n"] == experiment2.figures["n"]
+ )
+
-@pytest.mark.parametrize('random_seed', random_seeds)
+@pytest.mark.parametrize("random_seed", random_seeds)
def test_sampling_and_scores(random_seed: int):
"""
Testing the score calculation in experiments
@@ -74,19 +90,27 @@ def test_sampling_and_scores(random_seed: int):
scores = experiment.calculate_scores()
- if experiment.aggregation == 'som':
- value = float(experiment.figures['tp'] / experiment.figures['p'])
- assert abs(scores['sens'] - value) < 1e-10
- elif experiment.aggregation == 'mos':
- assert abs(np.mean([evaluation.scores['acc'] for evaluation in experiment.evaluations])\
- - scores['acc']) < 1e-10
-
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos'])
-@pytest.mark.parametrize('aggregation_folds', ['mos', 'som'])
-def test_get_dataset_score_bounds(random_seed: int,
- aggregation: str,
- aggregation_folds: str):
+ if experiment.aggregation == "som":
+ value = float(experiment.figures["tp"] / experiment.figures["p"])
+ assert abs(scores["sens"] - value) < 1e-10
+ elif experiment.aggregation == "mos":
+ assert (
+ abs(
+ np.mean(
+ [evaluation.scores["acc"] for evaluation in experiment.evaluations]
+ )
+ - scores["acc"]
+ )
+ < 1e-10
+ )
+
+
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos"])
+@pytest.mark.parametrize("aggregation_folds", ["mos", "som"])
+def test_get_dataset_score_bounds(
+ random_seed: int, aggregation: str, aggregation_folds: str
+):
"""
Testing the score bounds determination
@@ -96,9 +120,11 @@ def test_get_dataset_score_bounds(random_seed: int,
aggregation_folds (str): the aggregation on folds ('mos'/'som')
"""
- experiment = generate_experiment(random_state=random_seed,
- aggregation=aggregation,
- evaluation_params={'aggregation': aggregation_folds})
+ experiment = generate_experiment(
+ random_state=random_seed,
+ aggregation=aggregation,
+ evaluation_params={"aggregation": aggregation_folds},
+ )
experiment = Experiment(**experiment)
experiment.sample_figures().calculate_scores()
@@ -106,23 +132,29 @@ def test_get_dataset_score_bounds(random_seed: int,
for evaluation in experiment.evaluations:
for key in score_bounds:
- assert score_bounds[key][0] <= evaluation.scores[key] <= score_bounds[key][1]
+ assert (
+ score_bounds[key][0] <= evaluation.scores[key] <= score_bounds[key][1]
+ )
score_bounds = get_dataset_score_bounds(experiment, feasible=False)
for evaluation in experiment.evaluations:
for key in score_bounds:
if score_bounds[key][0] < 1.0:
- assert not score_bounds[key][0] <= evaluation.scores[key] <= score_bounds[key][1]
-
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
-@pytest.mark.parametrize('rounding_decimals', [2, 3, 4])
-def test_linear_programming_success(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+ assert (
+ not score_bounds[key][0]
+ <= evaluation.scores[key]
+ <= score_bounds[key][1]
+ )
+
+
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
+@pytest.mark.parametrize("rounding_decimals", [2, 3, 4])
+def test_linear_programming_success(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities in a successful scenario
@@ -133,8 +165,7 @@ def test_linear_programming_success(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- experiment = generate_experiment(random_state=random_seed,
- aggregation=aggregation)
+ experiment = generate_experiment(random_state=random_seed, aggregation=aggregation)
experiment = Experiment(**experiment)
experiment.sample_figures(random_state=random_seed)
@@ -143,27 +174,29 @@ def test_linear_programming_success(subset: list,
skeleton = Experiment(**experiment.to_dict())
- lp_program = solve(skeleton, scores, eps=10**(-rounding_decimals))
+ lp_program = solve(skeleton, scores, eps=10 ** (-rounding_decimals))
assert lp_program.status == 1
skeleton.populate(lp_program)
- assert compare_scores(scores,
- skeleton.calculate_scores(),
- eps=10**(-rounding_decimals),
- tolerance=1e-6)
-
- assert skeleton.check_bounds()['bounds_flag']
-
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos'])
-@pytest.mark.parametrize('rounding_decimals', [2, 3, 4])
-def test_linear_programming_success_with_bounds(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+ assert compare_scores(
+ scores,
+ skeleton.calculate_scores(),
+ eps=10 ** (-rounding_decimals),
+ tolerance=1e-6,
+ )
+
+ assert skeleton.check_bounds()["bounds_flag"]
+
+
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos"])
+@pytest.mark.parametrize("rounding_decimals", [2, 3, 4])
+def test_linear_programming_success_with_bounds(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities with bounds
@@ -174,28 +207,32 @@ def test_linear_programming_success_with_bounds(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- experiment, scores = generate_experiment(random_state=random_seed,
- aggregation=aggregation,
- return_scores=True,
- feasible_dataset_score_bounds=True,
- score_subset=subset)
+ experiment, scores = generate_experiment(
+ random_state=random_seed,
+ aggregation=aggregation,
+ return_scores=True,
+ feasible_dataset_score_bounds=True,
+ score_subset=subset,
+ )
experiment = Experiment(**experiment)
- lp_program = solve(experiment, scores, eps=10**(-rounding_decimals), solver=solver_timeout)
+ lp_program = solve(
+ experiment, scores, eps=10 ** (-rounding_decimals), solver=solver_timeout
+ )
assert lp_program.status in (0, 1)
- evaluate_timeout(lp_program, experiment, scores, 10**(-rounding_decimals), subset)
+ evaluate_timeout(lp_program, experiment, scores, 10 ** (-rounding_decimals), subset)
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos'])
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_linear_programming_failure_with_bounds(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos"])
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_linear_programming_failure_with_bounds(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities with bounds
@@ -206,28 +243,32 @@ def test_linear_programming_failure_with_bounds(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- experiment, scores = generate_experiment(random_state=random_seed,
- aggregation=aggregation,
- return_scores=True,
- feasible_dataset_score_bounds=False,
- score_subset=subset)
+ experiment, scores = generate_experiment(
+ random_state=random_seed,
+ aggregation=aggregation,
+ return_scores=True,
+ feasible_dataset_score_bounds=False,
+ score_subset=subset,
+ )
experiment = Experiment(**experiment)
- lp_program = solve(experiment, scores, eps=10**(-rounding_decimals), solver=solver_timeout)
+ lp_program = solve(
+ experiment, scores, eps=10 ** (-rounding_decimals), solver=solver_timeout
+ )
assert lp_program.status in (0, -1)
- evaluate_timeout(lp_program, experiment, scores, 10**(-rounding_decimals), subset)
+ evaluate_timeout(lp_program, experiment, scores, 10 ** (-rounding_decimals), subset)
+
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos'])
-@pytest.mark.parametrize('rounding_decimals', [2, 3])
-def test_linear_programming_success_both_bounds(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos"])
+@pytest.mark.parametrize("rounding_decimals", [2, 3])
+def test_linear_programming_success_both_bounds(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities with both bounds
@@ -238,34 +279,34 @@ def test_linear_programming_success_both_bounds(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- experiment, scores = generate_experiment(random_state=random_seed,
- aggregation=aggregation,
- evaluation_params={'aggregation': 'mos',
- 'feasible_fold_score_bounds': True},
- return_scores=True,
- rounding_decimals=rounding_decimals,
- feasible_dataset_score_bounds=True,
- score_subset=subset)
+ experiment, scores = generate_experiment(
+ random_state=random_seed,
+ aggregation=aggregation,
+ evaluation_params={"aggregation": "mos", "feasible_fold_score_bounds": True},
+ return_scores=True,
+ rounding_decimals=rounding_decimals,
+ feasible_dataset_score_bounds=True,
+ score_subset=subset,
+ )
experiment = Experiment(**experiment)
- lp_program = solve(experiment,
- scores,
- eps=10**(-rounding_decimals),
- solver=solver_timeout)
+ lp_program = solve(
+ experiment, scores, eps=10 ** (-rounding_decimals), solver=solver_timeout
+ )
assert lp_program.status in (0, 1)
- evaluate_timeout(lp_program, experiment, scores, 10**(-rounding_decimals), subset)
+ evaluate_timeout(lp_program, experiment, scores, 10 ** (-rounding_decimals), subset)
-@pytest.mark.parametrize('subset', two_combs + three_combs + four_combs)
-@pytest.mark.parametrize('random_seed', random_seeds)
-@pytest.mark.parametrize('aggregation', ['mos'])
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_linear_programming_failure_both_bounds(subset: list,
- random_seed: int,
- aggregation: str,
- rounding_decimals: int):
+
+@pytest.mark.parametrize("subset", two_combs + three_combs + four_combs)
+@pytest.mark.parametrize("random_seed", random_seeds)
+@pytest.mark.parametrize("aggregation", ["mos"])
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_linear_programming_failure_both_bounds(
+ subset: list, random_seed: int, aggregation: str, rounding_decimals: int
+):
"""
Testing the linear programming functionalities with both bounds
@@ -276,29 +317,32 @@ def test_linear_programming_failure_both_bounds(subset: list,
rounding_decimals (int): the number of decimals to round to
"""
- experiment, scores = generate_experiment(random_state=random_seed,
- aggregation=aggregation,
- evaluation_params={'aggregation': 'mos',
- 'feasible_fold_score_bounds': True},
- return_scores=True,
- rounding_decimals=rounding_decimals,
- feasible_dataset_score_bounds=False,
- score_subset=subset)
+ experiment, scores = generate_experiment(
+ random_state=random_seed,
+ aggregation=aggregation,
+ evaluation_params={"aggregation": "mos", "feasible_fold_score_bounds": True},
+ return_scores=True,
+ rounding_decimals=rounding_decimals,
+ feasible_dataset_score_bounds=False,
+ score_subset=subset,
+ )
experiment = Experiment(**experiment)
- lp_program = solve(experiment, scores, eps=10**(-rounding_decimals))
+ lp_program = solve(experiment, scores, eps=10 ** (-rounding_decimals))
assert lp_program.status in (-1, 0)
- evaluate_timeout(lp_program, experiment, scores, 10**(-rounding_decimals), subset)
+ evaluate_timeout(lp_program, experiment, scores, 10 ** (-rounding_decimals), subset)
+
def test_others():
"""
Testing other functionalities
"""
- experiment = generate_experiment(aggregation='som',
- feasible_dataset_score_bounds=True)
+ experiment = generate_experiment(
+ aggregation="som", feasible_dataset_score_bounds=True
+ )
with pytest.raises(ValueError):
Experiment(**experiment)
diff --git a/tests/aggregated/test_fold.py b/tests/aggregated/test_fold.py
index f5609b9..a7a4741 100644
--- a/tests/aggregated/test_fold.py
+++ b/tests/aggregated/test_fold.py
@@ -9,19 +9,31 @@
TOL = 1e-5
-two_combs = [['acc', 'sens'], ['acc', 'spec'], ['acc', 'bacc'],
- ['sens', 'spec'], ['sens', 'bacc'], ['spec', 'bacc']]
-three_combs = [['acc', 'sens', 'spec'], ['acc', 'sens', 'bacc'],
- ['acc', 'spec', 'bacc'], ['sens', 'spec', 'bacc']]
-four_combs = [['acc', 'sens', 'spec', 'bacc']]
+two_combs = [
+ ["acc", "sens"],
+ ["acc", "spec"],
+ ["acc", "bacc"],
+ ["sens", "spec"],
+ ["sens", "bacc"],
+ ["spec", "bacc"],
+]
+three_combs = [
+ ["acc", "sens", "spec"],
+ ["acc", "sens", "bacc"],
+ ["acc", "spec", "bacc"],
+ ["sens", "spec", "bacc"],
+]
+four_combs = [["acc", "sens", "spec", "bacc"]]
+
def test_fold_creation():
"""
Testing the creation of Fold objects
"""
- fold = Fold(p=5, n=10, identifier='dummy')
- assert fold.to_dict() == {'p': 5, 'n': 10, 'identifier': 'dummy'}
+ fold = Fold(p=5, n=10, identifier="dummy")
+ assert fold.to_dict() == {"p": 5, "n": 10, "identifier": "dummy"}
+
def test_fold_repr():
"""
@@ -30,6 +42,7 @@ def test_fold_repr():
assert isinstance(str(Fold(p=5, n=10)), str)
+
def test_fold_sampling():
"""
Testing the fold sampling
@@ -40,24 +53,31 @@ def test_fold_sampling():
scores = fold.calculate_scores()
- np.testing.assert_almost_equal(scores['acc'],
- (fold.tp + fold.tn)/(fold.p + fold.n))
- np.testing.assert_almost_equal(scores['sens'], fold.tp/fold.p)
- np.testing.assert_almost_equal(scores['spec'], fold.tn/fold.n)
- np.testing.assert_almost_equal(scores['bacc'], (scores['sens'] + scores['spec'])/2.0)
+ np.testing.assert_almost_equal(
+ scores["acc"], (fold.tp + fold.tn) / (fold.p + fold.n)
+ )
+ np.testing.assert_almost_equal(scores["sens"], fold.tp / fold.p)
+ np.testing.assert_almost_equal(scores["spec"], fold.tn / fold.n)
+ np.testing.assert_almost_equal(
+ scores["bacc"], (scores["sens"] + scores["spec"]) / 2.0
+ )
+
+ assert (
+ fold.sample_figures(5).calculate_scores()
+ != fold.sample_figures(6).calculate_scores()
+ )
- assert fold.sample_figures(5).calculate_scores() != fold.sample_figures(6).calculate_scores()
def test_linear_programming():
"""
Testing if the linear programming interfaces work
"""
- lp_problem = pl.LpProblem('dummy')
+ lp_problem = pl.LpProblem("dummy")
fold = Fold(5, 10)
- fold.init_lp(scores={'acc': 0.5, 'bacc': 0.5, 'sens': 0.5, 'spec': 0.5})
+ fold.init_lp(scores={"acc": 0.5, "bacc": 0.5, "sens": 0.5, "spec": 0.5})
assert isinstance(fold.tp, pl.LpVariable)
assert isinstance(fold.tn, pl.LpVariable)
diff --git a/tests/aggregated/test_folding.py b/tests/aggregated/test_folding.py
index 99306da..b4c5418 100644
--- a/tests/aggregated/test_folding.py
+++ b/tests/aggregated/test_folding.py
@@ -6,6 +6,7 @@
from mlscorecheck.aggregated import Folding, Dataset, generate_folding, generate_dataset
+
def test_instantiation():
"""
Testing the instantiation of a Folding
@@ -20,43 +21,46 @@ def test_instantiation():
with pytest.raises(ValueError):
Folding(n_folds=5, n_repeats=3)
- folding = Folding(n_folds=5, n_repeats=3, strategy='stratified_sklearn')
+ folding = Folding(n_folds=5, n_repeats=3, strategy="stratified_sklearn")
assert folding is not None
+
def test_to_dict():
"""
Testing the dictionary representation
"""
- folding = Folding(n_folds=5, n_repeats=3, strategy='stratified_sklearn')
+ folding = Folding(n_folds=5, n_repeats=3, strategy="stratified_sklearn")
folding2 = Folding(**folding.to_dict())
assert folding2.n_folds == 5
assert folding2.folds is None
+
def test_generate_folds():
"""
Testing the generation of folds
"""
- folding = Folding(n_folds=5, n_repeats=3, strategy='stratified_sklearn')
+ folding = Folding(n_folds=5, n_repeats=3, strategy="stratified_sklearn")
dataset = Dataset(p=5, n=10)
- assert len(folding.generate_folds(dataset, 'som')) == 1
- assert len(folding.generate_folds(dataset, 'mos')) == 15
+ assert len(folding.generate_folds(dataset, "som")) == 1
+ assert len(folding.generate_folds(dataset, "mos")) == 15
with pytest.raises(ValueError):
- folding.generate_folds(dataset=dataset, aggregation='dummy')
+ folding.generate_folds(dataset=dataset, aggregation="dummy")
- folding = Folding(folds=[{'p': 5, 'n': 10}, {'p': 10, 'n': 20}])
- assert len(folding.generate_folds(dataset=dataset, aggregation='som')) == 2
+ folding = Folding(folds=[{"p": 5, "n": 10}, {"p": 10, "n": 20}])
+ assert len(folding.generate_folds(dataset=dataset, aggregation="som")) == 2
with pytest.raises(ValueError):
- folding.generate_folds(dataset=Dataset(p=4, n=21), aggregation='som')
+ folding.generate_folds(dataset=Dataset(p=4, n=21), aggregation="som")
+
-@pytest.mark.parametrize('random_seed', range(10))
+@pytest.mark.parametrize("random_seed", range(10))
def test_generate_folding(random_seed: int):
"""
Testing the folding generation
diff --git a/tests/aggregated/test_folding_utils.py b/tests/aggregated/test_folding_utils.py
index 8fe42c0..a6d3733 100644
--- a/tests/aggregated/test_folding_utils.py
+++ b/tests/aggregated/test_folding_utils.py
@@ -9,48 +9,70 @@
import numpy as np
from sklearn.model_selection import StratifiedKFold
-from mlscorecheck.aggregated import (stratified_configurations_sklearn,
- determine_fold_configurations,
- _create_folds,
- repeated_kfolds_generator,
- fold_partitioning_generator,
- _check_specification_and_determine_p_n,
- determine_min_max_p)
+from mlscorecheck.individual import generate_multiclass_dataset
+from mlscorecheck.aggregated import (
+ stratified_configurations_sklearn,
+ determine_fold_configurations,
+ _create_folds,
+ repeated_kfolds_generator,
+ fold_partitioning_generator,
+ _check_specification_and_determine_p_n,
+ determine_min_max_p,
+ multiclass_stratified_folds,
+ transform_multiclass_fold_to_binary,
+ create_folds_multiclass,
+)
+
def test_generate_datasets_with_all_kfolds():
"""
Testing the generation of datasets with all kfold configurations
"""
- evaluation = {'dataset': {'p': 5, 'n': 7}, 'folding': {'n_folds': 3}}
+ evaluation = {"dataset": {"p": 5, "n": 7}, "folding": {"n_folds": 3}}
- datasets = list(repeated_kfolds_generator(evaluation,
- available_scores=['acc', 'bacc',
- 'sens', 'spec']))
+ datasets = list(
+ repeated_kfolds_generator(
+ evaluation, available_scores=["acc", "bacc", "sens", "spec"]
+ )
+ )
assert len(datasets) == 2
- evaluation = {'dataset': {'p': 5, 'n': 7},
- 'folding': {'n_folds': 3, 'n_repeats': 2}}
+ evaluation = {
+ "dataset": {"p": 5, "n": 7},
+ "folding": {"n_folds": 3, "n_repeats": 2},
+ }
- datasets = list(repeated_kfolds_generator(evaluation,
- available_scores=['acc', 'bacc',
- 'sens', 'spec']))
+ datasets = list(
+ repeated_kfolds_generator(
+ evaluation, available_scores=["acc", "bacc", "sens", "spec"]
+ )
+ )
assert len(datasets) == 4
- evaluation = {'dataset': {'p': 5, 'n': 7},
- 'folding': {'n_folds': 3, 'n_repeats': 2},
- 'fold_score_bounds': {'acc': (0.0, 1.0)}}
+ evaluation = {
+ "dataset": {"p": 5, "n": 7},
+ "folding": {"n_folds": 3, "n_repeats": 2},
+ "fold_score_bounds": {"acc": (0.0, 1.0)},
+ }
+
+ datasets = list(
+ repeated_kfolds_generator(
+ evaluation, available_scores=["acc", "bacc", "sens", "spec"]
+ )
+ )
+ assert "fold_score_bounds" in datasets[0]
- datasets = list(repeated_kfolds_generator(evaluation,
- available_scores=['acc', 'bacc',
- 'sens', 'spec']))
- assert 'fold_score_bounds' in datasets[0]
+ evaluation = {
+ "dataset": {"dataset_name": "common_datasets.appendicitis"},
+ "folding": {"n_folds": 3},
+ }
- evaluation = {'dataset': {'dataset_name': 'common_datasets.appendicitis'},
- 'folding': {'n_folds': 3}}
+ datasets = list(
+ repeated_kfolds_generator(
+ evaluation, available_scores=["acc", "bacc", "sens", "spec"]
+ )
+ )
- datasets = list(repeated_kfolds_generator(evaluation,
- available_scores=['acc', 'bacc',
- 'sens', 'spec']))
def test_exceptions():
"""
@@ -58,35 +80,42 @@ def test_exceptions():
"""
with pytest.raises(ValueError):
- _check_specification_and_determine_p_n(None, {'folds': []})
+ _check_specification_and_determine_p_n(None, {"folds": []})
with pytest.raises(ValueError):
- _check_specification_and_determine_p_n({'p': 2}, {})
+ _check_specification_and_determine_p_n({"p": 2}, {})
with pytest.raises(ValueError):
- _check_specification_and_determine_p_n({'p': 2, 'n': 5, 'dataset_name': 'dummy'}, {})
+ _check_specification_and_determine_p_n(
+ {"p": 2, "n": 5, "dataset_name": "dummy"}, {}
+ )
+
def test_create_folds():
"""
Testing the creation of folds
"""
- folds = _create_folds(5, 10, score_bounds={'acc': (0.0, 1.0)},
- n_repeats=1, n_folds=1)
+ folds = _create_folds(
+ 5, 10, score_bounds={"acc": (0.0, 1.0)}, n_repeats=1, n_folds=1
+ )
assert len(folds) == 1
- folds = _create_folds(p=5, n=10,
- n_folds=2,
- n_repeats=2,
- score_bounds={'acc': (0.0, 1.0)})
+ folds = _create_folds(
+ p=5, n=10, n_folds=2, n_repeats=2, score_bounds={"acc": (0.0, 1.0)}
+ )
assert len(folds) == 1
- folds = _create_folds(p=5, n=10,
- n_folds=2,
- n_repeats=2,
- folding='stratified_sklearn',
- score_bounds={'acc': (0.0, 1.0)})
+ folds = _create_folds(
+ p=5,
+ n=10,
+ n_folds=2,
+ n_repeats=2,
+ folding="stratified_sklearn",
+ score_bounds={"acc": (0.0, 1.0)},
+ )
assert len(folds) == 4
+
def sklearn_configurations(y_labels, n_splits):
"""
Generating the sklearn fold configurations
@@ -100,10 +129,13 @@ def sklearn_configurations(y_labels, n_splits):
"""
validator = StratifiedKFold(n_splits=n_splits)
- return [tuple(np.bincount(y_labels[test]).tolist())
- for _, test in validator.split(y_labels, y_labels, y_labels)]
+ return [
+ tuple(np.bincount(y_labels[test]).tolist())
+ for _, test in validator.split(y_labels, y_labels, y_labels)
+ ]
+
-@pytest.mark.parametrize('random_state', list(range(500)))
+@pytest.mark.parametrize("random_state", list(range(500)))
def test_stratified_configurations_sklearn(random_state):
"""
Testing the determination of the stratified sklearn fold configurations
@@ -115,14 +147,16 @@ def test_stratified_configurations_sklearn(random_state):
random_state = np.random.RandomState(random_state)
n_splits = random_state.randint(2, 40)
- n_items = random_state.randint(n_splits * 2, n_splits*100)
+ n_items = random_state.randint(n_splits * 2, n_splits * 100)
n_1 = random_state.randint(n_splits, n_items - n_splits + 1)
n_0 = n_items - n_1
y_labels = np.hstack([np.repeat(0, n_0), np.repeat(1, n_1)])
- assert stratified_configurations_sklearn(n_1, n_0, n_splits) \
- == sklearn_configurations(y_labels, n_splits)
+ assert stratified_configurations_sklearn(
+ n_1, n_0, n_splits
+ ) == sklearn_configurations(y_labels, n_splits)
+
def test_determine_fold_configurations():
"""
@@ -131,12 +165,13 @@ def test_determine_fold_configurations():
conf = determine_fold_configurations(10, 20, 4, 1)
- conf = [(tmp['n'], tmp['p']) for tmp in conf]
+ conf = [(tmp["n"], tmp["p"]) for tmp in conf]
assert conf == stratified_configurations_sklearn(10, 20, 4)
with pytest.raises(ValueError):
- determine_fold_configurations(10, 20, 4, 1, 'dummy')
+ determine_fold_configurations(10, 20, 4, 1, "dummy")
+
def any_zero(values):
"""
@@ -150,6 +185,7 @@ def any_zero(values):
"""
return any(val == 0 for val in values)
+
def test_any_zero():
"""
Testing the any_zero function
@@ -157,6 +193,7 @@ def test_any_zero():
assert any_zero([0, 1])
assert not any_zero([1, 1])
+
def test_fold_partitioning_generator():
"""
Testing the fold partitioning generator
@@ -166,46 +203,67 @@ def test_fold_partitioning_generator():
assert all((not any_zero(fold[0])) and (not any_zero(fold[1])) for fold in folds)
- folds = list(fold_partitioning_generator(p=6, n=6, k=3, p_non_zero=False, n_non_zero=True))
+ folds = list(
+ fold_partitioning_generator(p=6, n=6, k=3, p_non_zero=False, n_non_zero=True)
+ )
assert all(not any_zero(fold[1]) for fold in folds)
assert any(any_zero(fold[0]) for fold in folds)
- folds = list(fold_partitioning_generator(p=6, n=6, k=3, p_non_zero=True, n_non_zero=False))
+ folds = list(
+ fold_partitioning_generator(p=6, n=6, k=3, p_non_zero=True, n_non_zero=False)
+ )
assert all(not any_zero(fold[0]) for fold in folds)
assert any(any_zero(fold[1]) for fold in folds)
- folds = list(fold_partitioning_generator(p=6, n=6, k=3, p_non_zero=False, n_non_zero=False))
+ folds = list(
+ fold_partitioning_generator(p=6, n=6, k=3, p_non_zero=False, n_non_zero=False)
+ )
assert any(any_zero(fold[0]) for fold in folds)
assert any(any_zero(fold[1]) for fold in folds)
+
def test_fold_partitioning_generator_p_min():
"""
Testing the fold partitioning generator with p_min
"""
- folds = fold_partitioning_generator(p=6, n=7, k=3, p_non_zero=True, n_non_zero=True, p_min=2)
+ folds = fold_partitioning_generator(
+ p=6, n=7, k=3, p_non_zero=True, n_non_zero=True, p_min=2
+ )
assert len(list(folds)) == 1
- folds = list(fold_partitioning_generator(p=6, n=7, k=3, p_non_zero=False, n_non_zero=True,
- p_min=2))
+ folds = list(
+ fold_partitioning_generator(
+ p=6, n=7, k=3, p_non_zero=False, n_non_zero=True, p_min=2
+ )
+ )
assert len(list(folds)) == 1
- folds = list(fold_partitioning_generator(p=6, n=7, k=3, p_non_zero=True, n_non_zero=False,
- p_min=2))
+ folds = list(
+ fold_partitioning_generator(
+ p=6, n=7, k=3, p_non_zero=True, n_non_zero=False, p_min=2
+ )
+ )
assert len(list(folds)) == 1
- folds = list(fold_partitioning_generator(p=6, n=7, k=3, p_non_zero=False, n_non_zero=False,
- p_min=2))
+ folds = list(
+ fold_partitioning_generator(
+ p=6, n=7, k=3, p_non_zero=False, n_non_zero=False, p_min=2
+ )
+ )
assert len(list(folds)) == 1
-def exhaustive_min_max_p(*, p, k_a, k_b, c_a, c_b, p_non_zero, n_non_zero): # pylint: disable=too-many-locals
+
+def exhaustive_min_max_p(
+ *, p, k_a, k_b, c_a, c_b, p_non_zero, n_non_zero
+): # pylint: disable=too-many-locals
"""
Exhaustive search for the minimum and maximum p in folds of type A
@@ -222,8 +280,8 @@ def exhaustive_min_max_p(*, p, k_a, k_b, c_a, c_b, p_non_zero, n_non_zero): # py
int, int: the minimum and maximum number of positives total in folds of
type A
"""
- a_folds_p = [list(range(p+1))] * k_a
- b_folds_p = [list(range(p+1))] * k_b
+ a_folds_p = [list(range(p + 1))] * k_a
+ b_folds_p = [list(range(p + 1))] * k_b
min_p_a = p
max_p_a = 0
@@ -259,12 +317,15 @@ def exhaustive_min_max_p(*, p, k_a, k_b, c_a, c_b, p_non_zero, n_non_zero): # py
return min_p_a, max_p_a
-@pytest.mark.parametrize('p', list(range(2, 20, 3)))
-@pytest.mark.parametrize('n', list(range(5, 30, 3)))
-@pytest.mark.parametrize('k', list(range(2, 6)))
-@pytest.mark.parametrize('p_non_zero', [True, False])
-@pytest.mark.parametrize('n_non_zero', [True, False])
-def test_determine_min_max_p(p, n, k, p_non_zero, n_non_zero): # pylint: disable=invalid-name
+
+@pytest.mark.parametrize("p", list(range(2, 20, 3)))
+@pytest.mark.parametrize("n", list(range(5, 30, 3)))
+@pytest.mark.parametrize("k", list(range(2, 6)))
+@pytest.mark.parametrize("p_non_zero", [True, False])
+@pytest.mark.parametrize("n_non_zero", [True, False])
+def test_determine_min_max_p(
+ p, n, k, p_non_zero, n_non_zero
+): # pylint: disable=invalid-name
"""
Testing the determination of minimum and maximum p with exhaustive search
@@ -285,17 +346,68 @@ def test_determine_min_max_p(p, n, k, p_non_zero, n_non_zero): # pylint: disable
k_a = k_mod
c_a = k_div + 1
c_b = k_div
- min_p, max_p = determine_min_max_p(p=p, n=n,
- k_a=k_a, k_b=k_b,
- c_a=c_a,
- p_non_zero=p_non_zero,
- n_non_zero=n_non_zero)
-
- min_p_full, max_p_full = exhaustive_min_max_p(p=p,
- k_a=k_a, k_b=k_b,
- c_a=c_a, c_b=c_b,
- p_non_zero=p_non_zero,
- n_non_zero=n_non_zero)
+ min_p, max_p = determine_min_max_p(
+ p=p,
+ n=n,
+ k_a=k_a,
+ k_b=k_b,
+ c_a=c_a,
+ p_non_zero=p_non_zero,
+ n_non_zero=n_non_zero,
+ )
+
+ min_p_full, max_p_full = exhaustive_min_max_p(
+ p=p,
+ k_a=k_a,
+ k_b=k_b,
+ c_a=c_a,
+ c_b=c_b,
+ p_non_zero=p_non_zero,
+ n_non_zero=n_non_zero,
+ )
assert min_p == min_p_full
assert max_p == max_p_full
+
+
+def test_multiclass_stratified_folds():
+ """
+ Testing the generation of multiclass stratified folds
+ """
+
+ dataset = generate_multiclass_dataset(random_state=5)
+
+ folds = multiclass_stratified_folds(dataset, n_folds=3)
+
+ assert len(folds) == 3
+
+ counts = [0] * len(dataset)
+
+ for fold in folds:
+ for class_, count in fold.items():
+ counts[class_] += count
+
+ assert counts == list(dataset.values())
+
+
+def test_transform_multiclass_fold_to_binary():
+ """
+ Testing the transformation of a multiclass fold to binary folds
+ """
+
+ dataset = generate_multiclass_dataset(random_state=5)
+
+ bfolds = transform_multiclass_fold_to_binary(dataset)
+
+ assert len(bfolds) == len(dataset)
+
+
+def test_multiclass_create_folds_exception():
+ """
+ Testing the exception throwing of the multiclass fold creation
+ """
+
+ with pytest.raises(ValueError):
+ create_folds_multiclass(
+ dataset={"p": 5, "n": 7}, folding={"folds": "dummy", "n_repeats": 5}
+ )
diff --git a/tests/aggregated/test_generate_problems.py b/tests/aggregated/test_generate_problems.py
index 891aca6..d979655 100644
--- a/tests/aggregated/test_generate_problems.py
+++ b/tests/aggregated/test_generate_problems.py
@@ -2,24 +2,41 @@
This module tests the functionalities related to problem generation
"""
-from mlscorecheck.aggregated import generate_scores_for_testsets
+import pytest
+
+from mlscorecheck.aggregated import (
+ generate_scores_for_testsets,
+ generate_dataset_folding_multiclass,
+)
+
def test_generate_scores_for_testsets():
"""
Testing the generation of scores for multiple testsets
"""
- scores = generate_scores_for_testsets([{'p': 5, 'n': 10, 'identifier': 'a'},
- {'p': 6, 'n': 20, 'identifier': 'b'}],
- rounding_decimals=4,
- subset=['acc', 'sens'],
- aggregation='mos')
+ scores = generate_scores_for_testsets(
+ [{"p": 5, "n": 10, "identifier": "a"}, {"p": 6, "n": 20, "identifier": "b"}],
+ rounding_decimals=4,
+ subset=["acc", "sens"],
+ aggregation="mos",
+ )
assert len(scores) == 2
- scores = generate_scores_for_testsets([{'p': 5, 'n': 10, 'identifier': 'a'},
- {'p': 6, 'n': 20, 'identifier': 'b'}],
- rounding_decimals=4,
- subset=['acc', 'sens'],
- aggregation='som')
+ scores = generate_scores_for_testsets(
+ [{"p": 5, "n": 10, "identifier": "a"}, {"p": 6, "n": 20, "identifier": "b"}],
+ rounding_decimals=4,
+ subset=["acc", "sens"],
+ aggregation="som",
+ )
assert len(scores) == 4
+
+
+def test_exception():
+ """
+ Testing the exception throwing
+ """
+
+ with pytest.raises(ValueError):
+ generate_dataset_folding_multiclass(aggregation="dummy")
diff --git a/tests/aggregated/test_linear_programming.py b/tests/aggregated/test_linear_programming.py
index 5d2da25..58249ea 100644
--- a/tests/aggregated/test_linear_programming.py
+++ b/tests/aggregated/test_linear_programming.py
@@ -4,10 +4,8 @@
import pulp as pl
-from mlscorecheck.aggregated import (Evaluation,
- generate_evaluation,
- solve,
- add_bounds)
+from mlscorecheck.aggregated import Evaluation, generate_evaluation, solve, add_bounds
+
def test_solve():
"""
@@ -19,23 +17,30 @@ def test_solve():
scores = sample.calculate_scores()
- eps = {'acc': 0.001, 'sens': 0.001, 'spec': 0.001, 'bacc': 0.001}
+ eps = {"acc": 0.001, "sens": 0.001, "spec": 0.001, "bacc": 0.001}
assert solve(evaluation, scores, eps).status == 1
+
def test_add_bounds():
"""
Testing the add_bounds function
"""
- problem = pl.LpProblem('dummy')
- problem = add_bounds(problem, {'tn': pl.LpVariable('tn')}, {'tn': (0, 10)}, label='tn')
+ problem = pl.LpProblem("dummy")
+ problem = add_bounds(
+ problem, {"tn": pl.LpVariable("tn")}, {"tn": (0, 10)}, label="tn"
+ )
assert len(problem.constraints) == 2
- problem = pl.LpProblem('dummy')
- problem = add_bounds(problem, {'tn': pl.LpVariable('tn')}, {'tn': (0, None)}, label='tn')
+ problem = pl.LpProblem("dummy")
+ problem = add_bounds(
+ problem, {"tn": pl.LpVariable("tn")}, {"tn": (0, None)}, label="tn"
+ )
assert len(problem.constraints) == 1
- problem = pl.LpProblem('dummy')
- problem = add_bounds(problem, {'tn': pl.LpVariable('tn')}, {'tn': (None, 10)}, label='tn')
+ problem = pl.LpProblem("dummy")
+ problem = add_bounds(
+ problem, {"tn": pl.LpVariable("tn")}, {"tn": (None, 10)}, label="tn"
+ )
assert len(problem.constraints) == 1
diff --git a/tests/aggregated/test_utils.py b/tests/aggregated/test_utils.py
index 122d37d..0d4cf0f 100644
--- a/tests/aggregated/test_utils.py
+++ b/tests/aggregated/test_utils.py
@@ -2,9 +2,8 @@
This module tests the linear programming functionalities
"""
-from mlscorecheck.aggregated import (compare_scores,
- random_identifier,
- check_bounds)
+from mlscorecheck.aggregated import compare_scores, random_identifier, check_bounds
+
def test_random_identifier():
"""
@@ -14,24 +13,25 @@ def test_random_identifier():
assert len(random_identifier(16)) == 16
assert random_identifier(10) != random_identifier(10)
+
def test_check_bounds():
"""
Testing the check bounds function
"""
- bounds = {'acc': [0.5, 1.0],
- 'sens': [0.0, 0.5]}
+ bounds = {"acc": [0.5, 1.0], "sens": [0.0, 0.5]}
- assert check_bounds({'acc': 0.7, 'sens': 0.2}, bounds)
- assert not check_bounds({'acc': 0.2, 'sens': 0.8}, bounds)
+ assert check_bounds({"acc": 0.7, "sens": 0.2}, bounds)
+ assert not check_bounds({"acc": 0.2, "sens": 0.8}, bounds)
assert check_bounds({}, None) is None
+
def test_compare_scores():
"""
Testing the score comparison
"""
- scores0 = {'acc': 0.1}
- scores1 = {'acc': 0.1001}
+ scores0 = {"acc": 0.1}
+ scores1 = {"acc": 0.1001}
- assert compare_scores(scores0, scores1, eps=0.0, subset=['acc'], tolerance=1e-4)
- assert not compare_scores(scores0, scores1, eps=0.0, subset=['acc'], tolerance=1e-5)
+ assert compare_scores(scores0, scores1, eps=0.0, subset=["acc"], tolerance=1e-4)
+ assert not compare_scores(scores0, scores1, eps=0.0, subset=["acc"], tolerance=1e-5)
diff --git a/tests/bundles/test_isic2016.py b/tests/bundles/test_isic2016.py
deleted file mode 100644
index 65d56e6..0000000
--- a/tests/bundles/test_isic2016.py
+++ /dev/null
@@ -1,45 +0,0 @@
-"""
-This module tests the tests developed for the ISIC2016 dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.skinlesion import check_isic2016
-from mlscorecheck.aggregated import generate_scores_for_testsets
-from mlscorecheck.experiments import get_experiment
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-def test_consistency(random_seed):
- """
- Testing a consistent configuration
-
- Args:
- random_seed (int): the random seed to be used
- """
-
- testset = get_experiment('skinlesion.isic2016')
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- random_state=random_seed)
-
- results = check_isic2016(scores=scores, eps=1e-4)
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-def test_inconsistency(random_seed):
- """
- Testing an inconsistent configuration
-
- Args:
- random_seed (int): the random seed to be used
- """
-
- testset = get_experiment('skinlesion.isic2016')
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- random_state=random_seed)
-
- scores['acc'] = (1.0 + scores['spec']) / 2.0
-
- results = check_isic2016(scores=scores, eps=1e-4)
- assert results['inconsistency']
diff --git a/tests/bundles/test_isic2017.py b/tests/bundles/test_isic2017.py
deleted file mode 100644
index 16a7723..0000000
--- a/tests/bundles/test_isic2017.py
+++ /dev/null
@@ -1,67 +0,0 @@
-"""
-This module tests the tests developed for the ISIC2017 dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.skinlesion import check_isic2017, _prepare_testset_isic2017
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-subsets = [(['M'], ['SK', 'N']),
- (['SK'], ['M', 'N']),
- (['N'], ['SK', 'M']),
- (['M', 'SK'], ['N']),
- (['M', 'N'], ['SK']),
- (['SK', 'N'], ['M'])]
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('target_against', subsets)
-def test_consistency(random_seed, target_against):
- """
- Testing a consistent configuration
-
- Args:
- random_seed (int): the random seed to be used
- target_against (tuple(list,list)): the target and against classes
- """
-
- testset = _prepare_testset_isic2017(target=target_against[0],
- against=target_against[1])
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- random_state=random_seed,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation='som')
-
- results = check_isic2017(target=target_against[0],
- against=target_against[1],
- scores=scores,
- eps=1e-4)
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('target_against', subsets)
-def test_inconsistency(random_seed, target_against):
- """
- Testing an inconsistent configuration
-
- Args:
- random_seed (int): the random seed to be used
- target_against (tuple(list,list)): the target and against classes
- """
-
- testset = _prepare_testset_isic2017(target=target_against[0],
- against=target_against[1])
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- random_state=random_seed,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation='som')
-
- scores['acc'] = (1.0 + scores['spec']) / 2.0
-
- results = check_isic2017(target=target_against[0],
- against=target_against[1],
- scores=scores,
- eps=1e-4)
- assert results['inconsistency']
diff --git a/tests/bundles/test_retina_chasedb1.py b/tests/bundles/test_retina_chasedb1.py
deleted file mode 100644
index 7b1042b..0000000
--- a/tests/bundles/test_retina_chasedb1.py
+++ /dev/null
@@ -1,185 +0,0 @@
-"""
-This file tests the test bundle for the CHASEDB1 dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.retina import (check_chasedb1_vessel_image,
- check_chasedb1_vessel_aggregated)
-
-from mlscorecheck.experiments import get_experiment
-
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_mos(random_state):
- """
- Testing a consistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.chase_db1')['manual1']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
- results = check_chasedb1_vessel_aggregated(imageset='all',
- annotator='manual1',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_mos']
-
- results = check_chasedb1_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator='manual1',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_mos(random_state):
- """
- Testing an inconsistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.chase_db1')['manual1']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_chasedb1_vessel_aggregated(imageset='all',
- annotator='manual1',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_mos']
-
- results = check_chasedb1_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator='manual1',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_som(random_state):
- """
- Testing a consistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.chase_db1')['manual1']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_chasedb1_vessel_aggregated(imageset='all',
- annotator='manual1',
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_som']
-
- results = check_chasedb1_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator='manual1',
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_som(random_state):
- """
- Testing an inconsistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.chase_db1')['manual1']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_chasedb1_vessel_aggregated(imageset='all',
- annotator='manual1',
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_som']
-
- results = check_chasedb1_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator='manual1',
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_image(random_state):
- """
- Testing a consistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.chase_db1')['manual1']['images']
-
- scores = generate_scores_for_testsets([data[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_chasedb1_vessel_image(image_identifier=data[0]['identifier'],
- annotator='manual1',
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_image(random_state):
- """
- Testing an inconsistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
- data = get_experiment('retina.chase_db1')['manual1']['images']
-
- scores = generate_scores_for_testsets([data[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_chasedb1_vessel_image(image_identifier=data[0]['identifier'],
- annotator='manual1',
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']
diff --git a/tests/bundles/test_retina_diaretdb0.py b/tests/bundles/test_retina_diaretdb0.py
deleted file mode 100644
index d8e66b8..0000000
--- a/tests/bundles/test_retina_diaretdb0.py
+++ /dev/null
@@ -1,98 +0,0 @@
-"""
-This module tests the test bundle for the DIARETDB0 dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.retina import check_diaretdb0_class
-from mlscorecheck.bundles.retina import _prepare_configuration_diaretdb0
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-@pytest.mark.parametrize('class_name', ['neovascularisation',
- 'hardexudates',
- 'hemorrhages',
- 'softexudates',
- 'redsmalldots'])
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
-def test_success(class_name, random_seed, aggregation):
- """
- Testing consistent configurations
-
- Args:
- class_name (str): the name of the class to test
- random_seed (int): the random seed to be used
- aggregation (str): the aggregation to be used
- """
-
- testsets = _prepare_configuration_diaretdb0("test", "all", class_name)
- scores = generate_scores_for_testsets(testsets=testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation=aggregation,
- random_state=random_seed)
-
- results = check_diaretdb0_class(subset='test',
- batch='all',
- class_name=class_name,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency'][f'inconsistency_{aggregation}']
-
- testsets = _prepare_configuration_diaretdb0("test", ["1", "2"], class_name)
- scores = generate_scores_for_testsets(testsets=testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation=aggregation,
- random_state=random_seed)
-
- results = check_diaretdb0_class(subset='test',
- batch=["1", "2"],
- class_name=class_name,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency'][f'inconsistency_{aggregation}']
-
-@pytest.mark.parametrize('class_name', ['neovascularisation',
- 'hardexudates',
- 'hemorrhages',
- 'softexudates',
- 'redsmalldots'])
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
-def test_failure(class_name, random_seed, aggregation):
- """
- Testing inconsistent configurations
-
- Args:
- class_name (str): the name of the class to test
- random_seed (int): the random seed to be used
- aggregation (str): the aggregation to be used
- """
-
- testsets = _prepare_configuration_diaretdb0("test", "all", class_name)
- scores = generate_scores_for_testsets(testsets=testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation=aggregation,
- random_state=random_seed)
-
- scores['acc'] = (1.0 + scores['spec']) / 2.0
-
- results = check_diaretdb0_class(subset='test',
- batch='all',
- class_name=class_name,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency'][f'inconsistency_{aggregation}']
-
- results = check_diaretdb0_class(subset='test',
- batch=["1", "2"],
- class_name=class_name,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency'][f'inconsistency_{aggregation}']
diff --git a/tests/bundles/test_retina_diaretdb1.py b/tests/bundles/test_retina_diaretdb1.py
deleted file mode 100644
index cb1954d..0000000
--- a/tests/bundles/test_retina_diaretdb1.py
+++ /dev/null
@@ -1,319 +0,0 @@
-"""
-This module tests the test functionalities implemented for the DIARETDB1 dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.retina import (_prepare_configuration_diaretdb1,
- check_diaretdb1_class,
- check_diaretdb1_segmentation_image,
- check_diaretdb1_segmentation_aggregated,
- check_diaretdb1_segmentation_aggregated_assumption)
-
-from mlscorecheck.experiments import get_experiment
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-class_names = [['hardexudates'], ['softexudates'], ['hemorrhages'], ['redsmalldots'],
- ['hardexudates', 'softexudates'], ['hemorrhages', 'redsmalldots']]
-
-data = get_experiment('retina.diaretdb1')
-test_identifiers = data['test']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('subset', ['train', 'test'])
-@pytest.mark.parametrize('class_name', class_names)
-def test_check_success_class(random_seed: str,
- confidence: float,
- subset: str,
- class_name):
- """
- Testing the image labeling in a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- confidence (float): the confidence for thresholding
- subset (str): the subset to use ('train'/'test')
- class_name (str|list): the names of the lesions constituting the positive class
- """
-
- testset = _prepare_configuration_diaretdb1(subset=subset,
- class_name=class_name,
- confidence=confidence,
- pixel_level=False,
- assumption='fov')
-
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec'],
- aggregation='som',
- random_state=random_seed)
-
- results = check_diaretdb1_class(subset=subset,
- class_name=class_name,
- confidence=confidence,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('subset', ['train', 'test'])
-@pytest.mark.parametrize('class_name', class_names)
-def test_check_failure_class(random_seed: int,
- confidence: float,
- subset: str,
- class_name):
- """
- Testing the image labeling in an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- confidence (float): the confidence for thresholding
- subset (str): the subset to use ('train'/'test')
- class_name (str|list): the names of the lesions constituting the positive class
- """
-
- testset = _prepare_configuration_diaretdb1(subset=subset,
- class_name=class_name,
- confidence=confidence,
- pixel_level=False,
- assumption='fov')
-
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec'],
- aggregation='som',
- random_state=random_seed)
- scores['acc'] = (1.0 + scores['spec']) / 2.0
-
- results = check_diaretdb1_class(subset=subset,
- class_name=class_name,
- confidence=confidence,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('class_name', class_names)
-@pytest.mark.parametrize('assumption', ['fov', 'all'])
-def test_check_success_segmentation_image(random_seed: int,
- confidence: float,
- class_name,
- assumption: str):
- """
- Testing the image labeling in a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- confidence (float): the confidence for thresholding
- class_name (str|list): the names of the lesions constituting the positive class
- assumption (str): the assumption on the region of evaluation ('fov'/'all')
- """
-
- testsets = _prepare_configuration_diaretdb1(subset='train',
- class_name=class_name,
- confidence=confidence,
- pixel_level=True,
- assumption=assumption) + \
- _prepare_configuration_diaretdb1(subset='test',
- class_name=class_name,
- confidence=confidence,
- pixel_level=True,
- assumption=assumption)
-
- test_idx = 0
- for test_idx, test_item in enumerate(testsets):
- if test_item['p'] > 0:
- break
-
- scores = generate_scores_for_testsets([testsets[test_idx]],
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec'],
- aggregation='som',
- random_state=random_seed)
-
- results = check_diaretdb1_segmentation_image(image_identifier=testsets[test_idx]['identifier'],
- class_name=class_name,
- confidence=confidence,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency'][f'inconsistency_{assumption}']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('class_name', class_names)
-@pytest.mark.parametrize('assumption', ['fov', 'all'])
-def test_check_failure_segmentation_image(random_seed: int,
- confidence: float,
- class_name,
- assumption: str):
- """
- Testing the image labeling in an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- confidence (float): the confidence for thresholding
- class_name (str|list): the names of the lesions constituting the positive class
- assumption (str): the assumption on the region of evaluation ('fov'/'all')
- """
-
- testsets = _prepare_configuration_diaretdb1(subset='train',
- class_name=class_name,
- confidence=confidence,
- pixel_level=True,
- assumption=assumption) + \
- _prepare_configuration_diaretdb1(subset='test',
- class_name=class_name,
- confidence=confidence,
- pixel_level=True,
- assumption=assumption)
-
- test_idx = 0
- for test_idx, test_item in enumerate(testsets):
- if test_item['p'] > 0:
- break
-
- scores = generate_scores_for_testsets([testsets[test_idx]],
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec'],
- aggregation='som',
- random_state=random_seed)
-
- scores['acc'] = (1.0 + scores['spec']) / 2.0
-
- results = check_diaretdb1_segmentation_image(image_identifier=testsets[test_idx]['identifier'],
- class_name=class_name,
- confidence=confidence,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency'][f'inconsistency_{assumption}']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('subset', ['train', 'test', test_identifiers])
-@pytest.mark.parametrize('class_name', class_names)
-@pytest.mark.parametrize('assumption', ['fov', 'all'])
-@pytest.mark.parametrize('aggregation', ['mos', 'som'])
-def test_check_success_segmentation_aggregated(*,
- random_seed: int,
- confidence: float,
- subset,
- class_name,
- assumption: str,
- aggregation: str):
- """
- Testing the image labeling in a consistent setup with all images
-
- Args:
- random_seed (int): the random seed to use
- confidence (float): the confidence for thresholding
- subset (str|list): the subset to use ('train'/'test')
- class_name (str|list): the names of the lesions constituting the positive class
- assumption (str): the assumption on the region of evaluation ('fov'/'all')
- aggregation (str): the mode of aggregation ('som'/'mos')
- """
-
- testsets = _prepare_configuration_diaretdb1(subset=subset,
- class_name=class_name,
- confidence=confidence,
- pixel_level=True,
- assumption=assumption,
- only_valid=True)
-
- scores = generate_scores_for_testsets(testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec'],
- aggregation=aggregation,
- random_state=random_seed)
-
- results = check_diaretdb1_segmentation_aggregated(subset=subset,
- class_name=class_name,
- confidence=confidence,
- only_valid=True,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency'][f'inconsistency_{assumption}_{aggregation}']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('subset', ['train', 'test', test_identifiers])
-@pytest.mark.parametrize('class_name', class_names)
-@pytest.mark.parametrize('assumption', ['fov', 'all'])
-@pytest.mark.parametrize('aggregation', ['som', 'mos'])
-def test_check_failure_segmentation_aggregated(*,
- random_seed: int,
- confidence: float,
- subset,
- class_name,
- assumption: str,
- aggregation: str):
- """
- Testing the image labeling in an inconsistent setup with all images
-
- Args:
- random_seed (int): the random seed to use
- confidence (float): the confidence for thresholding
- subset (str|list): the subset to use ('train'/'test')
- class_name (str|list): the names of the lesions constituting the positive class
- assumption (str): the assumption on the region of evaluation ('fov'/'all')
- aggregation (str): the mode of aggregation ('som'/'mos')
- """
-
- testsets = _prepare_configuration_diaretdb1(subset=subset,
- class_name=class_name,
- confidence=confidence,
- pixel_level=True,
- assumption=assumption,
- only_valid=True)
-
- scores = generate_scores_for_testsets(testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec'],
- aggregation=aggregation,
- random_state=random_seed)
-
- scores['acc'] = (1.0 + scores['spec']) / 2.0
-
- results = check_diaretdb1_segmentation_aggregated(subset=subset,
- class_name=class_name,
- confidence=confidence,
- only_valid=True,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency'][f'inconsistency_{assumption}_{aggregation}']
-
-def test_no_mos():
- """
- Testing the case when MoS cannot be tested
- """
-
- testsets = _prepare_configuration_diaretdb1(subset=test_identifiers[:10],
- class_name='hardexudates',
- confidence=0.75,
- pixel_level=True,
- assumption='fov')
-
- scores = generate_scores_for_testsets(testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec'],
- aggregation='som',
- random_state=1)
-
- results = check_diaretdb1_segmentation_aggregated_assumption(
- subset=test_identifiers[:10],
- class_name='hardexudates',
- confidence=0.75,
- assumption='fov',
- only_valid=False,
- scores=scores,
- eps=1e-4)
-
- assert 'details_mos' not in results
diff --git a/tests/bundles/test_retina_drishti_gs.py b/tests/bundles/test_retina_drishti_gs.py
deleted file mode 100644
index d5ed9c7..0000000
--- a/tests/bundles/test_retina_drishti_gs.py
+++ /dev/null
@@ -1,166 +0,0 @@
-"""
-This module tests the tests developed for the DRISHTI_GS dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.retina import (_prepare_testsets_drishti_gs,
- check_drishti_gs_segmentation_image,
- check_drishti_gs_segmentation_aggregated)
-
-from mlscorecheck.experiments import get_experiment
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-data = get_experiment('retina.drishti_gs')
-test_identifiers = list(data['test'].keys())
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('image_identifier', test_identifiers)
-@pytest.mark.parametrize('target', ['OD', 'OC'])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-def test_image_consistent(random_seed: int,
- image_identifier: str,
- target: str,
- confidence: float):
- """
- Testing the image level tests with a consistent configuration
-
- Args:
- random_seed (int): the random seed to use
- image_identifier (str): the image identifier
- target (str): the target
- confidence (float): the confidence level
- """
- testset = _prepare_testsets_drishti_gs(subset=[image_identifier],
- target=target,
- confidence=confidence)[0]
-
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation='mos',
- random_state=random_seed)
-
- results = check_drishti_gs_segmentation_image(image_identifier=image_identifier,
- confidence=confidence,
- target=target,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('image_identifier', test_identifiers)
-@pytest.mark.parametrize('target', ['OD', 'OC'])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-def test_image_inconsistent(random_seed: int,
- image_identifier: str,
- target: str,
- confidence: float):
- """
- Testing the image level tests with an inconsistent configuration
-
- Args:
- random_seed (int): the random seed to use
- image_identifier (str): the image identifier
- target (str): the target
- confidence (float): the confidence level
- """
- testset = _prepare_testsets_drishti_gs(subset=[image_identifier],
- target=target,
- confidence=confidence)[0]
-
- scores = generate_scores_for_testsets([testset],
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation='mos',
- random_state=random_seed)
-
- scores['acc'] = (1.0 + scores['spec']) / 2.0
-
- results = check_drishti_gs_segmentation_image(image_identifier=image_identifier,
- confidence=confidence,
- target=target,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']
-
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('subset', ['train', 'test', test_identifiers])
-@pytest.mark.parametrize('target', ['OD', 'OC'])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('aggregation', ['som', 'mos'])
-def test_aggregated_consistent(random_seed: int,
- subset,
- target: str,
- confidence: float,
- aggregation: str):
- """
- Testing the consistency test for aggregated scores in a consistent case
-
- Args:
- random_seed (int): the random seed to use
- subset (str|list): the image subset to be used
- target (str): the target
- confidence (float): the confidence level
- aggregation (str): the aggregation to be used
- """
- testsets = _prepare_testsets_drishti_gs(subset=subset,
- target=target,
- confidence=confidence)
-
- scores = generate_scores_for_testsets(testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation=aggregation,
- random_state=random_seed)
-
- results = check_drishti_gs_segmentation_aggregated(subset=subset,
- confidence=confidence,
- target=target,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency'][f'inconsistency_{aggregation}']
-
-@pytest.mark.parametrize('random_seed', [1, 2, 3, 4, 5])
-@pytest.mark.parametrize('subset', ['train', 'test', test_identifiers])
-@pytest.mark.parametrize('target', ['OD', 'OC'])
-@pytest.mark.parametrize('confidence', [0.5, 0.75])
-@pytest.mark.parametrize('aggregation', ['som', 'mos'])
-def test_aggregated_inconsistent(random_seed: int,
- subset,
- target: str,
- confidence: float,
- aggregation: str):
- """
- Testing the consistency test for aggregated scores in an inconsistent case
-
- Args:
- random_seed (int): the random seed to use
- subset (str|list): the image subset to be used
- target (str): the target
- confidence (float): the confidence level
- aggregation (str): the aggregation to be used
- """
- testsets = _prepare_testsets_drishti_gs(subset=subset,
- target=target,
- confidence=confidence)
-
- scores = generate_scores_for_testsets(testsets,
- rounding_decimals=4,
- subset=['acc', 'sens', 'spec', 'f1p'],
- aggregation=aggregation,
- random_state=random_seed)
-
- scores['acc'] = (1.0 + scores['sens']) / 2.0
-
- results = check_drishti_gs_segmentation_aggregated(subset=subset,
- confidence=confidence,
- target=target,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency'][f'inconsistency_{aggregation}']
diff --git a/tests/bundles/test_retina_drive.py b/tests/bundles/test_retina_drive.py
deleted file mode 100644
index 971b4e9..0000000
--- a/tests/bundles/test_retina_drive.py
+++ /dev/null
@@ -1,184 +0,0 @@
-"""
-This file tests the test bundle for the DRIVE dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.retina import (check_drive_vessel_image,
- check_drive_vessel_aggregated)
-
-from mlscorecheck.experiments import get_experiment
-
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_mos(random_state):
- """
- Testing a consistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.drive')[(1, 'fov')]['train']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
- results = check_drive_vessel_aggregated(imageset='train',
- annotator=1,
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_fov_mos']
-
- results = check_drive_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator=1,
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_fov_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_mos(random_state):
- """
- Testing an inconsistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.drive')[(1, 'fov')]['train']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_drive_vessel_aggregated(imageset='train',
- annotator=1,
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_fov_mos']
-
- results = check_drive_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator=1,
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_fov_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_som(random_state):
- """
- Testing a consistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.drive')[(1, 'fov')]['train']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_drive_vessel_aggregated(imageset='train',
- annotator=1,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_fov_som']
-
- results = check_drive_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator=1,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_fov_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_som(random_state):
- """
- Testing an inconsistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.drive')[(1, 'fov')]['train']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_drive_vessel_aggregated(imageset='train',
- annotator=1,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_fov_som']
-
- results = check_drive_vessel_aggregated(imageset=[img['identifier'] for img in data],
- annotator=1,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_fov_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_image(random_state):
- """
- Testing a consistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.drive')[(1, 'fov')]['train']['images']
-
- scores = generate_scores_for_testsets([data[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_drive_vessel_image(image_identifier=data[0]['identifier'],
- annotator=1,
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_fov']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_image(random_state):
- """
- Testing an inconsistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
- data = get_experiment('retina.drive')[(1, 'fov')]['train']['images']
-
- scores = generate_scores_for_testsets([data[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_drive_vessel_image(image_identifier=data[0]['identifier'],
- annotator=1,
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_fov']
diff --git a/tests/bundles/test_retina_hrf.py b/tests/bundles/test_retina_hrf.py
deleted file mode 100644
index 9e43d30..0000000
--- a/tests/bundles/test_retina_hrf.py
+++ /dev/null
@@ -1,175 +0,0 @@
-"""
-This file tests the test bundle for the HRF dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.retina import (check_hrf_vessel_image,
- check_hrf_vessel_aggregated)
-
-from mlscorecheck.experiments import get_experiment
-
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_mos(random_state):
- """
- Testing a consistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.hrf')[('fov')]['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_hrf_vessel_aggregated(imageset='all',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_fov_mos']
-
- results = check_hrf_vessel_aggregated(imageset=[tmp['identifier'] for tmp in data],
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_fov_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_mos(random_state):
- """
- Testing an inconsistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.hrf')[('fov')]['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_hrf_vessel_aggregated(imageset='all',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_fov_mos']
-
- results = check_hrf_vessel_aggregated(imageset=[img['identifier'] for img in data],
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_fov_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_som(random_state):
- """
- Testing a consistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.hrf')['fov']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_hrf_vessel_aggregated(imageset='all',
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_fov_som']
-
- results = check_hrf_vessel_aggregated(imageset=[img['identifier'] for img in data],
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_fov_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_som(random_state):
- """
- Testing an inconsistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.hrf')['fov']['images']
-
- scores = generate_scores_for_testsets(data,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_hrf_vessel_aggregated(imageset='all',
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_fov_som']
-
- results = check_hrf_vessel_aggregated(imageset=[img['identifier'] for img in data],
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_fov_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_image(random_state):
- """
- Testing a consistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
-
- data = get_experiment('retina.hrf')['fov']['images']
-
- scores = generate_scores_for_testsets([data[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_hrf_vessel_image(image_identifier=data[0]['identifier'],
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_fov']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_image(random_state):
- """
- Testing an inconsistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
- data = get_experiment('retina.hrf')['fov']['images']
-
- scores = generate_scores_for_testsets([data[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_hrf_vessel_image(image_identifier=data[0]['identifier'],
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_fov']
diff --git a/tests/bundles/test_retina_stare.py b/tests/bundles/test_retina_stare.py
deleted file mode 100644
index 4dc05ec..0000000
--- a/tests/bundles/test_retina_stare.py
+++ /dev/null
@@ -1,185 +0,0 @@
-"""
-This file tests the test bundle for the STARE dataset
-"""
-
-import pytest
-
-from mlscorecheck.bundles.retina import (check_stare_vessel_image,
- check_stare_vessel_aggregated)
-
-from mlscorecheck.experiments import get_experiment
-
-from mlscorecheck.aggregated import generate_scores_for_testsets
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_mos(random_state):
- """
- Testing a consistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- dataah = get_experiment('retina.stare')['ah']['images']
-
- scores = generate_scores_for_testsets(dataah,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
- results = check_stare_vessel_aggregated(imageset='all',
- annotator='ah',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_mos']
-
- results = check_stare_vessel_aggregated(imageset=[img['identifier'] for img in dataah],
- annotator='ah',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert not results['inconsistency']['inconsistency_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_mos(random_state):
- """
- Testing an inconsistent setup with MoS aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- dataah = get_experiment('retina.stare')['ah']['images']
-
- scores = generate_scores_for_testsets(dataah,
- aggregation='mos',
- rounding_decimals=4,
- random_state=random_state)
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_stare_vessel_aggregated(imageset='all',
- annotator='ah',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_mos']
-
- results = check_stare_vessel_aggregated(imageset=[img['identifier'] for img in dataah],
- annotator='ah',
- scores=scores,
- eps=1e-4,
- verbosity=0)
-
- assert results['inconsistency']['inconsistency_mos']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_som(random_state):
- """
- Testing a consistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- dataah = get_experiment('retina.stare')['ah']['images']
-
- scores = generate_scores_for_testsets(dataah,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_stare_vessel_aggregated(imageset='all',
- annotator='ah',
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_som']
-
- results = check_stare_vessel_aggregated(imageset=[img['identifier'] for img in dataah],
- annotator='ah',
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']['inconsistency_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_som(random_state):
- """
- Testing an inconsistent setup with SoM aggregation
-
- Args:
- random_state (int): the random seed to use
- """
-
- dataah = get_experiment('retina.stare')['ah']['images']
-
- scores = generate_scores_for_testsets(dataah,
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_stare_vessel_aggregated(imageset='all',
- annotator='ah',
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_som']
-
- results = check_stare_vessel_aggregated(imageset=[img['identifier'] for img in dataah],
- annotator='ah',
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']['inconsistency_som']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_success_image(random_state):
- """
- Testing a consistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
-
- dataah = get_experiment('retina.stare')['ah']['images']
-
- scores = generate_scores_for_testsets([dataah[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- results = check_stare_vessel_image(image_identifier=dataah[0]['identifier'],
- annotator='ah',
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_state', [1, 2, 3, 4, 5])
-def test_failure_image(random_state):
- """
- Testing an inconsistent setup for an image
-
- Args:
- random_state (int): the random seed to use
- """
- dataah = get_experiment('retina.stare')['ah']['images']
-
- scores = generate_scores_for_testsets([dataah[0]],
- aggregation='som',
- rounding_decimals=4,
- random_state=random_state)
-
- scores['acc'] = (1.0 + scores['spec'])/2.0
-
- results = check_stare_vessel_image(image_identifier=dataah[0]['identifier'],
- annotator='ah',
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']
diff --git a/tests/bundles/test_tpehg.py b/tests/bundles/test_tpehg.py
deleted file mode 100644
index e0faf14..0000000
--- a/tests/bundles/test_tpehg.py
+++ /dev/null
@@ -1,17 +0,0 @@
-"""
-Testing the EHG testing
-"""
-
-from mlscorecheck.bundles.ehg import check_tpehg
-
-def test_check_tpehg():
- """
- Testing the check_tpehg function
- """
-
- results = check_tpehg(scores={'acc': 0.95, 'sens': 0.95, 'spec': 0.95},
- eps=1e-4,
- n_folds=5,
- n_repeats=1)
-
- assert 'inconsistency' in results
diff --git a/tests/check/binary/test_1_dataset_known_folds_mos_scores.py b/tests/check/binary/test_1_dataset_known_folds_mos_scores.py
new file mode 100644
index 0000000..45519da
--- /dev/null
+++ b/tests/check/binary/test_1_dataset_known_folds_mos_scores.py
@@ -0,0 +1,125 @@
+"""
+Testing the checking of scores on 1 dataset using kfold with mean of scores
+aggregation
+"""
+
+import pytest
+
+from mlscorecheck.check.binary import check_1_dataset_known_folds_mos
+from mlscorecheck.aggregated import generate_evaluation
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ evaluation, scores = generate_evaluation(
+ aggregation="mos",
+ random_state=random_seed,
+ return_scores=True,
+ rounding_decimals=rounding_decimals,
+ )
+
+ result = check_1_dataset_known_folds_mos(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure(random_seed: int, rounding_decimals: int):
+ """
+ Testing with an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ evaluation, scores = generate_evaluation(
+ aggregation="mos",
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_1_dataset_known_folds_mos(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency_bounds(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup and bounds
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ evaluation, scores = generate_evaluation(
+ aggregation="mos",
+ random_state=random_seed,
+ return_scores=True,
+ feasible_fold_score_bounds=True,
+ rounding_decimals=rounding_decimals,
+ )
+
+ result = check_1_dataset_known_folds_mos(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ fold_score_bounds=evaluation["fold_score_bounds"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ timeout=1,
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure_bounds(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a inconsistent setup and bounds
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ evaluation, scores = generate_evaluation(
+ aggregation="mos",
+ random_state=random_seed,
+ return_scores=True,
+ feasible_fold_score_bounds=False,
+ rounding_decimals=rounding_decimals,
+ )
+ scores = {"acc": 0.9, "bacc": 0.1, "sens": 0.1, "npv": 0.1, "ppv": 0.1, "f1": 0.9}
+
+ result = check_1_dataset_known_folds_mos(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ fold_score_bounds=evaluation["fold_score_bounds"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ timeout=1,
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
diff --git a/tests/check/binary/test_1_dataset_som_scores.py b/tests/check/binary/test_1_dataset_som_scores.py
new file mode 100644
index 0000000..8959d7f
--- /dev/null
+++ b/tests/check/binary/test_1_dataset_som_scores.py
@@ -0,0 +1,83 @@
+"""
+Testing the checking of scores on 1 dataset using kfold with SoM aggregation
+"""
+
+import pytest
+
+from mlscorecheck.check.binary import check_1_dataset_kfold_som
+from mlscorecheck.aggregated import generate_evaluation
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ evaluation, scores = generate_evaluation(
+ aggregation="som",
+ random_state=random_seed,
+ return_scores=True,
+ rounding_decimals=rounding_decimals,
+ )
+
+ result = check_1_dataset_kfold_som(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure(random_seed: int, rounding_decimals: int):
+ """
+ Testing with an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ evaluation, scores = generate_evaluation(
+ aggregation="som",
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_1_dataset_kfold_som(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert result["inconsistency"]
+
+
+def test_adding_strategy():
+ """
+ Testing the addition of strategy
+ """
+ evaluation = {
+ "dataset": {"p": 5, "n": 6},
+ "folding": {"n_folds": 2, "n_repeats": 1},
+ }
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_1_dataset_kfold_som(
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ scores=scores,
+ eps=10 ** (-4),
+ )
+
+ assert result["inconsistency"]
diff --git a/tests/check/binary/test_1_dataset_unknown_folds_mos_scores.py b/tests/check/binary/test_1_dataset_unknown_folds_mos_scores.py
new file mode 100644
index 0000000..77b0da0
--- /dev/null
+++ b/tests/check/binary/test_1_dataset_unknown_folds_mos_scores.py
@@ -0,0 +1,112 @@
+"""
+Testing the test functionality with MoS aggregation and unknown folds
+"""
+
+import pytest
+
+import numpy as np
+
+from mlscorecheck.check.binary import (
+ check_1_dataset_unknown_folds_mos,
+ estimate_n_evaluations,
+)
+from mlscorecheck.aggregated import Evaluation
+
+subsets = [["acc", "sens", "spec", "bacc"], ["acc", "sens"], ["acc", "spec"], ["acc"]]
+
+
+def test_estimation():
+ """
+ Testing the evaluation count estimation
+ """
+
+ count = estimate_n_evaluations(
+ dataset={"p": 5, "n": 11},
+ folding={"n_folds": 3, "n_repeats": 2},
+ available_scores=["acc", "sens", "spec"],
+ )
+
+ assert count == 16
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [4])
+@pytest.mark.parametrize("subset", subsets)
+def test_success(random_seed: int, rounding_decimals: int, subset: list):
+ """
+ Testing with successful configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ subset (list): the subset of scores to work with
+ """
+ random_state = np.random.RandomState(random_seed)
+
+ dataset = {"n": random_state.randint(1, 150), "p": random_state.randint(1, 15)}
+ folding = {
+ "n_folds": random_state.randint(1, 5),
+ "n_repeats": random_state.randint(1, 3),
+ "strategy": "stratified_sklearn",
+ }
+ folding["n_folds"] = min(folding["n_folds"], dataset["p"], dataset["n"])
+
+ evaluation = Evaluation(dataset=dataset, folding=folding, aggregation="mos")
+ evaluation.sample_figures(score_subset=subset)
+
+ scores = evaluation.calculate_scores(rounding_decimals, score_subset=subset)
+
+ evaluation = evaluation.to_dict()
+
+ results = check_1_dataset_unknown_folds_mos(
+ scores=scores,
+ eps=(10 ** (-rounding_decimals)),
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ fold_score_bounds=evaluation.get("fold_score_bounds"),
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [4])
+@pytest.mark.parametrize("subset", subsets)
+def test_failure(random_seed: int, rounding_decimals: int, subset: list):
+ """
+ Testing with successful configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ subset (list): the subset of scores to work with
+ """
+
+ random_state = np.random.RandomState(random_seed)
+
+ dataset = {"n": random_state.randint(1, 150), "p": random_state.randint(1, 15)}
+ folding = {
+ "n_folds": random_state.randint(1, 3),
+ "n_repeats": random_state.randint(1, 3),
+ "strategy": "stratified_sklearn",
+ }
+ folding["n_folds"] = min(folding["n_folds"], dataset["p"], dataset["n"])
+
+ evaluation = Evaluation(dataset=dataset, folding=folding, aggregation="mos")
+ evaluation.sample_figures(score_subset=subset)
+
+ evaluation = evaluation.to_dict()
+ del evaluation["aggregation"]
+
+ scores = {"sens": 0.7, "spec": 0.7, "bacc": 0.6, "acc": 0.1234}
+ scores = {key: value for key, value in scores.items() if key in subset}
+
+ results = check_1_dataset_unknown_folds_mos(
+ scores=scores,
+ eps=(10 ** (-rounding_decimals)),
+ dataset=evaluation["dataset"],
+ folding=evaluation["folding"],
+ fold_score_bounds=evaluation.get("fold_score_bounds"),
+ )
+
+ assert results["inconsistency"]
diff --git a/tests/check/binary/test_1_testset_no_kfold_scores.py b/tests/check/binary/test_1_testset_no_kfold_scores.py
new file mode 100644
index 0000000..683af71
--- /dev/null
+++ b/tests/check/binary/test_1_testset_no_kfold_scores.py
@@ -0,0 +1,103 @@
+"""
+Testing the use case regarding one dataset
+"""
+
+import warnings
+
+import pytest
+
+from mlscorecheck.check.binary import check_1_testset_no_kfold
+from mlscorecheck.individual import generate_1_problem
+from mlscorecheck.scores import calculate_scores
+
+k = 4 # pylint: disable=invalid-name
+eps = 10 ** (-k) # pylint: disable=invalid-name
+
+
+def test_parametrization():
+ """
+ Testing the parametrization
+ """
+ with pytest.raises(ValueError):
+ check_1_testset_no_kfold(scores={}, eps=1e-4, testset={"p": 5})
+
+
+def test_warnings():
+ """
+ Testing the warning
+ """
+
+ problem = {"name": "common_datasets.ADA", "n_repeats": 2}
+
+ scores = {"acc": 0.9, "sens": 0.89, "spec": 0.91}
+
+ with warnings.catch_warnings(record=True) as warns:
+ warnings.simplefilter("always")
+ result = check_1_testset_no_kfold(scores=scores, eps=1e-2, testset=problem)
+ assert len(warns) == 1
+
+ assert not result["inconsistency"]
+
+
+def test_consistent():
+ """
+ Testing a consistent configuration
+ """
+ evaluation, problem = generate_1_problem(random_state=5)
+
+ evaluation["beta_negative"] = 2
+ evaluation["beta_positive"] = 2
+
+ scores = calculate_scores(evaluation, rounding_decimals=k)
+ scores["beta_negative"] = 2
+ scores["beta_positive"] = 2
+
+ result = check_1_testset_no_kfold(scores=scores, eps=eps, testset=problem)
+
+ assert not result["inconsistency"]
+
+
+def test_failure():
+ """
+ Testing a failure
+ """
+ evaluation, problem = generate_1_problem(random_state=5)
+
+ evaluation["beta_negative"] = 2
+ evaluation["beta_positive"] = 2
+
+ scores = calculate_scores(evaluation, rounding_decimals=k)
+ scores["bacc"] = 0.9
+ scores["acc"] = 0.1
+ scores["beta_negative"] = 2
+ scores["beta_positive"] = 2
+
+ result = check_1_testset_no_kfold(scores=scores, eps=eps, testset=problem)
+
+ assert result["inconsistency"]
+
+
+def test_dataset():
+ """
+ Testing success with real dataset
+ """
+ problem = {"name": "common_datasets.ADA"}
+
+ scores = {"acc": 0.9, "sens": 0.89, "spec": 0.91}
+
+ result = check_1_testset_no_kfold(scores=scores, eps=1e-2, testset=problem)
+
+ assert not result["inconsistency"]
+
+
+def test_dataset_failure():
+ """
+ Testing failure with real dataset with extreme precision and random scores
+ """
+ problem = {"name": "common_datasets.ADA"}
+
+ scores = {"acc": 0.9, "sens": 0.89, "spec": 0.91}
+
+ result = check_1_testset_no_kfold(scores=scores, eps=0.00001, testset=problem)
+
+ assert result["inconsistency"]
diff --git a/tests/check/binary/test_n_datasets_mos_kfold_som_scores.py b/tests/check/binary/test_n_datasets_mos_kfold_som_scores.py
new file mode 100644
index 0000000..c17f419
--- /dev/null
+++ b/tests/check/binary/test_n_datasets_mos_kfold_som_scores.py
@@ -0,0 +1,145 @@
+"""
+Testing the checking of scores on multiple datasets using kfold with
+score-of-means aggregation over the folds and mean-of-scores aggregation
+over the datasets
+"""
+
+import pytest
+
+from mlscorecheck.check.binary import check_n_datasets_mos_kfold_som
+from mlscorecheck.aggregated import generate_experiment
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "som"},
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+
+ result = check_n_datasets_mos_kfold_som(
+ evaluations=experiment["evaluations"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure(random_seed: int, rounding_decimals: int):
+ """
+ Testing with an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "som"},
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_n_datasets_mos_kfold_som(
+ evaluations=experiment["evaluations"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency_bounds(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup and bounds
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "som"},
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ feasible_dataset_score_bounds=True,
+ )
+
+ result = check_n_datasets_mos_kfold_som(
+ evaluations=experiment["evaluations"],
+ dataset_score_bounds=experiment["dataset_score_bounds"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ timeout=2,
+ )
+
+ assert not result["inconsistency"] or result["lp_status"] == "timeout"
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure_bounds(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a inconsistent setup and bounds
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "som"},
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ feasible_dataset_score_bounds=False,
+ )
+
+ result = check_n_datasets_mos_kfold_som(
+ evaluations=experiment["evaluations"],
+ dataset_score_bounds=experiment["dataset_score_bounds"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ timeout=2,
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
+
+
+def test_exception():
+ """
+ Testing the throwing of an exception
+ """
+
+ with pytest.raises(ValueError):
+ check_n_datasets_mos_kfold_som(
+ evaluations=[{"aggregation": "mos"}], scores={}, eps=1e-4
+ )
+
+ with pytest.raises(ValueError):
+ check_n_datasets_mos_kfold_som(
+ evaluations=[{"aggregation": "som", "fold_score_bounds": {}}],
+ scores={},
+ eps=1e-4,
+ )
diff --git a/tests/check/binary/test_n_datasets_mos_known_folds_mos_scores.py b/tests/check/binary/test_n_datasets_mos_known_folds_mos_scores.py
new file mode 100644
index 0000000..22a255b
--- /dev/null
+++ b/tests/check/binary/test_n_datasets_mos_known_folds_mos_scores.py
@@ -0,0 +1,147 @@
+"""
+Testing the checking of scores on multiple datasets using kfold with
+mean-of-scores aggregation over the folds and mean-of-scores aggregation
+over the datasets
+"""
+
+import pytest
+
+from mlscorecheck.check.binary import check_n_datasets_mos_known_folds_mos
+from mlscorecheck.aggregated import generate_experiment
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "mos"},
+ rounding_decimals=rounding_decimals,
+ random_state=random_seed,
+ return_scores=True,
+ )
+
+ result = check_n_datasets_mos_known_folds_mos(
+ evaluations=experiment["evaluations"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure(random_seed: int, rounding_decimals: int):
+ """
+ Testing with an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "mos"},
+ rounding_decimals=rounding_decimals,
+ random_state=random_seed,
+ return_scores=True,
+ )
+
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_n_datasets_mos_known_folds_mos(
+ evaluations=experiment["evaluations"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency_bounds(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup and bounds
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "mos"},
+ rounding_decimals=rounding_decimals,
+ random_state=random_seed,
+ feasible_dataset_score_bounds=True,
+ return_scores=True,
+ )
+
+ result = check_n_datasets_mos_known_folds_mos(
+ evaluations=experiment["evaluations"],
+ dataset_score_bounds=experiment["dataset_score_bounds"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ timeout=2,
+ )
+
+ assert not result["inconsistency"] or result["lp_status"] == "timeout"
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure_bounds(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a inconsistent setup and bounds
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={"aggregation": "mos"},
+ rounding_decimals=rounding_decimals,
+ random_state=random_seed,
+ feasible_dataset_score_bounds=True,
+ return_scores=True,
+ )
+
+ scores = {"acc": 0.5, "sens": 0.1, "spec": 0.2, "npv": 0.1, "ppv": 0.1, "f1": 0.9}
+
+ result = check_n_datasets_mos_known_folds_mos(
+ evaluations=experiment["evaluations"],
+ dataset_score_bounds=experiment["dataset_score_bounds"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ timeout=2,
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
+
+
+def test_exception():
+ """
+ Testing the throwing of an exception
+ """
+
+ with pytest.raises(ValueError):
+ check_n_datasets_mos_known_folds_mos(
+ evaluations=[{"aggregation": "som"}], scores={}, eps=1e-4
+ )
+
+ with pytest.raises(ValueError):
+ check_n_datasets_mos_known_folds_mos(
+ evaluations=[{"aggregation": "mos", "fold_score_bounds": {}}],
+ scores={},
+ eps=1e-4,
+ )
diff --git a/tests/check/binary/test_n_datasets_mos_unknown_folds_mos_scores.py b/tests/check/binary/test_n_datasets_mos_unknown_folds_mos_scores.py
new file mode 100644
index 0000000..12db7c9
--- /dev/null
+++ b/tests/check/binary/test_n_datasets_mos_unknown_folds_mos_scores.py
@@ -0,0 +1,177 @@
+"""
+Testing the case with multiple datasets and unknown folds in MoS aggregations
+"""
+
+import pytest
+
+import numpy as np
+
+from mlscorecheck.check.binary import (
+ check_n_datasets_mos_unknown_folds_mos,
+ estimate_n_experiments,
+)
+from mlscorecheck.aggregated import generate_experiment
+
+subsets = [["acc", "sens", "spec", "bacc"], ["acc", "sens"], ["acc", "spec"], ["acc"]]
+
+
+def test_estimation():
+ """
+ Testing the evaluation count estimation
+ """
+
+ count = estimate_n_experiments(
+ evaluations=[
+ {"dataset": {"p": 5, "n": 11}, "folding": {"n_folds": 3, "n_repeats": 2}},
+ {"dataset": {"p": 6, "n": 9}, "folding": {"n_folds": 3, "n_repeats": 2}},
+ ],
+ available_scores=["acc", "sens", "spec"],
+ )
+
+ assert count == 144
+
+
+def generate_test_case(
+ random_seed: int, rounding_decimals: int, score_subset: list
+) -> (dict, dict):
+ """
+ Generate one random test case
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ score_subset (list): the list of scores to be used
+
+ Returns:
+ dict, dict: the experiment specification and the scores
+ """
+ evaluation_params = {
+ "max_p": 25,
+ "max_n": 200,
+ "max_folds": 4,
+ "max_repeats": 1,
+ "aggregation": "mos",
+ "no_folds": True,
+ "no_name": True,
+ }
+
+ random_state = np.random.RandomState(random_seed)
+ experiment, scores = generate_experiment(
+ random_state=random_state,
+ rounding_decimals=rounding_decimals,
+ evaluation_params=evaluation_params,
+ max_evaluations=2,
+ aggregation="mos",
+ return_scores=True,
+ )
+
+ n_experiments = estimate_n_experiments(
+ experiment["evaluations"], list(scores.keys())
+ )
+
+ while n_experiments > 1000 or len(experiment["evaluations"]) == 1:
+ experiment, scores = generate_experiment(
+ random_state=random_state,
+ rounding_decimals=rounding_decimals,
+ evaluation_params=evaluation_params,
+ max_evaluations=2,
+ aggregation="mos",
+ return_scores=True,
+ )
+
+ n_experiments = estimate_n_experiments(
+ experiment["evaluations"], list(scores.keys())
+ )
+ scores = {key: value for key, value in scores.items() if key in score_subset}
+ return experiment, scores
+
+
+def remove_strategy_fsom_folding(experiment):
+ """
+ Removes the "strategy" from the folding
+
+ Args:
+ experiment (dict): an experiment specification
+ """
+ for evaluation in experiment["evaluations"]:
+ del evaluation["folding"]["strategy"]
+
+
+def test_remove_strategy_fsom_folding():
+ """
+ Testing the remove_strategy_fsom_folding function
+ """
+ experiment = {
+ "evaluations": [
+ {"folding": {"strategy": "dummy0"}},
+ {"folding": {"strategy": "dummy1"}},
+ ]
+ }
+
+ remove_strategy_fsom_folding(experiment)
+
+ assert "strategy" not in experiment["evaluations"][0]["folding"]
+ assert "strategy" not in experiment["evaluations"][1]["folding"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(5)))
+@pytest.mark.parametrize("subset", subsets)
+def test_successful(random_seed: int, subset: list):
+ """
+ Testing a successful scenario
+
+ Args:
+ random_seed (int): the random seed to use
+ subset (list): the subset of scores to be used
+ """
+ experiment, scores = generate_test_case(random_seed, 4, score_subset=subset)
+
+ remove_strategy_fsom_folding(experiment)
+
+ results = check_n_datasets_mos_unknown_folds_mos(
+ evaluations=experiment["evaluations"], scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(5)))
+@pytest.mark.parametrize("subset", subsets)
+def test_failure(random_seed: int, subset: list):
+ """
+ Testing a failure
+
+ Args:
+ random_seed (int): the random seed to use
+ subset (list): the subset of scores to be used
+ """
+
+ experiment, scores = generate_test_case(random_seed, 4, score_subset=subset)
+
+ remove_strategy_fsom_folding(experiment)
+
+ scores = {"acc": 0.9, "sens": 0.1, "spec": 0.1, "bacc": 0.05}
+
+ results = check_n_datasets_mos_unknown_folds_mos(
+ evaluations=experiment["evaluations"], scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"]
+
+
+def test_exception():
+ """
+ Testing the exception
+ """
+
+ with pytest.raises(ValueError):
+ check_n_datasets_mos_unknown_folds_mos(
+ evaluations=[{"aggregation": "som"}], scores={}, eps=1e-4
+ )
+
+ with pytest.raises(ValueError):
+ check_n_datasets_mos_unknown_folds_mos(
+ evaluations=[{"aggregation": "mos", "fold_score_bounds": {}}],
+ scores={},
+ eps=1e-4,
+ )
diff --git a/tests/check/binary/test_n_datasets_som_kfold_som_scores.py b/tests/check/binary/test_n_datasets_som_kfold_som_scores.py
new file mode 100644
index 0000000..e3cdaad
--- /dev/null
+++ b/tests/check/binary/test_n_datasets_som_kfold_som_scores.py
@@ -0,0 +1,78 @@
+"""
+Testing the checking of scores on multiple datasets using kfold with
+score-of-means aggregation on both levels
+"""
+
+import pytest
+
+from mlscorecheck.check.binary import check_n_datasets_som_kfold_som
+from mlscorecheck.aggregated import generate_experiment
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimal places to round to
+ """
+
+ experiment, scores = generate_experiment(
+ aggregation="som",
+ evaluation_params={"aggregation": "som"},
+ random_state=random_seed,
+ return_scores=True,
+ rounding_decimals=rounding_decimals,
+ )
+
+ result = check_n_datasets_som_kfold_som(
+ evaluations=experiment["evaluations"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure(random_seed: int, rounding_decimals: int):
+ """
+ Testing with an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimal places to round to
+ """
+
+ experiment, scores = generate_experiment(
+ aggregation="som",
+ evaluation_params={"aggregation": "som"},
+ random_state=random_seed,
+ return_scores=True,
+ rounding_decimals=rounding_decimals,
+ )
+
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_n_datasets_som_kfold_som(
+ evaluations=experiment["evaluations"],
+ scores=scores,
+ eps=10 ** (-rounding_decimals),
+ )
+
+ assert result["inconsistency"]
+
+
+def test_exception():
+ """
+ Testing the throwing of an exception
+ """
+
+ with pytest.raises(ValueError):
+ check_n_datasets_som_kfold_som(
+ evaluations=[{"aggregation": "mos"}], scores={}, eps=1e-4
+ )
diff --git a/tests/check/binary/test_n_testsets_mos_no_kfold_scores.py b/tests/check/binary/test_n_testsets_mos_no_kfold_scores.py
new file mode 100644
index 0000000..bf1a108
--- /dev/null
+++ b/tests/check/binary/test_n_testsets_mos_no_kfold_scores.py
@@ -0,0 +1,74 @@
+"""
+Testing the checking of scores on multiple datasets with
+mean-of-scores aggregation over the datasets
+"""
+
+import pytest
+
+from mlscorecheck.check.binary import check_n_testsets_mos_no_kfold
+from mlscorecheck.aggregated import generate_experiment
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={
+ "aggregation": "mos",
+ "max_folds": 1,
+ "max_repeats": 1,
+ "no_name": True,
+ },
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+ testsets = [evaluation["dataset"] for evaluation in experiment["evaluations"]]
+
+ result = check_n_testsets_mos_no_kfold(
+ testsets=testsets, scores=scores, eps=10 ** (-rounding_decimals)
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure(random_seed: int, rounding_decimals: int):
+ """
+ Testing with an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="mos",
+ evaluation_params={
+ "aggregation": "mos",
+ "max_folds": 1,
+ "max_repeats": 1,
+ "no_name": True,
+ },
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+ testsets = [evaluation["dataset"] for evaluation in experiment["evaluations"]]
+
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_n_testsets_mos_no_kfold(
+ testsets=testsets, scores=scores, eps=10 ** (-rounding_decimals)
+ )
+
+ assert result["inconsistency"]
diff --git a/tests/check/binary/test_n_testsets_som_no_kfold_scores.py b/tests/check/binary/test_n_testsets_som_no_kfold_scores.py
new file mode 100644
index 0000000..d248052
--- /dev/null
+++ b/tests/check/binary/test_n_testsets_som_no_kfold_scores.py
@@ -0,0 +1,74 @@
+"""
+Testing the checking of scores on multiple datasets with
+score-of-means aggregation over the datasets
+"""
+
+import pytest
+
+from mlscorecheck.check.binary import check_n_testsets_som_no_kfold
+from mlscorecheck.aggregated import generate_experiment
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_consistency(random_seed: int, rounding_decimals: int):
+ """
+ Testing with a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+
+ experiment, scores = generate_experiment(
+ aggregation="som",
+ evaluation_params={
+ "aggregation": "som",
+ "max_folds": 1,
+ "max_repeats": 1,
+ "no_name": True,
+ },
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+ testsets = [evaluation["dataset"] for evaluation in experiment["evaluations"]]
+
+ result = check_n_testsets_som_no_kfold(
+ testsets=testsets, scores=scores, eps=10 ** (-rounding_decimals)
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+@pytest.mark.parametrize("rounding_decimals", [3, 4])
+def test_failure(random_seed: int, rounding_decimals: int):
+ """
+ Testing with an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ rounding_decimals (int): the number of decimals to round to
+ """
+ experiment, scores = generate_experiment(
+ aggregation="som",
+ evaluation_params={
+ "aggregation": "som",
+ "max_folds": 1,
+ "max_repeats": 1,
+ "no_name": True,
+ },
+ random_state=random_seed,
+ rounding_decimals=rounding_decimals,
+ return_scores=True,
+ )
+ testsets = [evaluation["dataset"] for evaluation in experiment["evaluations"]]
+
+ scores = {"acc": 0.9, "sens": 0.3, "spec": 0.5, "f1": 0.1}
+
+ result = check_n_testsets_som_no_kfold(
+ testsets=testsets, scores=scores, eps=10 ** (-rounding_decimals)
+ )
+
+ assert result["inconsistency"]
diff --git a/tests/check/bundles/test_isic2016.py b/tests/check/bundles/test_isic2016.py
new file mode 100644
index 0000000..2b9b4c9
--- /dev/null
+++ b/tests/check/bundles/test_isic2016.py
@@ -0,0 +1,47 @@
+"""
+This module tests the tests developed for the ISIC2016 dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.skinlesion import check_isic2016
+from mlscorecheck.aggregated import generate_scores_for_testsets
+from mlscorecheck.experiments import get_experiment
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+def test_consistency(random_seed):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ testset = get_experiment("skinlesion.isic2016")
+ scores = generate_scores_for_testsets(
+ [testset], rounding_decimals=4, random_state=random_seed
+ )
+
+ results = check_isic2016(scores=scores, eps=1e-4)
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+def test_inconsistency(random_seed):
+ """
+ Testing an inconsistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ testset = get_experiment("skinlesion.isic2016")
+ scores = generate_scores_for_testsets(
+ [testset], rounding_decimals=4, random_state=random_seed
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_isic2016(scores=scores, eps=1e-4)
+ assert results["inconsistency"]
diff --git a/tests/check/bundles/test_isic2017.py b/tests/check/bundles/test_isic2017.py
new file mode 100644
index 0000000..6bb21ee
--- /dev/null
+++ b/tests/check/bundles/test_isic2017.py
@@ -0,0 +1,78 @@
+"""
+This module tests the tests developed for the ISIC2017 dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.skinlesion import (
+ check_isic2017,
+ _prepare_testset_isic2017,
+)
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+subsets = [
+ (["M"], ["SK", "N"]),
+ (["SK"], ["M", "N"]),
+ (["N"], ["SK", "M"]),
+ (["M", "SK"], ["N"]),
+ (["M", "N"], ["SK"]),
+ (["SK", "N"], ["M"]),
+]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("target_against", subsets)
+def test_consistency(random_seed, target_against):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ target_against (tuple(list,list)): the target and against classes
+ """
+
+ testset = _prepare_testset_isic2017(
+ target=target_against[0], against=target_against[1]
+ )
+ scores = generate_scores_for_testsets(
+ [testset],
+ rounding_decimals=4,
+ random_state=random_seed,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation="som",
+ )
+
+ results = check_isic2017(
+ target=target_against[0], against=target_against[1], scores=scores, eps=1e-4
+ )
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("target_against", subsets)
+def test_inconsistency(random_seed, target_against):
+ """
+ Testing an inconsistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ target_against (tuple(list,list)): the target and against classes
+ """
+
+ testset = _prepare_testset_isic2017(
+ target=target_against[0], against=target_against[1]
+ )
+ scores = generate_scores_for_testsets(
+ [testset],
+ rounding_decimals=4,
+ random_state=random_seed,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation="som",
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_isic2017(
+ target=target_against[0], against=target_against[1], scores=scores, eps=1e-4
+ )
+ assert results["inconsistency"]
diff --git a/tests/check/bundles/test_retina_chasedb1.py b/tests/check/bundles/test_retina_chasedb1.py
new file mode 100644
index 0000000..bff32cc
--- /dev/null
+++ b/tests/check/bundles/test_retina_chasedb1.py
@@ -0,0 +1,193 @@
+"""
+This file tests the test bundle for the CHASEDB1 dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.retina import (
+ check_chasedb1_vessel_image,
+ check_chasedb1_vessel_aggregated,
+)
+
+from mlscorecheck.experiments import get_experiment
+
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_mos(random_state):
+ """
+ Testing a consistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.chase_db1")["manual1"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+ results = check_chasedb1_vessel_aggregated(
+ imageset="all", annotator="manual1", scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert not results["inconsistency"]["inconsistency_mos"]
+
+ results = check_chasedb1_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator="manual1",
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert not results["inconsistency"]["inconsistency_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_mos(random_state):
+ """
+ Testing an inconsistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.chase_db1")["manual1"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_chasedb1_vessel_aggregated(
+ imageset="all", annotator="manual1", scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert results["inconsistency"]["inconsistency_mos"]
+
+ results = check_chasedb1_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator="manual1",
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert results["inconsistency"]["inconsistency_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_som(random_state):
+ """
+ Testing a consistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.chase_db1")["manual1"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_chasedb1_vessel_aggregated(
+ imageset="all", annotator="manual1", scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"]["inconsistency_som"]
+
+ results = check_chasedb1_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator="manual1",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"]["inconsistency_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_som(random_state):
+ """
+ Testing an inconsistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.chase_db1")["manual1"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_chasedb1_vessel_aggregated(
+ imageset="all", annotator="manual1", scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"]["inconsistency_som"]
+
+ results = check_chasedb1_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator="manual1",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"]["inconsistency_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_image(random_state):
+ """
+ Testing a consistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.chase_db1")["manual1"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [data[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_chasedb1_vessel_image(
+ image_identifier=data[0]["identifier"],
+ annotator="manual1",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_image(random_state):
+ """
+ Testing an inconsistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+ data = get_experiment("retina.chase_db1")["manual1"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [data[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_chasedb1_vessel_image(
+ image_identifier=data[0]["identifier"],
+ annotator="manual1",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"]
diff --git a/tests/check/bundles/test_retina_diaretdb0.py b/tests/check/bundles/test_retina_diaretdb0.py
new file mode 100644
index 0000000..85d395d
--- /dev/null
+++ b/tests/check/bundles/test_retina_diaretdb0.py
@@ -0,0 +1,108 @@
+"""
+This module tests the test bundle for the DIARETDB0 dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.retina import check_diaretdb0_class
+from mlscorecheck.check.bundles.retina import _prepare_configuration_diaretdb0
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+
+@pytest.mark.parametrize(
+ "class_name",
+ [
+ "neovascularisation",
+ "hardexudates",
+ "hemorrhages",
+ "softexudates",
+ "redsmalldots",
+ ],
+)
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
+def test_success(class_name, random_seed, aggregation):
+ """
+ Testing consistent configurations
+
+ Args:
+ class_name (str): the name of the class to test
+ random_seed (int): the random seed to be used
+ aggregation (str): the aggregation to be used
+ """
+
+ testsets = _prepare_configuration_diaretdb0("test", "all", class_name)
+ scores = generate_scores_for_testsets(
+ testsets=testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation=aggregation,
+ random_state=random_seed,
+ )
+
+ results = check_diaretdb0_class(
+ subset="test", batch="all", class_name=class_name, scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"][f"inconsistency_{aggregation}"]
+
+ testsets = _prepare_configuration_diaretdb0("test", ["1", "2"], class_name)
+ scores = generate_scores_for_testsets(
+ testsets=testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation=aggregation,
+ random_state=random_seed,
+ )
+
+ results = check_diaretdb0_class(
+ subset="test", batch=["1", "2"], class_name=class_name, scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"][f"inconsistency_{aggregation}"]
+
+
+@pytest.mark.parametrize(
+ "class_name",
+ [
+ "neovascularisation",
+ "hardexudates",
+ "hemorrhages",
+ "softexudates",
+ "redsmalldots",
+ ],
+)
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
+def test_failure(class_name, random_seed, aggregation):
+ """
+ Testing inconsistent configurations
+
+ Args:
+ class_name (str): the name of the class to test
+ random_seed (int): the random seed to be used
+ aggregation (str): the aggregation to be used
+ """
+
+ testsets = _prepare_configuration_diaretdb0("test", "all", class_name)
+ scores = generate_scores_for_testsets(
+ testsets=testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation=aggregation,
+ random_state=random_seed,
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_diaretdb0_class(
+ subset="test", batch="all", class_name=class_name, scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"][f"inconsistency_{aggregation}"]
+
+ results = check_diaretdb0_class(
+ subset="test", batch=["1", "2"], class_name=class_name, scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"][f"inconsistency_{aggregation}"]
diff --git a/tests/check/bundles/test_retina_diaretdb1.py b/tests/check/bundles/test_retina_diaretdb1.py
new file mode 100644
index 0000000..2cd0d1f
--- /dev/null
+++ b/tests/check/bundles/test_retina_diaretdb1.py
@@ -0,0 +1,377 @@
+"""
+This module tests the test functionalities implemented for the DIARETDB1 dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.retina import (
+ _prepare_configuration_diaretdb1,
+ check_diaretdb1_class,
+ check_diaretdb1_segmentation_image,
+ check_diaretdb1_segmentation_aggregated,
+ check_diaretdb1_segmentation_aggregated_assumption,
+)
+
+from mlscorecheck.experiments import get_experiment
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+class_names = [
+ ["hardexudates"],
+ ["softexudates"],
+ ["hemorrhages"],
+ ["redsmalldots"],
+ ["hardexudates", "softexudates"],
+ ["hemorrhages", "redsmalldots"],
+]
+
+data = get_experiment("retina.diaretdb1")
+test_identifiers = data["test"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("subset", ["train", "test"])
+@pytest.mark.parametrize("class_name", class_names)
+def test_check_success_class(
+ random_seed: str, confidence: float, subset: str, class_name
+):
+ """
+ Testing the image labeling in a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ confidence (float): the confidence for thresholding
+ subset (str): the subset to use ('train'/'test')
+ class_name (str|list): the names of the lesions constituting the positive class
+ """
+
+ testset = _prepare_configuration_diaretdb1(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=False,
+ assumption="fov",
+ )
+
+ scores = generate_scores_for_testsets(
+ [testset],
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec"],
+ aggregation="som",
+ random_state=random_seed,
+ )
+
+ results = check_diaretdb1_class(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("subset", ["train", "test"])
+@pytest.mark.parametrize("class_name", class_names)
+def test_check_failure_class(
+ random_seed: int, confidence: float, subset: str, class_name
+):
+ """
+ Testing the image labeling in an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ confidence (float): the confidence for thresholding
+ subset (str): the subset to use ('train'/'test')
+ class_name (str|list): the names of the lesions constituting the positive class
+ """
+
+ testset = _prepare_configuration_diaretdb1(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=False,
+ assumption="fov",
+ )
+
+ scores = generate_scores_for_testsets(
+ [testset],
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec"],
+ aggregation="som",
+ random_state=random_seed,
+ )
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_diaretdb1_class(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("class_name", class_names)
+@pytest.mark.parametrize("assumption", ["fov", "all"])
+def test_check_success_segmentation_image(
+ random_seed: int, confidence: float, class_name, assumption: str
+):
+ """
+ Testing the image labeling in a consistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ confidence (float): the confidence for thresholding
+ class_name (str|list): the names of the lesions constituting the positive class
+ assumption (str): the assumption on the region of evaluation ('fov'/'all')
+ """
+
+ testsets = _prepare_configuration_diaretdb1(
+ subset="train",
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=True,
+ assumption=assumption,
+ ) + _prepare_configuration_diaretdb1(
+ subset="test",
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=True,
+ assumption=assumption,
+ )
+
+ test_idx = 0
+ for test_idx, test_item in enumerate(testsets):
+ if test_item["p"] > 0:
+ break
+
+ scores = generate_scores_for_testsets(
+ [testsets[test_idx]],
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec"],
+ aggregation="som",
+ random_state=random_seed,
+ )
+
+ results = check_diaretdb1_segmentation_image(
+ image_identifier=testsets[test_idx]["identifier"],
+ class_name=class_name,
+ confidence=confidence,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"][f"inconsistency_{assumption}"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("class_name", class_names)
+@pytest.mark.parametrize("assumption", ["fov", "all"])
+def test_check_failure_segmentation_image(
+ random_seed: int, confidence: float, class_name, assumption: str
+):
+ """
+ Testing the image labeling in an inconsistent setup
+
+ Args:
+ random_seed (int): the random seed to use
+ confidence (float): the confidence for thresholding
+ class_name (str|list): the names of the lesions constituting the positive class
+ assumption (str): the assumption on the region of evaluation ('fov'/'all')
+ """
+
+ testsets = _prepare_configuration_diaretdb1(
+ subset="train",
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=True,
+ assumption=assumption,
+ ) + _prepare_configuration_diaretdb1(
+ subset="test",
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=True,
+ assumption=assumption,
+ )
+
+ test_idx = 0
+ for test_idx, test_item in enumerate(testsets):
+ if test_item["p"] > 0:
+ break
+
+ scores = generate_scores_for_testsets(
+ [testsets[test_idx]],
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec"],
+ aggregation="som",
+ random_state=random_seed,
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_diaretdb1_segmentation_image(
+ image_identifier=testsets[test_idx]["identifier"],
+ class_name=class_name,
+ confidence=confidence,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"][f"inconsistency_{assumption}"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("subset", ["train", "test", test_identifiers])
+@pytest.mark.parametrize("class_name", class_names)
+@pytest.mark.parametrize("assumption", ["fov", "all"])
+@pytest.mark.parametrize("aggregation", ["mos", "som"])
+def test_check_success_segmentation_aggregated(
+ *,
+ random_seed: int,
+ confidence: float,
+ subset,
+ class_name,
+ assumption: str,
+ aggregation: str,
+):
+ """
+ Testing the image labeling in a consistent setup with all images
+
+ Args:
+ random_seed (int): the random seed to use
+ confidence (float): the confidence for thresholding
+ subset (str|list): the subset to use ('train'/'test')
+ class_name (str|list): the names of the lesions constituting the positive class
+ assumption (str): the assumption on the region of evaluation ('fov'/'all')
+ aggregation (str): the mode of aggregation ('som'/'mos')
+ """
+
+ testsets = _prepare_configuration_diaretdb1(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=True,
+ assumption=assumption,
+ only_valid=True,
+ )
+
+ scores = generate_scores_for_testsets(
+ testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec"],
+ aggregation=aggregation,
+ random_state=random_seed,
+ )
+
+ results = check_diaretdb1_segmentation_aggregated(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ only_valid=True,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"][f"inconsistency_{assumption}_{aggregation}"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("subset", ["train", "test", test_identifiers])
+@pytest.mark.parametrize("class_name", class_names)
+@pytest.mark.parametrize("assumption", ["fov", "all"])
+@pytest.mark.parametrize("aggregation", ["som", "mos"])
+def test_check_failure_segmentation_aggregated(
+ *,
+ random_seed: int,
+ confidence: float,
+ subset,
+ class_name,
+ assumption: str,
+ aggregation: str,
+):
+ """
+ Testing the image labeling in an inconsistent setup with all images
+
+ Args:
+ random_seed (int): the random seed to use
+ confidence (float): the confidence for thresholding
+ subset (str|list): the subset to use ('train'/'test')
+ class_name (str|list): the names of the lesions constituting the positive class
+ assumption (str): the assumption on the region of evaluation ('fov'/'all')
+ aggregation (str): the mode of aggregation ('som'/'mos')
+ """
+
+ testsets = _prepare_configuration_diaretdb1(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ pixel_level=True,
+ assumption=assumption,
+ only_valid=True,
+ )
+
+ scores = generate_scores_for_testsets(
+ testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec"],
+ aggregation=aggregation,
+ random_state=random_seed,
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_diaretdb1_segmentation_aggregated(
+ subset=subset,
+ class_name=class_name,
+ confidence=confidence,
+ only_valid=True,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"][f"inconsistency_{assumption}_{aggregation}"]
+
+
+def test_no_mos():
+ """
+ Testing the case when MoS cannot be tested
+ """
+
+ testsets = _prepare_configuration_diaretdb1(
+ subset=test_identifiers[:10],
+ class_name="hardexudates",
+ confidence=0.75,
+ pixel_level=True,
+ assumption="fov",
+ )
+
+ scores = generate_scores_for_testsets(
+ testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec"],
+ aggregation="som",
+ random_state=1,
+ )
+
+ results = check_diaretdb1_segmentation_aggregated_assumption(
+ subset=test_identifiers[:10],
+ class_name="hardexudates",
+ confidence=0.75,
+ assumption="fov",
+ only_valid=False,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert "details_mos" not in results
diff --git a/tests/check/bundles/test_retina_drishti_gs.py b/tests/check/bundles/test_retina_drishti_gs.py
new file mode 100644
index 0000000..f1db847
--- /dev/null
+++ b/tests/check/bundles/test_retina_drishti_gs.py
@@ -0,0 +1,173 @@
+"""
+This module tests the tests developed for the DRISHTI_GS dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.retina import (
+ _prepare_testsets_drishti_gs,
+ check_drishti_gs_segmentation_image,
+ check_drishti_gs_segmentation_aggregated,
+)
+
+from mlscorecheck.experiments import get_experiment
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+data = get_experiment("retina.drishti_gs")
+test_identifiers = list(data["test"].keys())
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("image_identifier", test_identifiers)
+@pytest.mark.parametrize("target", ["OD", "OC"])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+def test_image_consistent(
+ random_seed: int, image_identifier: str, target: str, confidence: float
+):
+ """
+ Testing the image level tests with a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ image_identifier (str): the image identifier
+ target (str): the target
+ confidence (float): the confidence level
+ """
+ testset = _prepare_testsets_drishti_gs(
+ subset=[image_identifier], target=target, confidence=confidence
+ )[0]
+
+ scores = generate_scores_for_testsets(
+ [testset],
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation="mos",
+ random_state=random_seed,
+ )
+
+ results = check_drishti_gs_segmentation_image(
+ image_identifier=image_identifier,
+ confidence=confidence,
+ target=target,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("image_identifier", test_identifiers)
+@pytest.mark.parametrize("target", ["OD", "OC"])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+def test_image_inconsistent(
+ random_seed: int, image_identifier: str, target: str, confidence: float
+):
+ """
+ Testing the image level tests with an inconsistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ image_identifier (str): the image identifier
+ target (str): the target
+ confidence (float): the confidence level
+ """
+ testset = _prepare_testsets_drishti_gs(
+ subset=[image_identifier], target=target, confidence=confidence
+ )[0]
+
+ scores = generate_scores_for_testsets(
+ [testset],
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation="mos",
+ random_state=random_seed,
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_drishti_gs_segmentation_image(
+ image_identifier=image_identifier,
+ confidence=confidence,
+ target=target,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("subset", ["train", "test", test_identifiers])
+@pytest.mark.parametrize("target", ["OD", "OC"])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("aggregation", ["som", "mos"])
+def test_aggregated_consistent(
+ random_seed: int, subset, target: str, confidence: float, aggregation: str
+):
+ """
+ Testing the consistency test for aggregated scores in a consistent case
+
+ Args:
+ random_seed (int): the random seed to use
+ subset (str|list): the image subset to be used
+ target (str): the target
+ confidence (float): the confidence level
+ aggregation (str): the aggregation to be used
+ """
+ testsets = _prepare_testsets_drishti_gs(
+ subset=subset, target=target, confidence=confidence
+ )
+
+ scores = generate_scores_for_testsets(
+ testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation=aggregation,
+ random_state=random_seed,
+ )
+
+ results = check_drishti_gs_segmentation_aggregated(
+ subset=subset, confidence=confidence, target=target, scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"][f"inconsistency_{aggregation}"]
+
+
+@pytest.mark.parametrize("random_seed", [1, 2, 3, 4, 5])
+@pytest.mark.parametrize("subset", ["train", "test", test_identifiers])
+@pytest.mark.parametrize("target", ["OD", "OC"])
+@pytest.mark.parametrize("confidence", [0.5, 0.75])
+@pytest.mark.parametrize("aggregation", ["som", "mos"])
+def test_aggregated_inconsistent(
+ random_seed: int, subset, target: str, confidence: float, aggregation: str
+):
+ """
+ Testing the consistency test for aggregated scores in an inconsistent case
+
+ Args:
+ random_seed (int): the random seed to use
+ subset (str|list): the image subset to be used
+ target (str): the target
+ confidence (float): the confidence level
+ aggregation (str): the aggregation to be used
+ """
+ testsets = _prepare_testsets_drishti_gs(
+ subset=subset, target=target, confidence=confidence
+ )
+
+ scores = generate_scores_for_testsets(
+ testsets,
+ rounding_decimals=4,
+ subset=["acc", "sens", "spec", "f1p"],
+ aggregation=aggregation,
+ random_state=random_seed,
+ )
+
+ scores["acc"] = (1.0 + scores["sens"]) / 2.0
+
+ results = check_drishti_gs_segmentation_aggregated(
+ subset=subset, confidence=confidence, target=target, scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"][f"inconsistency_{aggregation}"]
diff --git a/tests/check/bundles/test_retina_drive.py b/tests/check/bundles/test_retina_drive.py
new file mode 100644
index 0000000..55d48e7
--- /dev/null
+++ b/tests/check/bundles/test_retina_drive.py
@@ -0,0 +1,186 @@
+"""
+This file tests the test bundle for the DRIVE dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.retina import (
+ check_drive_vessel_image,
+ check_drive_vessel_aggregated,
+)
+
+from mlscorecheck.experiments import get_experiment
+
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_mos(random_state):
+ """
+ Testing a consistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.drive")[(1, "fov")]["train"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+ results = check_drive_vessel_aggregated(
+ imageset="train", annotator=1, scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov_mos"]
+
+ results = check_drive_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator=1,
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_mos(random_state):
+ """
+ Testing an inconsistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.drive")[(1, "fov")]["train"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_drive_vessel_aggregated(
+ imageset="train", annotator=1, scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert results["inconsistency"]["inconsistency_fov_mos"]
+
+ results = check_drive_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator=1,
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert results["inconsistency"]["inconsistency_fov_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_som(random_state):
+ """
+ Testing a consistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.drive")[(1, "fov")]["train"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_drive_vessel_aggregated(
+ imageset="train", annotator=1, scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov_som"]
+
+ results = check_drive_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator=1,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_som(random_state):
+ """
+ Testing an inconsistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.drive")[(1, "fov")]["train"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_drive_vessel_aggregated(
+ imageset="train", annotator=1, scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"]["inconsistency_fov_som"]
+
+ results = check_drive_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ annotator=1,
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"]["inconsistency_fov_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_image(random_state):
+ """
+ Testing a consistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.drive")[(1, "fov")]["train"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [data[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_drive_vessel_image(
+ image_identifier=data[0]["identifier"], annotator=1, scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_image(random_state):
+ """
+ Testing an inconsistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+ data = get_experiment("retina.drive")[(1, "fov")]["train"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [data[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_drive_vessel_image(
+ image_identifier=data[0]["identifier"], annotator=1, scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"]["inconsistency_fov"]
diff --git a/tests/check/bundles/test_retina_hrf.py b/tests/check/bundles/test_retina_hrf.py
new file mode 100644
index 0000000..3e47843
--- /dev/null
+++ b/tests/check/bundles/test_retina_hrf.py
@@ -0,0 +1,175 @@
+"""
+This file tests the test bundle for the HRF dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.retina import (
+ check_hrf_vessel_image,
+ check_hrf_vessel_aggregated,
+)
+
+from mlscorecheck.experiments import get_experiment
+
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_mos(random_state):
+ """
+ Testing a consistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.hrf")[("fov")]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_hrf_vessel_aggregated(
+ imageset="all", scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov_mos"]
+
+ results = check_hrf_vessel_aggregated(
+ imageset=[tmp["identifier"] for tmp in data],
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_mos(random_state):
+ """
+ Testing an inconsistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.hrf")[("fov")]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_hrf_vessel_aggregated(
+ imageset="all", scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert results["inconsistency"]["inconsistency_fov_mos"]
+
+ results = check_hrf_vessel_aggregated(
+ imageset=[img["identifier"] for img in data],
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert results["inconsistency"]["inconsistency_fov_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_som(random_state):
+ """
+ Testing a consistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.hrf")["fov"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_hrf_vessel_aggregated(imageset="all", scores=scores, eps=1e-4)
+
+ assert not results["inconsistency"]["inconsistency_fov_som"]
+
+ results = check_hrf_vessel_aggregated(
+ imageset=[img["identifier"] for img in data], scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_som(random_state):
+ """
+ Testing an inconsistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.hrf")["fov"]["images"]
+
+ scores = generate_scores_for_testsets(
+ data, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_hrf_vessel_aggregated(imageset="all", scores=scores, eps=1e-4)
+
+ assert results["inconsistency"]["inconsistency_fov_som"]
+
+ results = check_hrf_vessel_aggregated(
+ imageset=[img["identifier"] for img in data], scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"]["inconsistency_fov_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_image(random_state):
+ """
+ Testing a consistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ data = get_experiment("retina.hrf")["fov"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [data[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_hrf_vessel_image(
+ image_identifier=data[0]["identifier"], scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"]["inconsistency_fov"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_image(random_state):
+ """
+ Testing an inconsistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+ data = get_experiment("retina.hrf")["fov"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [data[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_hrf_vessel_image(
+ image_identifier=data[0]["identifier"], scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"]["inconsistency_fov"]
diff --git a/tests/check/bundles/test_retina_stare.py b/tests/check/bundles/test_retina_stare.py
new file mode 100644
index 0000000..869699f
--- /dev/null
+++ b/tests/check/bundles/test_retina_stare.py
@@ -0,0 +1,193 @@
+"""
+This file tests the test bundle for the STARE dataset
+"""
+
+import pytest
+
+from mlscorecheck.check.bundles.retina import (
+ check_stare_vessel_image,
+ check_stare_vessel_aggregated,
+)
+
+from mlscorecheck.experiments import get_experiment
+
+from mlscorecheck.aggregated import generate_scores_for_testsets
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_mos(random_state):
+ """
+ Testing a consistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ dataah = get_experiment("retina.stare")["ah"]["images"]
+
+ scores = generate_scores_for_testsets(
+ dataah, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+ results = check_stare_vessel_aggregated(
+ imageset="all", annotator="ah", scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert not results["inconsistency"]["inconsistency_mos"]
+
+ results = check_stare_vessel_aggregated(
+ imageset=[img["identifier"] for img in dataah],
+ annotator="ah",
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert not results["inconsistency"]["inconsistency_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_mos(random_state):
+ """
+ Testing an inconsistent setup with MoS aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ dataah = get_experiment("retina.stare")["ah"]["images"]
+
+ scores = generate_scores_for_testsets(
+ dataah, aggregation="mos", rounding_decimals=4, random_state=random_state
+ )
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_stare_vessel_aggregated(
+ imageset="all", annotator="ah", scores=scores, eps=1e-4, verbosity=0
+ )
+
+ assert results["inconsistency"]["inconsistency_mos"]
+
+ results = check_stare_vessel_aggregated(
+ imageset=[img["identifier"] for img in dataah],
+ annotator="ah",
+ scores=scores,
+ eps=1e-4,
+ verbosity=0,
+ )
+
+ assert results["inconsistency"]["inconsistency_mos"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_som(random_state):
+ """
+ Testing a consistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ dataah = get_experiment("retina.stare")["ah"]["images"]
+
+ scores = generate_scores_for_testsets(
+ dataah, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_stare_vessel_aggregated(
+ imageset="all", annotator="ah", scores=scores, eps=1e-4
+ )
+
+ assert not results["inconsistency"]["inconsistency_som"]
+
+ results = check_stare_vessel_aggregated(
+ imageset=[img["identifier"] for img in dataah],
+ annotator="ah",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"]["inconsistency_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_som(random_state):
+ """
+ Testing an inconsistent setup with SoM aggregation
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ dataah = get_experiment("retina.stare")["ah"]["images"]
+
+ scores = generate_scores_for_testsets(
+ dataah, aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_stare_vessel_aggregated(
+ imageset="all", annotator="ah", scores=scores, eps=1e-4
+ )
+
+ assert results["inconsistency"]["inconsistency_som"]
+
+ results = check_stare_vessel_aggregated(
+ imageset=[img["identifier"] for img in dataah],
+ annotator="ah",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"]["inconsistency_som"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_success_image(random_state):
+ """
+ Testing a consistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+
+ dataah = get_experiment("retina.stare")["ah"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [dataah[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ results = check_stare_vessel_image(
+ image_identifier=dataah[0]["identifier"],
+ annotator="ah",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("random_state", [1, 2, 3, 4, 5])
+def test_failure_image(random_state):
+ """
+ Testing an inconsistent setup for an image
+
+ Args:
+ random_state (int): the random seed to use
+ """
+ dataah = get_experiment("retina.stare")["ah"]["images"]
+
+ scores = generate_scores_for_testsets(
+ [dataah[0]], aggregation="som", rounding_decimals=4, random_state=random_state
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ results = check_stare_vessel_image(
+ image_identifier=dataah[0]["identifier"],
+ annotator="ah",
+ scores=scores,
+ eps=1e-4,
+ )
+
+ assert results["inconsistency"]
diff --git a/tests/check/bundles/test_tpehg.py b/tests/check/bundles/test_tpehg.py
new file mode 100644
index 0000000..6e8aecf
--- /dev/null
+++ b/tests/check/bundles/test_tpehg.py
@@ -0,0 +1,20 @@
+"""
+Testing the EHG testing
+"""
+
+from mlscorecheck.check.bundles.ehg import check_tpehg
+
+
+def test_check_tpehg():
+ """
+ Testing the check_tpehg function
+ """
+
+ results = check_tpehg(
+ scores={"acc": 0.95, "sens": 0.95, "spec": 0.95},
+ eps=1e-4,
+ n_folds=5,
+ n_repeats=1,
+ )
+
+ assert "inconsistency" in results
diff --git a/tests/check/multiclass/test_1_dataset_known_folds_mos_macro.py b/tests/check/multiclass/test_1_dataset_known_folds_mos_macro.py
new file mode 100644
index 0000000..cbed0a5
--- /dev/null
+++ b/tests/check/multiclass/test_1_dataset_known_folds_mos_macro.py
@@ -0,0 +1,59 @@
+"""
+This module tests the 1 dataset, kfold MoS, macro averaging consistency test
+"""
+
+import pytest
+
+from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_macro
+from mlscorecheck.aggregated import generate_dataset_folding_multiclass
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_consistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="macro",
+ aggregation="mos",
+ max_class_size=100,
+ rounding_decimals=4,
+ # subset=['acc', 'sens', 'spec', 'f1p', 'bacc']
+ )
+
+ result = check_1_dataset_known_folds_mos_macro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_inconsistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="macro",
+ aggregation="mos",
+ rounding_decimals=4,
+ # subset=['acc', 'sens', 'spec', 'f1p', 'bacc']
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ result = check_1_dataset_known_folds_mos_macro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4, timeout=2
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
diff --git a/tests/check/multiclass/test_1_dataset_known_folds_mos_micro.py b/tests/check/multiclass/test_1_dataset_known_folds_mos_micro.py
new file mode 100644
index 0000000..ba1f46a
--- /dev/null
+++ b/tests/check/multiclass/test_1_dataset_known_folds_mos_micro.py
@@ -0,0 +1,58 @@
+"""
+This module tests the 1 dataset, kfold MoS, micro averaging consistency test
+"""
+
+import pytest
+
+from mlscorecheck.check.multiclass import check_1_dataset_known_folds_mos_micro
+from mlscorecheck.aggregated import generate_dataset_folding_multiclass
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_consistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="micro",
+ aggregation="mos",
+ rounding_decimals=4,
+ # subset=['acc', 'sens', 'spec', 'f1p', 'bacc']
+ )
+
+ result = check_1_dataset_known_folds_mos_micro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_inconsistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="micro",
+ aggregation="mos",
+ rounding_decimals=4,
+ # subset=['acc', 'sens', 'spec', 'f1p', 'bacc']
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ result = check_1_dataset_known_folds_mos_micro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4, timeout=2
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
diff --git a/tests/check/multiclass/test_1_dataset_known_folds_som_macro.py b/tests/check/multiclass/test_1_dataset_known_folds_som_macro.py
new file mode 100644
index 0000000..41e5e32
--- /dev/null
+++ b/tests/check/multiclass/test_1_dataset_known_folds_som_macro.py
@@ -0,0 +1,56 @@
+"""
+This module tests the 1 dataset, kfold SoM, macro averaging consistency test
+"""
+
+import pytest
+
+from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_macro
+from mlscorecheck.aggregated import generate_dataset_folding_multiclass
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_consistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="macro",
+ aggregation="som",
+ rounding_decimals=4,
+ )
+
+ result = check_1_dataset_known_folds_som_macro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_inconsistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="macro",
+ aggregation="som",
+ rounding_decimals=4,
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ result = check_1_dataset_known_folds_som_macro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4, timeout=2
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
diff --git a/tests/check/multiclass/test_1_dataset_known_folds_som_micro.py b/tests/check/multiclass/test_1_dataset_known_folds_som_micro.py
new file mode 100644
index 0000000..3814d56
--- /dev/null
+++ b/tests/check/multiclass/test_1_dataset_known_folds_som_micro.py
@@ -0,0 +1,56 @@
+"""
+This module tests the 1 dataset, kfold SoM, micro averaging consistency test
+"""
+
+import pytest
+
+from mlscorecheck.check.multiclass import check_1_dataset_known_folds_som_micro
+from mlscorecheck.aggregated import generate_dataset_folding_multiclass
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_consistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="micro",
+ aggregation="som",
+ rounding_decimals=4,
+ )
+
+ result = check_1_dataset_known_folds_som_micro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4
+ )
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", range(20))
+def test_inconsistent(random_seed: int):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
+
+ dataset, folding, scores = generate_dataset_folding_multiclass(
+ random_state=random_seed,
+ average="micro",
+ aggregation="som",
+ rounding_decimals=4,
+ )
+
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ result = check_1_dataset_known_folds_som_micro(
+ dataset=dataset, folding=folding, scores=scores, eps=1e-4
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
diff --git a/tests/check/multiclass/test_1_testset_no_kfold_macro.py b/tests/check/multiclass/test_1_testset_no_kfold_macro.py
new file mode 100644
index 0000000..3bb4859
--- /dev/null
+++ b/tests/check/multiclass/test_1_testset_no_kfold_macro.py
@@ -0,0 +1,67 @@
+"""
+This module tests the 1 testset no kfold macro averaging consistency test
+"""
+
+import pytest
+
+from mlscorecheck.check.multiclass import check_1_testset_no_kfold_macro
+from mlscorecheck.individual import (
+ generate_multiclass_dataset,
+ sample_multiclass_dataset,
+)
+from mlscorecheck.scores import calculate_multiclass_scores
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+def test_consistent_configuration(random_seed):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ """
+
+ dataset = generate_multiclass_dataset(random_state=random_seed)
+ confm = sample_multiclass_dataset(dataset=dataset, random_state=random_seed)
+
+ scores = calculate_multiclass_scores(
+ confm,
+ average="macro",
+ additional_symbols={"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=4,
+ )
+
+ scores = scores | {"beta_positive": 2, "beta_negative": 2}
+
+ result = check_1_testset_no_kfold_macro(testset=dataset, scores=scores, eps=1e-4)
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+def test_inconsistent_configuration(random_seed):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ """
+
+ dataset = generate_multiclass_dataset(random_state=random_seed)
+ confm = sample_multiclass_dataset(dataset=dataset, random_state=random_seed)
+
+ scores = calculate_multiclass_scores(
+ confm,
+ average="macro",
+ additional_symbols={"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=4,
+ )
+
+ scores = scores | {"beta_positive": 2, "beta_negative": 2}
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ result = check_1_testset_no_kfold_macro(
+ testset=dataset, scores=scores, eps=1e-4, timeout=2
+ )
+
+ assert result["inconsistency"] or result["lp_status"] == "timeout"
diff --git a/tests/check/multiclass/test_1_testset_no_kfold_micro.py b/tests/check/multiclass/test_1_testset_no_kfold_micro.py
new file mode 100644
index 0000000..01df9da
--- /dev/null
+++ b/tests/check/multiclass/test_1_testset_no_kfold_micro.py
@@ -0,0 +1,68 @@
+"""
+This module tests the 1 testset no kfold micro averaging consistency test
+"""
+
+import pytest
+
+from mlscorecheck.check.multiclass import check_1_testset_no_kfold_micro
+from mlscorecheck.individual import (
+ generate_multiclass_dataset,
+ sample_multiclass_dataset,
+)
+from mlscorecheck.scores import calculate_multiclass_scores
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+def test_consistent_configuration(random_seed):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ """
+
+ dataset = generate_multiclass_dataset(random_state=random_seed)
+ confm = sample_multiclass_dataset(dataset=dataset, random_state=random_seed)
+
+ scores = calculate_multiclass_scores(
+ confm,
+ average="micro",
+ additional_symbols={"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=4,
+ )
+
+ scores = scores | {"beta_positive": 2, "beta_negative": 2}
+
+ print(dataset)
+ print(scores)
+
+ result = check_1_testset_no_kfold_micro(testset=dataset, scores=scores, eps=1e-4)
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+def test_inconsistent_configuration(random_seed):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ """
+
+ dataset = generate_multiclass_dataset(random_state=random_seed)
+ confm = sample_multiclass_dataset(dataset=dataset, random_state=random_seed)
+
+ scores = calculate_multiclass_scores(
+ confm,
+ average="micro",
+ additional_symbols={"beta_positive": 2, "beta_negative": 2},
+ rounding_decimals=4,
+ )
+
+ scores = scores | {"beta_positive": 2, "beta_negative": 2}
+ scores["acc"] = (1.0 + scores["spec"]) / 2.0
+
+ result = check_1_testset_no_kfold_micro(testset=dataset, scores=scores, eps=1e-4)
+
+ assert result["inconsistency"]
diff --git a/tests/check/regression/test_check_1_testset_no_kfold.py b/tests/check/regression/test_check_1_testset_no_kfold.py
new file mode 100644
index 0000000..9d2f863
--- /dev/null
+++ b/tests/check/regression/test_check_1_testset_no_kfold.py
@@ -0,0 +1,81 @@
+"""
+Testing the consistency tests for regression scores in the 1 testset no kfold case
+"""
+
+import pytest
+
+from mlscorecheck.core import safe_eval
+from mlscorecheck.check.regression import (
+ generate_regression_problem_and_scores,
+ check_1_testset_no_kfold,
+ score_formulas,
+)
+
+
+@pytest.mark.parametrize("random_seed", list(range(20)))
+@pytest.mark.parametrize(
+ "subset", [None, ["mae", "rmse"], ["mae", "mse"], ["mae", "r2"], ["mae"]]
+)
+def test_consistency(random_seed, subset):
+ """
+ Testing a consistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ subset (list): the score subset to use
+ """
+
+ var, n_samples, scores = generate_regression_problem_and_scores(
+ random_state=random_seed, rounding_decimals=4, subset=subset
+ )
+
+ result = check_1_testset_no_kfold(var, n_samples, scores, eps=1e-4)
+
+ assert not result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(20)))
+@pytest.mark.parametrize(
+ "subset", [None, ["mae", "rmse"], ["mae", "mse"], ["mae", "r2"], ["mae"]]
+)
+def test_inconsistency(random_seed, subset):
+ """
+ Testing an iconsistent configuration
+
+ Args:
+ random_seed (int): the random seed to use
+ subset (list): the score subset to use
+ """
+
+ var, n_samples, scores = generate_regression_problem_and_scores(
+ random_state=random_seed, rounding_decimals=4, subset=subset
+ )
+
+ scores["mae"] = 0.6
+ scores["rmse"] = 0.5
+
+ result = check_1_testset_no_kfold(var, n_samples, scores, eps=1e-4)
+
+ assert result["inconsistency"]
+
+
+@pytest.mark.parametrize("random_seed", list(range(20)))
+def test_score_formulas(random_seed):
+ """
+ Testing the score formulas
+
+ Args:
+ random_seed (int): the random seed to use
+ """
+
+ var, n_samples, scores = generate_regression_problem_and_scores(
+ random_state=random_seed
+ )
+
+ for key, solutions in score_formulas.items():
+ for sol_key, solution in solutions.items():
+ score_tmp = safe_eval(
+ solution, scores | {"var": var, "n_samples": n_samples}
+ )
+ print(key, sol_key, score_tmp, scores[key])
+ assert abs(score_tmp - scores[key]) < 1e-8
diff --git a/tests/check/test_1_dataset_known_folds_mos_scores.py b/tests/check/test_1_dataset_known_folds_mos_scores.py
deleted file mode 100644
index 389af34..0000000
--- a/tests/check/test_1_dataset_known_folds_mos_scores.py
+++ /dev/null
@@ -1,105 +0,0 @@
-"""
-Testing the checking of scores on 1 dataset using kfold with mean of scores
-aggregation
-"""
-
-import pytest
-
-from mlscorecheck.check import check_1_dataset_known_folds_mos_scores
-from mlscorecheck.aggregated import (generate_evaluation)
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- evaluation, scores = generate_evaluation(aggregation='mos',
- random_state=random_seed,
- return_scores=True,
- rounding_decimals=rounding_decimals)
-
- result = check_1_dataset_known_folds_mos_scores(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure(random_seed: int, rounding_decimals: int):
- """
- Testing with an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- evaluation, scores = generate_evaluation(aggregation='mos',
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_1_dataset_known_folds_mos_scores(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency_bounds(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup and bounds
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- evaluation, scores = generate_evaluation(aggregation='mos',
- random_state=random_seed,
- return_scores=True,
- feasible_fold_score_bounds=True,
- rounding_decimals=rounding_decimals)
-
- result = check_1_dataset_known_folds_mos_scores(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- fold_score_bounds=evaluation['fold_score_bounds'],
- scores=scores,
- eps=10**(-rounding_decimals),
- timeout=1)
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure_bounds(random_seed: int, rounding_decimals: int):
- """
- Testing with a inconsistent setup and bounds
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- evaluation, scores = generate_evaluation(aggregation='mos',
- random_state=random_seed,
- return_scores=True,
- feasible_fold_score_bounds=False,
- rounding_decimals=rounding_decimals)
- scores = {'acc': 0.9, 'bacc': 0.1, 'sens': 0.1, 'npv': 0.1, 'ppv': 0.1, 'f1': 0.9}
-
- result = check_1_dataset_known_folds_mos_scores(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- fold_score_bounds=evaluation['fold_score_bounds'],
- scores=scores,
- eps=10**(-rounding_decimals),
- timeout=1)
-
- assert result['inconsistency'] or result['lp_status'] == 'timeout'
diff --git a/tests/check/test_1_dataset_som_scores.py b/tests/check/test_1_dataset_som_scores.py
deleted file mode 100644
index 22da35a..0000000
--- a/tests/check/test_1_dataset_som_scores.py
+++ /dev/null
@@ -1,67 +0,0 @@
-"""
-Testing the checking of scores on 1 dataset using kfold with SoM aggregation
-"""
-
-import pytest
-
-from mlscorecheck.check import check_1_dataset_som_scores
-from mlscorecheck.aggregated import (generate_evaluation)
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- evaluation, scores = generate_evaluation(aggregation='som',
- random_state=random_seed,
- return_scores=True,
- rounding_decimals=rounding_decimals)
-
- result = check_1_dataset_som_scores(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure(random_seed: int, rounding_decimals: int):
- """
- Testing with an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- evaluation, scores = generate_evaluation(aggregation='som',
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_1_dataset_som_scores(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert result['inconsistency']
-
-def test_adding_strategy():
- """
- Testing the addition of strategy
- """
- evaluation = {'dataset': {'p': 5, 'n': 6}, 'folding': {'n_folds': 2, 'n_repeats': 1}}
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_1_dataset_som_scores(dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- scores=scores,
- eps=10**(-4))
-
- assert result['inconsistency']
diff --git a/tests/check/test_1_dataset_unknown_folds_mos_scores.py b/tests/check/test_1_dataset_unknown_folds_mos_scores.py
deleted file mode 100644
index 30e5d23..0000000
--- a/tests/check/test_1_dataset_unknown_folds_mos_scores.py
+++ /dev/null
@@ -1,105 +0,0 @@
-"""
-Testing the test functionality with MoS aggregation and unknown folds
-"""
-
-import pytest
-
-import numpy as np
-
-from mlscorecheck.check import (check_1_dataset_unknown_folds_mos_scores,
- estimate_n_evaluations)
-from mlscorecheck.aggregated import Evaluation
-
-subsets = [['acc', 'sens', 'spec', 'bacc'], ['acc', 'sens'], ['acc', 'spec'], ['acc']]
-
-def test_estimation():
- """
- Testing the evaluation count estimation
- """
-
- count = estimate_n_evaluations(dataset={'p': 5, 'n': 11},
- folding={'n_folds': 3, 'n_repeats': 2},
- available_scores=['acc', 'sens', 'spec'])
-
- assert count == 16
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [4])
-@pytest.mark.parametrize('subset', subsets)
-def test_success(random_seed: int, rounding_decimals: int, subset: list):
- """
- Testing with successful configuration
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- subset (list): the subset of scores to work with
- """
- random_state = np.random.RandomState(random_seed)
-
- dataset = {'n': random_state.randint(1, 150),
- 'p': random_state.randint(1, 15)}
- folding = {'n_folds': random_state.randint(1, 5),
- 'n_repeats': random_state.randint(1, 3),
- 'strategy': 'stratified_sklearn'}
- folding['n_folds'] = min(folding['n_folds'], dataset['p'], dataset['n'])
-
- evaluation = Evaluation(dataset=dataset,
- folding=folding,
- aggregation='mos')
- evaluation.sample_figures(score_subset=subset)
-
- scores = evaluation.calculate_scores(rounding_decimals, score_subset=subset)
-
- evaluation = evaluation.to_dict()
-
- results = check_1_dataset_unknown_folds_mos_scores(
- scores=scores,
- eps=(10**(-rounding_decimals)),
- dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- fold_score_bounds=evaluation.get('fold_score_bounds'))
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [4])
-@pytest.mark.parametrize('subset', subsets)
-def test_failure(random_seed: int, rounding_decimals: int, subset: list):
- """
- Testing with successful configuration
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- subset (list): the subset of scores to work with
- """
-
- random_state = np.random.RandomState(random_seed)
-
- dataset = {'n': random_state.randint(1, 150),
- 'p': random_state.randint(1, 15)}
- folding = {'n_folds': random_state.randint(1, 3),
- 'n_repeats': random_state.randint(1, 3),
- 'strategy': 'stratified_sklearn'}
- folding['n_folds'] = min(folding['n_folds'], dataset['p'], dataset['n'])
-
- evaluation = Evaluation(dataset=dataset,
- folding=folding,
- aggregation='mos')
- evaluation.sample_figures(score_subset=subset)
-
- evaluation = evaluation.to_dict()
- del evaluation['aggregation']
-
- scores = {'sens': 0.7, 'spec': 0.7, 'bacc': 0.6, 'acc': 0.1234}
- scores = {key: value for key, value in scores.items() if key in subset}
-
- results = check_1_dataset_unknown_folds_mos_scores(
- scores=scores,
- eps=(10**(-rounding_decimals)),
- dataset=evaluation['dataset'],
- folding=evaluation['folding'],
- fold_score_bounds=evaluation.get('fold_score_bounds'))
-
- assert results['inconsistency']
diff --git a/tests/check/test_1_testset_no_kfold_scores.py b/tests/check/test_1_testset_no_kfold_scores.py
deleted file mode 100644
index 35dd183..0000000
--- a/tests/check/test_1_testset_no_kfold_scores.py
+++ /dev/null
@@ -1,117 +0,0 @@
-"""
-Testing the use case regarding one dataset
-"""
-
-import warnings
-
-import pytest
-
-from mlscorecheck.check import (check_1_testset_no_kfold_scores)
-from mlscorecheck.individual import calculate_scores, generate_1_problem
-
-k = 4 # pylint: disable=invalid-name
-eps = 10**(-k) # pylint: disable=invalid-name
-
-def test_parametrization():
- """
- Testing the parametrization
- """
- with pytest.raises(ValueError):
- check_1_testset_no_kfold_scores(scores={},
- eps=1e-4,
- testset={'p': 5})
-
-def test_warnings():
- """
- Testing the warning
- """
-
- problem = {'name': 'common_datasets.ADA',
- 'n_repeats': 2}
-
- scores = {'acc': 0.9,
- 'sens': 0.89,
- 'spec': 0.91}
-
- with warnings.catch_warnings(record=True) as warns:
- warnings.simplefilter("always")
- result = check_1_testset_no_kfold_scores(scores=scores,
- eps=1e-2,
- testset=problem)
- assert len(warns) == 1
-
- assert not result['inconsistency']
-
-def test_consistent():
- """
- Testing a consistent configuration
- """
- evaluation, problem = generate_1_problem(random_state=5)
-
- evaluation['beta_negative'] = 2
- evaluation['beta_positive'] = 2
-
- scores = calculate_scores(evaluation,
- rounding_decimals=k)
- scores['beta_negative'] = 2
- scores['beta_positive'] = 2
-
- result = check_1_testset_no_kfold_scores(scores=scores,
- eps=eps,
- testset=problem)
-
- assert not result['inconsistency']
-
-def test_failure():
- """
- Testing a failure
- """
- evaluation, problem = generate_1_problem(random_state=5)
-
- evaluation['beta_negative'] = 2
- evaluation['beta_positive'] = 2
-
- scores = calculate_scores(evaluation,
- rounding_decimals=k)
- scores['bacc'] = 0.9
- scores['acc'] = 0.1
- scores['beta_negative'] = 2
- scores['beta_positive'] = 2
-
- result = check_1_testset_no_kfold_scores(scores=scores,
- eps=eps,
- testset=problem)
-
- assert result['inconsistency']
-
-def test_dataset():
- """
- Testing success with real dataset
- """
- problem = {'name': 'common_datasets.ADA'}
-
- scores = {'acc': 0.9,
- 'sens': 0.89,
- 'spec': 0.91}
-
- result = check_1_testset_no_kfold_scores(scores=scores,
- eps=1e-2,
- testset=problem)
-
- assert not result['inconsistency']
-
-def test_dataset_failure():
- """
- Testing failure with real dataset with extreme precision and random scores
- """
- problem = {'name': 'common_datasets.ADA'}
-
- scores = {'acc': 0.9,
- 'sens': 0.89,
- 'spec': 0.91}
-
- result = check_1_testset_no_kfold_scores(scores=scores,
- eps=0.00001,
- testset=problem)
-
- assert result['inconsistency']
diff --git a/tests/check/test_n_datasets_mos_kfold_som_scores.py b/tests/check/test_n_datasets_mos_kfold_som_scores.py
deleted file mode 100644
index 98eb98e..0000000
--- a/tests/check/test_n_datasets_mos_kfold_som_scores.py
+++ /dev/null
@@ -1,125 +0,0 @@
-"""
-Testing the checking of scores on multiple datasets using kfold with
-score-of-means aggregation over the folds and mean-of-scores aggregation
-over the datasets
-"""
-
-import pytest
-
-from mlscorecheck.check import check_n_datasets_mos_kfold_som_scores
-from mlscorecheck.aggregated import (generate_experiment)
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
-
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'som'},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
-
- result = check_n_datasets_mos_kfold_som_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure(random_seed: int, rounding_decimals: int):
- """
- Testing with an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'som'},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
-
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_n_datasets_mos_kfold_som_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency_bounds(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup and bounds
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'som'},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True,
- feasible_dataset_score_bounds=True)
-
- result = check_n_datasets_mos_kfold_som_scores(
- evaluations=experiment['evaluations'],
- dataset_score_bounds=experiment['dataset_score_bounds'],
- scores=scores,
- eps=10**(-rounding_decimals),
- timeout=2)
-
- assert not result['inconsistency'] or result['lp_status'] == 'timeout'
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure_bounds(random_seed: int, rounding_decimals: int):
- """
- Testing with a inconsistent setup and bounds
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'som'},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True,
- feasible_dataset_score_bounds=False)
-
- result = check_n_datasets_mos_kfold_som_scores(
- evaluations=experiment['evaluations'],
- dataset_score_bounds=experiment['dataset_score_bounds'],
- scores=scores,
- eps=10**(-rounding_decimals),
- timeout=2)
-
- assert result['inconsistency'] or result['lp_status'] == 'timeout'
-
-def test_exception():
- """
- Testing the throwing of an exception
- """
-
- with pytest.raises(ValueError):
- check_n_datasets_mos_kfold_som_scores(evaluations=[{'aggregation': 'mos'}],
- scores={},
- eps=1e-4)
-
- with pytest.raises(ValueError):
- check_n_datasets_mos_kfold_som_scores(evaluations=[{'aggregation': 'som',
- 'fold_score_bounds': {}}],
- scores={},
- eps=1e-4)
diff --git a/tests/check/test_n_datasets_mos_known_folds_mos_scores.py b/tests/check/test_n_datasets_mos_known_folds_mos_scores.py
deleted file mode 100644
index 793e414..0000000
--- a/tests/check/test_n_datasets_mos_known_folds_mos_scores.py
+++ /dev/null
@@ -1,127 +0,0 @@
-"""
-Testing the checking of scores on multiple datasets using kfold with
-mean-of-scores aggregation over the folds and mean-of-scores aggregation
-over the datasets
-"""
-
-import pytest
-
-from mlscorecheck.check import check_n_datasets_mos_known_folds_mos_scores
-from mlscorecheck.aggregated import (generate_experiment)
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
-
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'mos'},
- rounding_decimals=rounding_decimals,
- random_state=random_seed,
- return_scores=True)
-
- result = check_n_datasets_mos_known_folds_mos_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure(random_seed: int, rounding_decimals: int):
- """
- Testing with an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'mos'},
- rounding_decimals=rounding_decimals,
- random_state=random_seed,
- return_scores=True)
-
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_n_datasets_mos_known_folds_mos_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency_bounds(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup and bounds
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'mos'},
- rounding_decimals=rounding_decimals,
- random_state=random_seed,
- feasible_dataset_score_bounds=True,
- return_scores=True)
-
- result = check_n_datasets_mos_known_folds_mos_scores(
- evaluations=experiment['evaluations'],
- dataset_score_bounds=experiment['dataset_score_bounds'],
- scores=scores,
- eps=10**(-rounding_decimals),
- timeout=2)
-
- assert not result['inconsistency'] or result['lp_status'] == 'timeout'
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure_bounds(random_seed: int, rounding_decimals: int):
- """
- Testing with a inconsistent setup and bounds
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'mos'},
- rounding_decimals=rounding_decimals,
- random_state=random_seed,
- feasible_dataset_score_bounds=True,
- return_scores=True)
-
- scores = {'acc': 0.5, 'sens': 0.1, 'spec': 0.2, 'npv': 0.1, 'ppv': 0.1, 'f1': 0.9}
-
- result = check_n_datasets_mos_known_folds_mos_scores(
- evaluations=experiment['evaluations'],
- dataset_score_bounds=experiment['dataset_score_bounds'],
- scores=scores,
- eps=10**(-rounding_decimals),
- timeout=2)
-
- assert result['inconsistency'] or result['lp_status'] == 'timeout'
-
-def test_exception():
- """
- Testing the throwing of an exception
- """
-
- with pytest.raises(ValueError):
- check_n_datasets_mos_known_folds_mos_scores(evaluations=[{'aggregation': 'som'}],
- scores={},
- eps=1e-4)
-
- with pytest.raises(ValueError):
- check_n_datasets_mos_known_folds_mos_scores(evaluations=[{'aggregation': 'mos',
- 'fold_score_bounds': {}}],
- scores={},
- eps=1e-4)
diff --git a/tests/check/test_n_datasets_mos_unknown_folds_mos_scores.py b/tests/check/test_n_datasets_mos_unknown_folds_mos_scores.py
deleted file mode 100644
index a70d93d..0000000
--- a/tests/check/test_n_datasets_mos_unknown_folds_mos_scores.py
+++ /dev/null
@@ -1,154 +0,0 @@
-"""
-Testing the case with multiple datasets and unknown folds in MoS aggregations
-"""
-
-import pytest
-
-import numpy as np
-
-from mlscorecheck.check import (check_n_datasets_mos_unknown_folds_mos_scores,
- estimate_n_experiments)
-from mlscorecheck.aggregated import (generate_experiment)
-
-subsets = [['acc', 'sens', 'spec', 'bacc'], ['acc', 'sens'], ['acc', 'spec'], ['acc']]
-
-def test_estimation():
- """
- Testing the evaluation count estimation
- """
-
- count = estimate_n_experiments(evaluations=[{'dataset': {'p': 5, 'n': 11},
- 'folding': {'n_folds': 3, 'n_repeats': 2}},
- {'dataset': {'p': 6, 'n': 9},
- 'folding': {'n_folds': 3, 'n_repeats': 2}}],
- available_scores=['acc', 'sens', 'spec'])
-
- assert count == 144
-
-def generate_test_case(random_seed: int,
- rounding_decimals: int,
- score_subset: list) -> (dict, dict):
- """
- Generate one random test case
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- score_subset (list): the list of scores to be used
-
- Returns:
- dict, dict: the experiment specification and the scores
- """
- evaluation_params = {'max_p': 25,
- 'max_n': 200,
- 'max_folds': 4,
- 'max_repeats': 1,
- 'aggregation': 'mos',
- 'no_folds': True,
- 'no_name': True}
-
- random_state = np.random.RandomState(random_seed)
- experiment, scores = generate_experiment(random_state=random_state,
- rounding_decimals=rounding_decimals,
- evaluation_params=evaluation_params,
- max_evaluations=2,
- aggregation='mos',
- return_scores=True)
-
- n_experiments = estimate_n_experiments(experiment['evaluations'],
- list(scores.keys()))
-
- while n_experiments > 1000\
- or len(experiment['evaluations']) == 1:
- experiment, scores = generate_experiment(random_state=random_state,
- rounding_decimals=rounding_decimals,
- evaluation_params=evaluation_params,
- max_evaluations=2,
- aggregation='mos',
- return_scores=True)
-
- n_experiments = estimate_n_experiments(experiment['evaluations'],
- list(scores.keys()))
- scores = {key: value for key, value in scores.items() if key in score_subset}
- return experiment, scores
-
-def remove_strategy_fsom_folding(experiment):
- """
- Removes the "strategy" from the folding
-
- Args:
- experiment (dict): an experiment specification
- """
- for evaluation in experiment['evaluations']:
- del evaluation['folding']['strategy']
-
-def test_remove_strategy_fsom_folding():
- """
- Testing the remove_strategy_fsom_folding function
- """
- experiment = {'evaluations': [{'folding': {'strategy': 'dummy0'}},
- {'folding': {'strategy': 'dummy1'}}]}
-
- remove_strategy_fsom_folding(experiment)
-
- assert 'strategy' not in experiment['evaluations'][0]['folding']
- assert 'strategy' not in experiment['evaluations'][1]['folding']
-
-@pytest.mark.parametrize('random_seed', list(range(5)))
-@pytest.mark.parametrize('subset', subsets)
-def test_successful(random_seed: int, subset: list):
- """
- Testing a successful scenario
-
- Args:
- random_seed (int): the random seed to use
- subset (list): the subset of scores to be used
- """
- experiment, scores = generate_test_case(random_seed, 4, score_subset=subset)
-
- remove_strategy_fsom_folding(experiment)
-
- results = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=1e-4)
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(5)))
-@pytest.mark.parametrize('subset', subsets)
-def test_failure(random_seed: int, subset: list):
- """
- Testing a failure
-
- Args:
- random_seed (int): the random seed to use
- subset (list): the subset of scores to be used
- """
-
- experiment, scores = generate_test_case(random_seed, 4, score_subset=subset)
-
- remove_strategy_fsom_folding(experiment)
-
- scores = {'acc': 0.9, 'sens': 0.1, 'spec': 0.1, 'bacc': 0.05}
-
- results = check_n_datasets_mos_unknown_folds_mos_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=1e-4)
-
- assert results['inconsistency']
-
-def test_exception():
- """
- Testing the exception
- """
-
- with pytest.raises(ValueError):
- check_n_datasets_mos_unknown_folds_mos_scores(evaluations=[{'aggregation': 'som'}],
- scores={},
- eps=1e-4)
-
- with pytest.raises(ValueError):
- check_n_datasets_mos_unknown_folds_mos_scores(evaluations=[{'aggregation': 'mos',
- 'fold_score_bounds': {}}],
- scores={},
- eps=1e-4)
diff --git a/tests/check/test_n_datasets_som_kfold_som_scores.py b/tests/check/test_n_datasets_som_kfold_som_scores.py
deleted file mode 100644
index 1688ab8..0000000
--- a/tests/check/test_n_datasets_som_kfold_som_scores.py
+++ /dev/null
@@ -1,67 +0,0 @@
-"""
-Testing the checking of scores on multiple datasets using kfold with
-score-of-means aggregation on both levels
-"""
-
-import pytest
-
-from mlscorecheck.check import check_n_datasets_som_kfold_som_scores
-from mlscorecheck.aggregated import (generate_experiment)
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimal places to round to
- """
-
- experiment, scores = generate_experiment(aggregation='som',
- evaluation_params={'aggregation': 'som'},
- random_state=random_seed,
- return_scores=True,
- rounding_decimals=rounding_decimals)
-
- result = check_n_datasets_som_kfold_som_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure(random_seed: int, rounding_decimals: int):
- """
- Testing with an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimal places to round to
- """
-
- experiment, scores = generate_experiment(aggregation='som',
- evaluation_params={'aggregation': 'som'},
- random_state=random_seed,
- return_scores=True,
- rounding_decimals=rounding_decimals)
-
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_n_datasets_som_kfold_som_scores(evaluations=experiment['evaluations'],
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert result['inconsistency']
-
-def test_exception():
- """
- Testing the throwing of an exception
- """
-
- with pytest.raises(ValueError):
- check_n_datasets_som_kfold_som_scores(evaluations=[{'aggregation': 'mos'}],
- scores={},
- eps=1e-4)
diff --git a/tests/check/test_n_testsets_mos_no_kfold_scores.py b/tests/check/test_n_testsets_mos_no_kfold_scores.py
deleted file mode 100644
index 552d423..0000000
--- a/tests/check/test_n_testsets_mos_no_kfold_scores.py
+++ /dev/null
@@ -1,64 +0,0 @@
-"""
-Testing the checking of scores on multiple datasets with
-mean-of-scores aggregation over the datasets
-"""
-
-import pytest
-
-from mlscorecheck.check import check_n_testsets_mos_no_kfold_scores
-from mlscorecheck.aggregated import (generate_experiment)
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
-
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'mos',
- 'max_folds': 1,
- 'max_repeats': 1,
- 'no_name': True},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
- testsets = [evaluation['dataset'] for evaluation in experiment['evaluations']]
-
- result = check_n_testsets_mos_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure(random_seed: int, rounding_decimals: int):
- """
- Testing with an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='mos',
- evaluation_params={'aggregation': 'mos',
- 'max_folds': 1,
- 'max_repeats': 1,
- 'no_name': True},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
- testsets = [evaluation['dataset'] for evaluation in experiment['evaluations']]
-
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_n_testsets_mos_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert result['inconsistency']
diff --git a/tests/check/test_n_testsets_som_no_kfold_scores.py b/tests/check/test_n_testsets_som_no_kfold_scores.py
deleted file mode 100644
index 7d8aede..0000000
--- a/tests/check/test_n_testsets_som_no_kfold_scores.py
+++ /dev/null
@@ -1,64 +0,0 @@
-"""
-Testing the checking of scores on multiple datasets with
-score-of-means aggregation over the datasets
-"""
-
-import pytest
-
-from mlscorecheck.check import check_n_testsets_som_no_kfold_scores
-from mlscorecheck.aggregated import (generate_experiment)
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_consistency(random_seed: int, rounding_decimals: int):
- """
- Testing with a consistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
-
- experiment, scores = generate_experiment(aggregation='som',
- evaluation_params={'aggregation': 'som',
- 'max_folds': 1,
- 'max_repeats': 1,
- 'no_name': True},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
- testsets = [evaluation['dataset'] for evaluation in experiment['evaluations']]
-
- result = check_n_testsets_som_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert not result['inconsistency']
-
-@pytest.mark.parametrize('random_seed', list(range(10)))
-@pytest.mark.parametrize('rounding_decimals', [3, 4])
-def test_failure(random_seed: int, rounding_decimals: int):
- """
- Testing with an inconsistent setup
-
- Args:
- random_seed (int): the random seed to use
- rounding_decimals (int): the number of decimals to round to
- """
- experiment, scores = generate_experiment(aggregation='som',
- evaluation_params={'aggregation': 'som',
- 'max_folds': 1,
- 'max_repeats': 1,
- 'no_name': True},
- random_state=random_seed,
- rounding_decimals=rounding_decimals,
- return_scores=True)
- testsets = [evaluation['dataset'] for evaluation in experiment['evaluations']]
-
- scores = {'acc': 0.9, 'sens': 0.3, 'spec': 0.5, 'f1': 0.1}
-
- result = check_n_testsets_som_no_kfold_scores(testsets=testsets,
- scores=scores,
- eps=10**(-rounding_decimals))
-
- assert result['inconsistency']
diff --git a/tests/core/test_dict.py b/tests/core/test_dict.py
index 6466df6..7d57922 100644
--- a/tests/core/test_dict.py
+++ b/tests/core/test_dict.py
@@ -4,21 +4,23 @@
from mlscorecheck.core import dict_mean, dict_minmax
+
def test_dict_mean():
"""
Testing the calculation of the mean of dicts
"""
- means = dict_mean([{'a': 1, 'b': 2}, {'a': 3, 'b': 4}])
- assert means['a'] == 2
- assert means['b'] == 3
+ means = dict_mean([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+ assert means["a"] == 2
+ assert means["b"] == 3
+
def test_dict_minmax():
"""
Testing calculating the minmax of dicts
"""
- minmax = dict_minmax([{'a': 1, 'b': 2}, {'a': 3, 'b': 4}, {'a': 0, 'b': 2}])
+ minmax = dict_minmax([{"a": 1, "b": 2}, {"a": 3, "b": 4}, {"a": 0, "b": 2}])
- assert minmax['a'] == [0, 3]
- assert minmax['b'] == [2, 4]
+ assert minmax["a"] == [0, 3]
+ assert minmax["b"] == [2, 4]
diff --git a/tests/core/test_numerical_uncertainty.py b/tests/core/test_numerical_uncertainty.py
index 2934e9c..1aee357 100644
--- a/tests/core/test_numerical_uncertainty.py
+++ b/tests/core/test_numerical_uncertainty.py
@@ -4,8 +4,8 @@
import pytest
-from mlscorecheck.core import (check_uncertainty_and_tolerance,
- update_uncertainty)
+from mlscorecheck.core import check_uncertainty_and_tolerance, update_uncertainty
+
def test_uncertainty_and_tolerance():
"""
@@ -13,10 +13,11 @@ def test_uncertainty_and_tolerance():
"""
with pytest.raises(ValueError):
- check_uncertainty_and_tolerance({'acc': 1e-7, 'sens': 1e-4}, 2*1e-8)
+ check_uncertainty_and_tolerance({"acc": 1e-7, "sens": 1e-4}, 2 * 1e-8)
with pytest.raises(ValueError):
- check_uncertainty_and_tolerance(1e-7, 2*1e-8)
+ check_uncertainty_and_tolerance(1e-7, 2 * 1e-8)
+
def test_update_uncertainty():
"""
@@ -24,4 +25,4 @@ def test_update_uncertainty():
"""
assert abs(update_uncertainty(1e-2, 1e-3) - (1e-2 + 1e-3)) <= 1e-5
- assert abs(update_uncertainty({'acc': 1e-2}, 1e-3)['acc'] - (1e-2 + 1e-3)) <= 1e-5
+ assert abs(update_uncertainty({"acc": 1e-2}, 1e-3)["acc"] - (1e-2 + 1e-3)) <= 1e-5
diff --git a/tests/core/test_random_state.py b/tests/core/test_random_state.py
index 0d0d7e6..e7183ee 100644
--- a/tests/core/test_random_state.py
+++ b/tests/core/test_random_state.py
@@ -6,6 +6,7 @@
from mlscorecheck.core import init_random_state
+
def test_init_random_state():
"""
Testing the init random state functionality
diff --git a/tests/core/test_rounding.py b/tests/core/test_rounding.py
index 1b52c40..f4c6579 100644
--- a/tests/core/test_rounding.py
+++ b/tests/core/test_rounding.py
@@ -6,18 +6,19 @@
from mlscorecheck.core import round_scores
+
def test_round_scores():
"""
Testing the rounding of scores
"""
- scores = {'acc': 0.123456, 'sens': 0.654321}
+ scores = {"acc": 0.123456, "sens": 0.654321}
rounded = round_scores(scores, 3)
- assert float(np.round(scores['acc'], 3)) == rounded['acc']
- assert float(np.round(scores['sens'], 3)) == rounded['sens']
+ assert float(np.round(scores["acc"], 3)) == rounded["acc"]
+ assert float(np.round(scores["sens"], 3)) == rounded["sens"]
- assert round_scores(scores)['acc'] == scores['acc']
+ assert round_scores(scores)["acc"] == scores["acc"]
assert round_scores(0.51, 1) == 0.5
diff --git a/tests/core/test_safe_eval.py b/tests/core/test_safe_eval.py
index 9aa852c..48986e8 100644
--- a/tests/core/test_safe_eval.py
+++ b/tests/core/test_safe_eval.py
@@ -4,12 +4,14 @@
from mlscorecheck.core import safe_eval, safe_call
+
def test_safe_eval():
"""
Testing the safe_eval
"""
- assert safe_eval('a*b', {'a': 2, 'b': 3}) == 6
+ assert safe_eval("a*b", {"a": 2, "b": 3}) == 6
+
def mock_function(*, a):
"""
@@ -21,13 +23,14 @@ def mock_function(*, a):
Returns:
int|float: the double of the parameter
"""
- return a*2
+ return a * 2
+
def test_safe_call():
"""
Testing the safe call
"""
- assert safe_call(mock_function, {'a': 2, 'b': 3}, [{'a': 2, 'b': 3}]) is None
- assert safe_call(mock_function, {'a': 2, 'b': 3}, [{'a': 2, 'b': 4}]) == 4
- assert safe_call(mock_function, {'a': 2, 'b': 3}, [{'a': 'a'}]) is None
- assert safe_call(mock_function, {'a': 2, 'b': 3}) == 4
+ assert safe_call(mock_function, {"a": 2, "b": 3}, [{"a": 2, "b": 3}]) is None
+ assert safe_call(mock_function, {"a": 2, "b": 3}, [{"a": 2, "b": 4}]) == 4
+ assert safe_call(mock_function, {"a": 2, "b": 3}, [{"a": "a"}]) is None
+ assert safe_call(mock_function, {"a": 2, "b": 3}) == 4
diff --git a/tests/experiments/test_load_datasets.py b/tests/experiments/test_load_datasets.py
index 497d65e..36129a4 100644
--- a/tests/experiments/test_load_datasets.py
+++ b/tests/experiments/test_load_datasets.py
@@ -9,23 +9,24 @@
import pytest
-from mlscorecheck.experiments import (lookup_dataset,
- resolve_pn)
+from mlscorecheck.experiments import lookup_dataset, resolve_pn
+
def test_lookup_dataset():
"""
Testing the lookup dataset functionality
"""
- path = os.path.join('experiments', 'machine_learning', 'common_datasets.json')
- sio = files('mlscorecheck').joinpath(path).read_text()
+ path = os.path.join("experiments", "machine_learning", "common_datasets.json")
+ sio = files("mlscorecheck").joinpath(path).read_text()
data = json.loads(sio)
- for entry in data['datasets']:
- dataset = lookup_dataset('common_datasets.' + entry['name'])
+ for entry in data["datasets"]:
+ dataset = lookup_dataset("common_datasets." + entry["name"])
+
+ assert entry["p"] == dataset["p"]
+ assert entry["n"] == dataset["n"]
- assert entry['p'] == dataset['p']
- assert entry['n'] == dataset['n']
def test_exception():
"""
@@ -33,13 +34,23 @@ def test_exception():
"""
with pytest.raises(ValueError):
- lookup_dataset('dummy')
+ lookup_dataset("dummy")
+
def testresolve_pn():
"""
Testing the resolution of p and n figures
"""
- assert 'p' in resolve_pn({'dataset': 'common_datasets.ADA'})
- assert len(resolve_pn([{'dataset': 'common_datasets.ADA'},
- {'dataset': 'common_datasets.ecoli1'}])) == 2
+ assert "p" in resolve_pn({"dataset": "common_datasets.ADA"})
+ assert (
+ len(
+ resolve_pn(
+ [
+ {"dataset": "common_datasets.ADA"},
+ {"dataset": "common_datasets.ecoli1"},
+ ]
+ )
+ )
+ == 2
+ )
diff --git a/tests/experiments/test_load_experiments.py b/tests/experiments/test_load_experiments.py
index b4a8ee9..69221c1 100644
--- a/tests/experiments/test_load_experiments.py
+++ b/tests/experiments/test_load_experiments.py
@@ -4,28 +4,36 @@
import pytest
-from mlscorecheck.experiments import (load_drive,
- load_tpehg,
- load_stare,
- load_isic2016,
- load_isic2017,
- load_chase_db1,
- load_diaretdb0,
- load_diaretdb1,
- load_hrf,
- load_drishti_gs,
- get_experiment)
-
-@pytest.mark.parametrize('key', ['retina.drive',
- 'retina.stare',
- 'retina.chase_db1',
- 'retina.diaretdb0',
- 'retina.diaretdb1',
- 'retina.hrf',
- 'retina.drishti_gs',
- 'ehg.tpehg',
- 'skinlesion.isic2016',
- 'skinlesion.isic2017'])
+from mlscorecheck.experiments import (
+ load_drive,
+ load_tpehg,
+ load_stare,
+ load_isic2016,
+ load_isic2017,
+ load_chase_db1,
+ load_diaretdb0,
+ load_diaretdb1,
+ load_hrf,
+ load_drishti_gs,
+ get_experiment,
+)
+
+
+@pytest.mark.parametrize(
+ "key",
+ [
+ "retina.drive",
+ "retina.stare",
+ "retina.chase_db1",
+ "retina.diaretdb0",
+ "retina.diaretdb1",
+ "retina.hrf",
+ "retina.drishti_gs",
+ "ehg.tpehg",
+ "skinlesion.isic2016",
+ "skinlesion.isic2017",
+ ],
+)
def test_get_experiment(key):
"""
Testing the get experiment function
@@ -35,13 +43,15 @@ def test_get_experiment(key):
assert len(get_experiment(key)) > 0
assert len(get_experiment(key)) > 0
+
def test_get_experiment_exception():
"""
Testing if get_experiment throws the exception
"""
with pytest.raises(ValueError):
- get_experiment('dummy')
+ get_experiment("dummy")
+
def test_load_drishti_gs():
"""
@@ -50,6 +60,7 @@ def test_load_drishti_gs():
assert len(load_drishti_gs()) > 0
+
def test_load_hrf():
"""
Testing the loading of HRF
@@ -57,6 +68,7 @@ def test_load_hrf():
assert len(load_hrf()) > 0
+
def test_load_diaretdb1():
"""
Testing the loading of DIARETDB1
@@ -64,6 +76,7 @@ def test_load_diaretdb1():
assert len(load_diaretdb1()) > 0
+
def test_load_diaretdb0():
"""
Testing the loading of DIARETDB0
@@ -71,6 +84,7 @@ def test_load_diaretdb0():
assert len(load_diaretdb0()) > 0
+
def test_load_chase_db1():
"""
Testing the loading of CHASE db 1
@@ -78,6 +92,7 @@ def test_load_chase_db1():
assert len(load_chase_db1()) > 0
+
def test_load_isic2017():
"""
Testing the loading of ISIC2017
@@ -85,6 +100,7 @@ def test_load_isic2017():
assert len(load_isic2017()) > 0
+
def test_load_isic2016():
"""
Testing the loading of ISIC2016
@@ -92,6 +108,7 @@ def test_load_isic2016():
assert len(load_isic2016()) > 0
+
def test_load_stare():
"""
Testing the loading of STARE
@@ -99,6 +116,7 @@ def test_load_stare():
assert len(load_stare()) > 0
+
def test_load_drive():
"""
Testing the loading of drive
@@ -106,9 +124,10 @@ def test_load_drive():
assert len(load_drive()) == 4
+
def test_load_tpehg():
"""
Testing the loading of the EHG dataset
"""
- assert 'p' in load_tpehg()
+ assert "p" in load_tpehg()
diff --git a/tests/individual/test_calculate_scores.py b/tests/individual/test_calculate_scores.py
deleted file mode 100644
index 1e1b893..0000000
--- a/tests/individual/test_calculate_scores.py
+++ /dev/null
@@ -1,28 +0,0 @@
-"""
-This module tests the score calculation capabilities
-"""
-
-from mlscorecheck.individual import calculate_scores, calculate_scores_for_lp
-from mlscorecheck.scores import score_functions_with_solutions
-
-def test_calculate_scores():
- """
- Testing the score calculation
- """
-
- scores = calculate_scores({'p': 40, 'n': 20, 'tp': 34, 'tn': 13,
- 'beta_positive': 2, 'beta_negative': 2})
- assert len(scores) == len(score_functions_with_solutions)
-
-def test_calculate_scores_for_lp():
- """
- Testing the score calculation for linear programming
- """
- scores = calculate_scores_for_lp({'p': 40, 'n': 20, 'tp': 34, 'tn': 13})
- assert len(scores) == 4
-
- scores = calculate_scores_for_lp({'p': 0, 'n': 20, 'tp': 0, 'tn': 13})
- assert len(scores) == 2
-
- scores = calculate_scores_for_lp({'p': 4, 'n': 0, 'tp': 3, 'tn': 0})
- assert len(scores) == 2
diff --git a/tests/individual/test_check_tptn_solutions.py b/tests/individual/test_check_tptn_solutions.py
index 5352a38..dddd458 100644
--- a/tests/individual/test_check_tptn_solutions.py
+++ b/tests/individual/test_check_tptn_solutions.py
@@ -5,19 +5,36 @@
import pytest
-from mlscorecheck.individual import (check_scores_tptn_pairs,
- _check_scores_tptn_pairs,
- generate_1_problem,
- calculate_scores,
- sqrt)
-from mlscorecheck.scores import (score_functions_standardized_without_complements)
+from mlscorecheck.individual import (
+ check_scores_tptn_pairs,
+ _check_scores_tptn_pairs,
+ generate_1_problem,
+ sqrt,
+)
+from mlscorecheck.scores import (
+ score_functions_standardized_without_complements,
+ calculate_scores,
+)
score_functions = score_functions_standardized_without_complements
-@pytest.mark.parametrize('figure', ['tp', 'tn', None])
-@pytest.mark.parametrize('zeros', [[], ['tp'], ['tn'], ['fp'], ['fn'],
- ['tp', 'tn'], ['tp', 'fp'], ['fp', 'fn'], ['tn', 'fn']])
-@pytest.mark.parametrize('random_seed', list(range(10)))
+
+@pytest.mark.parametrize("figure", ["tp", "tn", None])
+@pytest.mark.parametrize(
+ "zeros",
+ [
+ [],
+ ["tp"],
+ ["tn"],
+ ["fp"],
+ ["fn"],
+ ["tp", "tn"],
+ ["tp", "fp"],
+ ["fp", "fn"],
+ ["tn", "fn"],
+ ],
+)
+@pytest.mark.parametrize("random_seed", list(range(10)))
def test_check_scores_tptn_pairs(figure: str, zeros: list, random_seed: int):
"""
This function tests the check functionality
@@ -28,36 +45,45 @@ def test_check_scores_tptn_pairs(figure: str, zeros: list, random_seed: int):
random_seed (int): the random seed to use
"""
- evaluation, problem = generate_1_problem(random_state=random_seed,
- zeros=zeros)
+ evaluation, problem = generate_1_problem(random_state=random_seed, zeros=zeros)
- evaluation['beta_positive'] = 1
- evaluation['beta_negative'] = 1
- evaluation['sqrt'] = sqrt
+ evaluation["beta_positive"] = 1
+ evaluation["beta_negative"] = 1
+ evaluation["sqrt"] = sqrt
scores = calculate_scores(evaluation)
scores = {key: value for key, value in scores.items() if value is not None}
- scores['beta_positive'] = 1
- scores['beta_negative'] = 1
-
- results = _check_scores_tptn_pairs(p=problem['p'],
- n=problem['n'],
- scores=scores,
- eps=1e-4,
- solve_for=figure)
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('prefilter_by_pairs', [True, False])
-@pytest.mark.parametrize('figure', ['tp', 'tn', None])
-@pytest.mark.parametrize('zeros', [[], ['tp'], ['tn'], ['fp'], ['fn'],
- ['tp', 'tn'], ['tp', 'fp'], ['fp', 'fn'], ['tn', 'fn']])
-@pytest.mark.parametrize('random_seed', list(range(10)))
-def test_check_scores_tptn_pairs_prefilter(figure: str,
- zeros: list,
- random_seed: int,
- prefilter_by_pairs: bool):
+ scores["beta_positive"] = 1
+ scores["beta_negative"] = 1
+
+ results = _check_scores_tptn_pairs(
+ p=problem["p"], n=problem["n"], scores=scores, eps=1e-4, solve_for=figure
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("prefilter_by_pairs", [True, False])
+@pytest.mark.parametrize("figure", ["tp", "tn", None])
+@pytest.mark.parametrize(
+ "zeros",
+ [
+ [],
+ ["tp"],
+ ["tn"],
+ ["fp"],
+ ["fn"],
+ ["tp", "tn"],
+ ["tp", "fp"],
+ ["fp", "fn"],
+ ["tn", "fn"],
+ ],
+)
+@pytest.mark.parametrize("random_seed", list(range(10)))
+def test_check_scores_tptn_pairs_prefilter(
+ figure: str, zeros: list, random_seed: int, prefilter_by_pairs: bool
+):
"""
This function tests the check functionality with prefiltering
@@ -68,37 +94,50 @@ def test_check_scores_tptn_pairs_prefilter(figure: str,
prefilter_by_pairs (bool): whether to prefilter by the pairwise solutions
"""
- evaluation, problem = generate_1_problem(random_state=random_seed,
- zeros=zeros)
+ evaluation, problem = generate_1_problem(random_state=random_seed, zeros=zeros)
- evaluation['beta_positive'] = 1
- evaluation['beta_negative'] = 1
- evaluation['sqrt'] = sqrt
+ evaluation["beta_positive"] = 1
+ evaluation["beta_negative"] = 1
+ evaluation["sqrt"] = sqrt
scores = calculate_scores(evaluation)
scores = {key: value for key, value in scores.items() if value is not None}
- scores['beta_positive'] = 1
- scores['beta_negative'] = 1
-
- results = check_scores_tptn_pairs(p=problem['p'],
- n=problem['n'],
- scores=scores,
- eps=1e-4,
- solve_for=figure,
- prefilter_by_pairs=prefilter_by_pairs)
-
- assert not results['inconsistency']
-
-@pytest.mark.parametrize('prefilter_by_pairs', [True, False])
-@pytest.mark.parametrize('figure', ['tp', 'tn', None])
-@pytest.mark.parametrize('zeros', [[], ['tp'], ['tn'], ['fp'], ['fn'],
- ['tp', 'tn'], ['tp', 'fp'], ['fp', 'fn'], ['tn', 'fn']])
-@pytest.mark.parametrize('random_seed', list(range(10)))
-def test_check_scores_tptn_pairs_prefilter_failure(figure: str,
- zeros: list,
- random_seed: int,
- prefilter_by_pairs: bool):
+ scores["beta_positive"] = 1
+ scores["beta_negative"] = 1
+
+ results = check_scores_tptn_pairs(
+ p=problem["p"],
+ n=problem["n"],
+ scores=scores,
+ eps=1e-4,
+ solve_for=figure,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
+
+ assert not results["inconsistency"]
+
+
+@pytest.mark.parametrize("prefilter_by_pairs", [True, False])
+@pytest.mark.parametrize("figure", ["tp", "tn", None])
+@pytest.mark.parametrize(
+ "zeros",
+ [
+ [],
+ ["tp"],
+ ["tn"],
+ ["fp"],
+ ["fn"],
+ ["tp", "tn"],
+ ["tp", "fp"],
+ ["fp", "fn"],
+ ["tn", "fn"],
+ ],
+)
+@pytest.mark.parametrize("random_seed", list(range(10)))
+def test_check_scores_tptn_pairs_prefilter_failure(
+ figure: str, zeros: list, random_seed: int, prefilter_by_pairs: bool
+):
"""
This function tests the check functionality with failure
@@ -109,30 +148,32 @@ def test_check_scores_tptn_pairs_prefilter_failure(figure: str,
prefilter_by_pairs (bool): whether to prefilter by pair-solutions
"""
- evaluation, problem = generate_1_problem(random_state=random_seed,
- zeros=zeros)
+ evaluation, problem = generate_1_problem(random_state=random_seed, zeros=zeros)
- evaluation['beta_positive'] = 1
- evaluation['beta_negative'] = 1
- evaluation['sqrt'] = sqrt
+ evaluation["beta_positive"] = 1
+ evaluation["beta_negative"] = 1
+ evaluation["sqrt"] = sqrt
scores = calculate_scores(evaluation)
scores = {key: value for key, value in scores.items() if value is not None}
- scores['bacc'] = 0.8
- scores['sens'] = 0.85
- scores['spec'] = 0.86
+ scores["bacc"] = 0.8
+ scores["sens"] = 0.85
+ scores["spec"] = 0.86
+
+ scores["beta_positive"] = 1
+ scores["beta_negative"] = 1
- scores['beta_positive'] = 1
- scores['beta_negative'] = 1
+ results = check_scores_tptn_pairs(
+ p=problem["p"],
+ n=problem["n"],
+ scores=scores,
+ eps=1e-4,
+ solve_for=figure,
+ prefilter_by_pairs=prefilter_by_pairs,
+ )
- results = check_scores_tptn_pairs(p=problem['p'],
- n=problem['n'],
- scores=scores,
- eps=1e-4,
- solve_for=figure,
- prefilter_by_pairs=prefilter_by_pairs)
+ assert results["inconsistency"]
- assert results['inconsistency']
def test_check_parametrization():
"""
@@ -140,4 +181,4 @@ def test_check_parametrization():
"""
with pytest.raises(ValueError):
- _check_scores_tptn_pairs(5, 10, scores={}, eps=1e-4, solve_for='dummy')
+ _check_scores_tptn_pairs(5, 10, scores={}, eps=1e-4, solve_for="dummy")
diff --git a/tests/individual/test_complex_interval.py b/tests/individual/test_complex_interval.py
index 6ac278c..548701f 100644
--- a/tests/individual/test_complex_interval.py
+++ b/tests/individual/test_complex_interval.py
@@ -4,6 +4,7 @@
from mlscorecheck.individual import Interval, IntervalUnion, ComplexInterval
+
def test_arithmetics():
"""
Testing the arithmetics
@@ -33,12 +34,14 @@ def test_arithmetics():
assert (5 - ci0).real == 5 - int0
assert (ci0 - 5).real == int0 - 5
assert (5 * ci0).real == 5 * int0
- assert (5 / ci0).real == (ComplexInterval(Interval(5, 5), Interval(0, 0)) / ci0).real
+ assert (5 / ci0).real == (
+ ComplexInterval(Interval(5, 5), Interval(0, 0)) / ci0
+ ).real
assert (ci0 / 5).real == ci0.real / 5
assert ci0 != int0
assert ci0 + ci1 == ci0 + ci1
- assert not ci0 + ci1 != ci0 + ci1 # pylint: disable = unneeded-not
+ assert not ci0 + ci1 != ci0 + ci1 # pylint: disable = unneeded-not
assert -ci0 == (-1) * ci0
diff --git a/tests/individual/test_expression.py b/tests/individual/test_expression.py
index f0a1adf..0294779 100644
--- a/tests/individual/test_expression.py
+++ b/tests/individual/test_expression.py
@@ -4,11 +4,12 @@
from mlscorecheck.individual import Expression
+
def test_expression():
"""
Testing the expression abstraction
"""
- exp = Expression('a*b', ['a', 'b'])
- assert exp.evaluate({'a': 2, 'b': 3}) == 6
+ exp = Expression("a*b", ["a", "b"])
+ assert exp.evaluate({"a": 2, "b": 3}) == 6
assert len(exp.to_dict()) == 3
diff --git a/tests/individual/test_interval.py b/tests/individual/test_interval.py
index 64cc352..68de0f3 100644
--- a/tests/individual/test_interval.py
+++ b/tests/individual/test_interval.py
@@ -6,6 +6,7 @@
from mlscorecheck.individual import Interval, IntervalUnion
+
def test_interval_constructor():
"""
Testing the interval constructor
@@ -16,6 +17,7 @@ def test_interval_constructor():
assert Interval(2, 1) == Interval(1, 0)
+
def test_interval_equality():
"""
Testing the interval equality
@@ -26,11 +28,13 @@ def test_interval_equality():
assert interval == Interval(-2, 5)
assert interval != Interval(-2, 6)
+
def test_interval_repr():
"""
Testing the representation
"""
- assert str(Interval(0, 1)) == '(0, 1)'
+ assert str(Interval(0, 1)) == "(0, 1)"
+
def test_to_tuple():
"""
@@ -38,6 +42,7 @@ def test_to_tuple():
"""
assert Interval(0, 1).to_tuple() == (0, 1)
+
def test_interval_addition():
"""
Testing the interval addition
@@ -59,6 +64,7 @@ def test_interval_addition():
assert add.lower_bound == int0.lower_bound + 2
+
def test_interval_subtraction():
"""
Testing the interval subtraction
@@ -79,6 +85,7 @@ def test_interval_subtraction():
assert sub.lower_bound == 2 - int0.upper_bound
+
def test_interval_multiplication():
"""
Testing the interval multiplication
@@ -88,10 +95,12 @@ def test_interval_multiplication():
mult = int0 * int1
- terms = [int0.lower_bound * int1.lower_bound,
- int0.lower_bound * int1.upper_bound,
- int0.upper_bound * int1.lower_bound,
- int0.upper_bound * int1.upper_bound]
+ terms = [
+ int0.lower_bound * int1.lower_bound,
+ int0.lower_bound * int1.upper_bound,
+ int0.upper_bound * int1.lower_bound,
+ int0.upper_bound * int1.upper_bound,
+ ]
assert mult.lower_bound == min(terms)
assert mult.upper_bound == max(terms)
@@ -104,6 +113,7 @@ def test_interval_multiplication():
assert mult.lower_bound == 2 * int0.lower_bound
+
def test_interval_division_scalar():
"""
Testing the interval division with scalar
@@ -114,8 +124,8 @@ def test_interval_division_scalar():
assert isinstance(div, IntervalUnion)
assert div.intervals[0].lower_bound == -np.inf
- assert div.intervals[0].upper_bound == 1.0/int0.lower_bound
- assert div.intervals[1].lower_bound == 1.0/int0.upper_bound
+ assert div.intervals[0].upper_bound == 1.0 / int0.lower_bound
+ assert div.intervals[1].lower_bound == 1.0 / int0.upper_bound
assert div.intervals[1].upper_bound == np.inf
div = 1.0 / Interval(1, 2)
@@ -138,6 +148,7 @@ def test_interval_division_scalar():
assert int0.lower_bound == -div.upper_bound and int0.upper_bound == -div.lower_bound
+
def test_interval_division_composite():
"""
Testing the interval division with intervals
@@ -149,10 +160,12 @@ def test_interval_division_composite():
div1 = int0 / int1
- terms = [int0.lower_bound * inverse.lower_bound,
- int0.lower_bound * inverse.upper_bound,
- int0.upper_bound * inverse.lower_bound,
- int0.upper_bound * inverse.upper_bound]
+ terms = [
+ int0.lower_bound * inverse.lower_bound,
+ int0.lower_bound * inverse.upper_bound,
+ int0.upper_bound * inverse.lower_bound,
+ int0.upper_bound * inverse.upper_bound,
+ ]
assert div1.lower_bound == min(terms)
assert div1.upper_bound == max(terms)
@@ -162,10 +175,12 @@ def test_interval_division_composite():
div1 = int0 / int1
- terms = [int0.lower_bound * inverse.lower_bound,
- int0.lower_bound * inverse.upper_bound,
- int0.upper_bound * inverse.lower_bound,
- int0.upper_bound * inverse.upper_bound]
+ terms = [
+ int0.lower_bound * inverse.lower_bound,
+ int0.lower_bound * inverse.upper_bound,
+ int0.upper_bound * inverse.lower_bound,
+ int0.upper_bound * inverse.upper_bound,
+ ]
assert div1.lower_bound == min(terms)
assert div1.upper_bound == max(terms)
@@ -175,10 +190,12 @@ def test_interval_division_composite():
div1 = int0 / int1
- terms = [int0.lower_bound * inverse.lower_bound,
- int0.lower_bound * inverse.upper_bound,
- int0.upper_bound * inverse.lower_bound,
- int0.upper_bound * inverse.upper_bound]
+ terms = [
+ int0.lower_bound * inverse.lower_bound,
+ int0.lower_bound * inverse.upper_bound,
+ int0.upper_bound * inverse.lower_bound,
+ int0.upper_bound * inverse.upper_bound,
+ ]
assert div1.lower_bound == min(terms)
assert div1.upper_bound == max(terms)
@@ -189,22 +206,27 @@ def test_interval_division_composite():
div1 = int0 / int1
- terms = [int0.lower_bound * inverse.intervals[0].lower_bound,
- int0.lower_bound * inverse.intervals[0].upper_bound,
- int0.upper_bound * inverse.intervals[0].lower_bound,
- int0.upper_bound * inverse.intervals[0].upper_bound]
+ terms = [
+ int0.lower_bound * inverse.intervals[0].lower_bound,
+ int0.lower_bound * inverse.intervals[0].upper_bound,
+ int0.upper_bound * inverse.intervals[0].lower_bound,
+ int0.upper_bound * inverse.intervals[0].upper_bound,
+ ]
assert div1.intervals[0].lower_bound == min(terms)
assert div1.intervals[0].upper_bound == max(terms)
- terms = [int0.lower_bound * inverse.intervals[1].lower_bound,
- int0.lower_bound * inverse.intervals[1].upper_bound,
- int0.upper_bound * inverse.intervals[1].lower_bound,
- int0.upper_bound * inverse.intervals[1].upper_bound]
+ terms = [
+ int0.lower_bound * inverse.intervals[1].lower_bound,
+ int0.lower_bound * inverse.intervals[1].upper_bound,
+ int0.upper_bound * inverse.intervals[1].lower_bound,
+ int0.upper_bound * inverse.intervals[1].upper_bound,
+ ]
assert div1.intervals[1].lower_bound == min(terms)
assert div1.intervals[1].upper_bound == max(terms)
+
def test_interval_union_constructor():
"""
Testing the interval union constructor
@@ -214,16 +236,18 @@ def test_interval_union_constructor():
assert IntervalUnion((0, 1)) == IntervalUnion([Interval(0, 1)])
assert IntervalUnion([(0, 1)]) == IntervalUnion([Interval(0, 1)])
+
def test_interval_union_pow():
"""
Testing the power operation on interval unions
"""
intu = IntervalUnion([(-1, 2)])
- intu = intu ** 2
+ intu = intu**2
assert intu == IntervalUnion([(0, 4)])
+
def test_interval_union_to_tuple():
"""
Testing the tuple conversion of interval union
@@ -233,97 +257,108 @@ def test_interval_union_to_tuple():
assert intu.to_tuple() == [(0, 1), (3, 4)]
+
def test_interval_union_simplify():
"""
Testing the interval union simplification
"""
# simple
- intun = IntervalUnion([Interval(1, 2),
- Interval(3, 5),
- Interval(4, 6),
- Interval(5, 8),
- Interval(10, 11)])
+ intun = IntervalUnion(
+ [
+ Interval(1, 2),
+ Interval(3, 5),
+ Interval(4, 6),
+ Interval(5, 8),
+ Interval(10, 11),
+ ]
+ )
assert len(intun.intervals) == 3
assert intun.intervals[1].lower_bound == 3
assert intun.intervals[1].upper_bound == 8
# shuffled
- intun = IntervalUnion([Interval(1, 2),
- Interval(3, 5),
- Interval(10, 11),
- Interval(5, 8),
- Interval(4, 6)])
+ intun = IntervalUnion(
+ [
+ Interval(1, 2),
+ Interval(3, 5),
+ Interval(10, 11),
+ Interval(5, 8),
+ Interval(4, 6),
+ ]
+ )
assert len(intun.intervals) == 3
assert intun.intervals[1].lower_bound == 3
assert intun.intervals[1].upper_bound == 8
# subsets
- intun = IntervalUnion([Interval(1.5, 1.6),
- Interval(1, 2),
- Interval(3, 5),
- Interval(4, 6),
- Interval(5, 8),
- Interval(10, 11),
- Interval(4, 4)])
+ intun = IntervalUnion(
+ [
+ Interval(1.5, 1.6),
+ Interval(1, 2),
+ Interval(3, 5),
+ Interval(4, 6),
+ Interval(5, 8),
+ Interval(10, 11),
+ Interval(4, 4),
+ ]
+ )
assert len(intun.intervals) == 3
assert intun.intervals[1].lower_bound == 3
assert intun.intervals[1].upper_bound == 8
# first union
- intun = IntervalUnion([Interval(3, 5),
- Interval(4, 6),
- Interval(5, 8),
- Interval(10, 11)])
+ intun = IntervalUnion(
+ [Interval(3, 5), Interval(4, 6), Interval(5, 8), Interval(10, 11)]
+ )
assert len(intun.intervals) == 2
assert intun.intervals[0].lower_bound == 3
assert intun.intervals[0].upper_bound == 8
# last union
- intun = IntervalUnion([Interval(1, 2),
- Interval(3, 5),
- Interval(4, 6),
- Interval(5, 8)])
+ intun = IntervalUnion(
+ [Interval(1, 2), Interval(3, 5), Interval(4, 6), Interval(5, 8)]
+ )
assert len(intun.intervals) == 2
assert intun.intervals[1].lower_bound == 3
assert intun.intervals[1].upper_bound == 8
# duplicate
- intun = IntervalUnion([Interval(1, 2),
- Interval(3, 5),
- Interval(1, 2),
- Interval(5, 8),
- Interval(10, 11)])
+ intun = IntervalUnion(
+ [
+ Interval(1, 2),
+ Interval(3, 5),
+ Interval(1, 2),
+ Interval(5, 8),
+ Interval(10, 11),
+ ]
+ )
assert len(intun.intervals) == 3
assert intun.intervals[1].lower_bound == 3
assert intun.intervals[1].upper_bound == 8
# all one
- intun = IntervalUnion([Interval(4, 4),
- Interval(3, 5),
- Interval(4, 6),
- Interval(5, 8),
- Interval(4, 7)])
+ intun = IntervalUnion(
+ [Interval(4, 4), Interval(3, 5), Interval(4, 6), Interval(5, 8), Interval(4, 7)]
+ )
assert len(intun.intervals) == 1
assert intun.intervals[0].lower_bound == 3
assert intun.intervals[0].upper_bound == 8
+
def test_interval_union_add():
"""
Testing addition with interval unions
"""
- intun0 = IntervalUnion([Interval(1, 2),
- Interval(10, 20),
- Interval(100, 200)])
- intun1 = IntervalUnion([Interval(-1, 1),
- Interval(2, 8)])
+ intun0 = IntervalUnion([Interval(1, 2), Interval(10, 20), Interval(100, 200)])
+ intun1 = IntervalUnion([Interval(-1, 1), Interval(2, 8)])
add = intun0 + intun1
@@ -347,15 +382,13 @@ def test_interval_union_add():
assert len(add.intervals) == len(intun0.intervals)
assert add.intervals[0] == intun0.intervals[0] + 2
+
def test_interval_union_subtract():
"""
Testing subtraction with interval unions
"""
- intun0 = IntervalUnion([Interval(1, 2),
- Interval(10, 20),
- Interval(100, 200)])
- intun1 = IntervalUnion([Interval(-1, 1),
- Interval(2, 8)])
+ intun0 = IntervalUnion([Interval(1, 2), Interval(10, 20), Interval(100, 200)])
+ intun1 = IntervalUnion([Interval(-1, 1), Interval(2, 8)])
sub = intun0 - intun1
@@ -379,15 +412,13 @@ def test_interval_union_subtract():
assert len(sub.intervals) == len(intun0.intervals)
assert sub.intervals[2] == (2 - intun0.intervals[0])
+
def test_interval_union_multiply():
"""
Testing addition with interval unions
"""
- intun0 = IntervalUnion([Interval(1, 2),
- Interval(10, 20),
- Interval(100, 200)])
- intun1 = IntervalUnion([Interval(-1, 1),
- Interval(2, 8)])
+ intun0 = IntervalUnion([Interval(1, 2), Interval(10, 20), Interval(100, 200)])
+ intun1 = IntervalUnion([Interval(-1, 1), Interval(2, 8)])
mult = intun0 * intun1
@@ -411,17 +442,15 @@ def test_interval_union_multiply():
assert len(mult.intervals) == len(intun0.intervals)
assert mult.intervals[0] == intun0.intervals[0] * 2
+
def test_interval_union_divide():
"""
Testing addition with interval unions
"""
- intun0 = IntervalUnion([Interval(1, 2),
- Interval(10, 20),
- Interval(100, 200)])
- intun1 = IntervalUnion([Interval(-1, 1),
- Interval(2, 8),
- Interval(0, 1.5),
- Interval(-2.2, 0)])
+ intun0 = IntervalUnion([Interval(1, 2), Interval(10, 20), Interval(100, 200)])
+ intun1 = IntervalUnion(
+ [Interval(-1, 1), Interval(2, 8), Interval(0, 1.5), Interval(-2.2, 0)]
+ )
div = intun0 / intun1
@@ -449,6 +478,7 @@ def test_interval_union_divide():
assert len(div.intervals) == len(intun0.intervals)
assert div.intervals[0] == 2 / intun0.intervals[2]
+
def test_interval_union_equality():
"""
Testing the interval union equality
@@ -461,11 +491,13 @@ def test_interval_union_equality():
assert intun != IntervalUnion([Interval(-2, 6)])
assert intun != IntervalUnion([Interval(0, 1), Interval(2, 3)])
+
def test_interval_union_repr():
"""
Testing the representation
"""
- assert str(IntervalUnion([Interval(0, 1), Interval(2, 3)])) == '(0, 1) | (2, 3)'
+ assert str(IntervalUnion([Interval(0, 1), Interval(2, 3)])) == "(0, 1) | (2, 3)"
+
def test_cross_interval_intervalunion():
"""
@@ -480,7 +512,7 @@ def test_cross_interval_intervalunion():
assert res0 == res1
res0 = int0 - intun
- res1 = (-1)*(intun - int0)
+ res1 = (-1) * (intun - int0)
assert res0 == res1
@@ -494,6 +526,7 @@ def test_cross_interval_intervalunion():
assert (1.0 / res0) == res1
+
def test_intersection():
"""
Testing the interval intersection
@@ -505,6 +538,7 @@ def test_intersection():
assert Interval(0, 10).intersection(Interval(3, 4)) == Interval(3, 4)
assert Interval(3, 4).intersection(Interval(0, 10)) == Interval(3, 4)
+
def test_integer():
"""
Testing the integer condition
@@ -521,6 +555,7 @@ def test_integer():
assert not Interval(0.5, 0.6).integer()
assert not Interval(-0.6, -0.5).integer()
+
def test_contains():
"""
Testing the contain function
@@ -535,32 +570,39 @@ def test_contains():
assert Interval(-np.inf, 5).contains(-np.inf)
assert Interval(5, np.inf).contains(10)
+
def test_union_intersection():
"""
Testing the intersections of unions
"""
- assert IntervalUnion([Interval(0, 1), Interval(1, 2)])\
- .intersection(IntervalUnion([Interval(1, 2), Interval(2, 3)])) == \
- IntervalUnion([Interval(1, 2)])
+ assert IntervalUnion([Interval(0, 1), Interval(1, 2)]).intersection(
+ IntervalUnion([Interval(1, 2), Interval(2, 3)])
+ ) == IntervalUnion([Interval(1, 2)])
- assert IntervalUnion([Interval(0, 1), Interval(2, 3)])\
- .intersection(IntervalUnion([Interval(1, 2), Interval(2, 3)])) == \
- IntervalUnion([Interval(1, 1), Interval(2, 2), Interval(2, 3)])
+ assert IntervalUnion([Interval(0, 1), Interval(2, 3)]).intersection(
+ IntervalUnion([Interval(1, 2), Interval(2, 3)])
+ ) == IntervalUnion([Interval(1, 1), Interval(2, 2), Interval(2, 3)])
- assert IntervalUnion([Interval(0, 1), Interval(2, 3)])\
- .intersection(IntervalUnion([Interval(1.5, 1.8), Interval(3.2, 4)])) == \
- IntervalUnion([])
+ assert IntervalUnion([Interval(0, 1), Interval(2, 3)]).intersection(
+ IntervalUnion([Interval(1.5, 1.8), Interval(3.2, 4)])
+ ) == IntervalUnion([])
- assert IntervalUnion([Interval(0, 1), Interval(2, 3)])\
- .intersection(IntervalUnion([Interval(0.5, 2.5), Interval(2.8, 3.2)])) == \
- IntervalUnion([Interval(0, 1).intersection(Interval(0.5, 2.5)),
- Interval(0, 1).intersection(Interval(2.8, 3.2)),
- Interval(2, 3).intersection(Interval(0.5, 2.5)),
- Interval(2, 3).intersection(Interval(2.8, 3.2))])
+ assert IntervalUnion([Interval(0, 1), Interval(2, 3)]).intersection(
+ IntervalUnion([Interval(0.5, 2.5), Interval(2.8, 3.2)])
+ ) == IntervalUnion(
+ [
+ Interval(0, 1).intersection(Interval(0.5, 2.5)),
+ Interval(0, 1).intersection(Interval(2.8, 3.2)),
+ Interval(2, 3).intersection(Interval(0.5, 2.5)),
+ Interval(2, 3).intersection(Interval(2.8, 3.2)),
+ ]
+ )
+
+ assert IntervalUnion([Interval(0, 1), Interval(1, 2)]).intersection(
+ Interval(0.5, 1.5)
+ ) == Interval(0.5, 1.5)
- assert IntervalUnion([Interval(0, 1), Interval(1, 2)])\
- .intersection(Interval(0.5, 1.5)) == Interval(0.5, 1.5)
def test_union_integer():
"""
@@ -579,6 +621,7 @@ def test_union_integer():
assert not IntervalUnion([Interval(0.5, 0.6)]).integer()
assert not IntervalUnion([Interval(-0.6, -0.5), Interval(0.5, 0.6)]).integer()
+
def test_union_contains():
"""
Testing the contain function for interval union
@@ -598,6 +641,7 @@ def test_union_contains():
assert not IntervalUnion([Interval(0.5, 0.6)]).contains(0)
assert not IntervalUnion([Interval(-0.6, -0.5), Interval(0.5, 0.6)]).contains(0)
+
def test_interval_shrink_to_integer():
"""
Testing the shrinking of intervals to integer boundaries
@@ -607,13 +651,16 @@ def test_interval_shrink_to_integer():
assert Interval(0.2, 1.2).shrink_to_integers() == Interval(1, 1)
assert Interval(0.2, 0.3).shrink_to_integers() == Interval(1, 0)
+
def test_interval_union_shrink_to_integer():
"""
Testing shrinking the interval union to integer boundaries
"""
- assert IntervalUnion([Interval(0.2, 4.5), Interval(6.3, 6.8)])\
- .shrink_to_integers() == IntervalUnion([Interval(1, 4), Interval(7, 6)])
+ assert IntervalUnion(
+ [Interval(0.2, 4.5), Interval(6.3, 6.8)]
+ ).shrink_to_integers() == IntervalUnion([Interval(1, 4), Interval(7, 6)])
+
def test_is_empty():
"""
@@ -625,6 +672,7 @@ def test_is_empty():
assert IntervalUnion([Interval(1, 0)]).is_empty()
assert IntervalUnion([]).is_empty()
+
def test_negation():
"""
Testing the negation operators
@@ -633,6 +681,7 @@ def test_negation():
assert -Interval(0, 1) == Interval(-1, 0)
assert -IntervalUnion([Interval(0, 1)]) == IntervalUnion([Interval(-1, 0)])
+
def test_integer_counts():
"""
Testing the integer counts
@@ -643,11 +692,25 @@ def test_integer_counts():
assert IntervalUnion([Interval(0, 1), Interval(1, 0)]).integer_counts() == 2
assert IntervalUnion([]).integer_counts() == 0
+
def test_odd_power():
"""
Testing the odd powers
"""
- assert Interval(0, 2)**3 == Interval(0, 8)
- assert Interval(-1, 2)**3 == Interval(-1, 8)
- assert Interval(-2, -1)**3 == Interval(-8, -1)
+ assert Interval(0, 2) ** 3 == Interval(0, 8)
+ assert Interval(-1, 2) ** 3 == Interval(-1, 8)
+ assert Interval(-2, -1) ** 3 == Interval(-8, -1)
+
+
+def test_representing_integer():
+ """
+ Testing the extraction of a representing integer
+ """
+
+ assert Interval(0, 1).representing_int() == 0
+ assert Interval(1, 0).representing_int() is None
+
+ assert IntervalUnion([Interval(0, 1)]).representing_int() == 0
+ assert IntervalUnion([Interval(1, 0)]).representing_int() is None
+ assert IntervalUnion([]).representing_int() is None
diff --git a/tests/individual/test_pair_solutions.py b/tests/individual/test_pair_solutions.py
index 3f30172..7979749 100644
--- a/tests/individual/test_pair_solutions.py
+++ b/tests/individual/test_pair_solutions.py
@@ -5,23 +5,28 @@
import pytest
from mlscorecheck.core import safe_call
-from mlscorecheck.individual import (load_solutions,
- Solution,
- Interval,
- IntervalUnion,
- sqrt)
-
-from mlscorecheck.scores import (score_functions_without_complements,
- score_functions_standardized_without_complements,
- score_specifications)
-
-from mlscorecheck.individual import (generate_1_problem)
+from mlscorecheck.individual import (
+ load_solutions,
+ Solution,
+ Interval,
+ IntervalUnion,
+ sqrt,
+)
+
+from mlscorecheck.scores import (
+ score_functions_without_complements,
+ score_functions_standardized_without_complements,
+ score_specifications,
+)
+
+from mlscorecheck.individual import generate_1_problem
solutions = load_solutions()
scores = score_specifications
functions = score_functions_without_complements
functions_standardized = score_functions_standardized_without_complements
+
def check_tptn(tp, tn, result, eps=1e-5):
"""
Checks the tp and tn values against the results
@@ -40,16 +45,17 @@ def check_tptn(tp, tn, result, eps=1e-5):
zero_division = False
for res in result:
- if res.get('message') == 'zero division':
+ if res.get("message") == "zero division":
zero_division = True
continue
- if res.get('message') is not None:
+ if res.get("message") is not None:
continue
- flags.append(abs(tp - res['tp']) < eps and abs(tn - res['tn']) < eps)
+ flags.append(abs(tp - res["tp"]) < eps and abs(tn - res["tn"]) < eps)
return len(flags) == 0 or zero_division, not any(flags)
+
def adjust_evaluation(evaluation):
"""
Making the evaluation inconsistent
@@ -57,14 +63,27 @@ def adjust_evaluation(evaluation):
Args:
evaluation (dict): the evaluation to be rendered inconsistent
"""
- if evaluation['tn'] != 0:
- evaluation['tn'] = evaluation['tn'] * 10 + 5
- elif evaluation['tp'] != 0:
- evaluation['tp'] = evaluation['tp'] * 10 + 5
+ if evaluation["tn"] != 0:
+ evaluation["tn"] = evaluation["tn"] * 10 + 5
+ elif evaluation["tp"] != 0:
+ evaluation["tp"] = evaluation["tp"] * 10 + 5
+
@pytest.mark.parametrize("sol", list(solutions.keys()))
-@pytest.mark.parametrize("zeros", [[], ['tp'], ['tn'], ['fp'], ['fn'], ['tp', 'fp'],
- ['tn', 'fn'], ['tn', 'tp'], ['fn', 'fp']])
+@pytest.mark.parametrize(
+ "zeros",
+ [
+ [],
+ ["tp"],
+ ["tn"],
+ ["fp"],
+ ["fn"],
+ ["tp", "fp"],
+ ["tn", "fn"],
+ ["tn", "tp"],
+ ["fn", "fp"],
+ ],
+)
@pytest.mark.parametrize("random_state", [3, 5, 7])
def test_solution(sol: tuple, zeros: list, random_state: int):
"""
@@ -76,29 +95,29 @@ def test_solution(sol: tuple, zeros: list, random_state: int):
random_state (int): the seed of the random state
"""
- evaluation, _ = generate_1_problem(zeros=zeros,
- add_complements=True,
- random_state=random_state)
- evaluation['beta_positive'] = 2
- evaluation['beta_negative'] = 2
- evaluation['sqrt'] = sqrt
+ evaluation, _ = generate_1_problem(
+ zeros=zeros, add_complements=True, random_state=random_state
+ )
+ evaluation["beta_positive"] = 2
+ evaluation["beta_negative"] = 2
+ evaluation["sqrt"] = sqrt
- score0 = safe_call(functions[sol[0]], evaluation, scores[sol[0]].get('nans'))
- score1 = safe_call(functions[sol[1]], evaluation, scores[sol[1]].get('nans'))
+ score0 = safe_call(functions[sol[0]], evaluation, scores[sol[0]].get("nans"))
+ score1 = safe_call(functions[sol[1]], evaluation, scores[sol[1]].get("nans"))
if score0 is None or score1 is None:
return
- result = solutions[sol].evaluate({**evaluation,
- **{sol[0]: score0,
- sol[1]: score1}})
+ result = solutions[sol].evaluate({**evaluation, **{sol[0]: score0, sol[1]: score1}})
- solvable, failed = check_tptn(evaluation['tp'], evaluation['tn'], result)
+ solvable, failed = check_tptn(evaluation["tp"], evaluation["tn"], result)
assert solvable or not failed
+
@pytest.mark.parametrize("sol", list(solutions.keys()))
-@pytest.mark.parametrize("zeros", [[], ['tp'], ['tn'], ['fp'], ['fn'], ['tp', 'fp'],
- ['tn', 'fn']])
+@pytest.mark.parametrize(
+ "zeros", [[], ["tp"], ["tn"], ["fp"], ["fn"], ["tp", "fp"], ["tn", "fn"]]
+)
@pytest.mark.parametrize("random_state", [3, 5, 7])
def test_solution_failure(sol: tuple, zeros: list, random_state: int):
"""
@@ -110,40 +129,42 @@ def test_solution_failure(sol: tuple, zeros: list, random_state: int):
random_state (int): the seed of the random state
"""
- evaluation, _ = generate_1_problem(zeros=zeros,
- add_complements=True,
- random_state=random_state)
- evaluation['beta_positive'] = 2
- evaluation['beta_negative'] = 2
- evaluation['sqrt'] = sqrt
+ evaluation, _ = generate_1_problem(
+ zeros=zeros, add_complements=True, random_state=random_state
+ )
+ evaluation["beta_positive"] = 2
+ evaluation["beta_negative"] = 2
+ evaluation["sqrt"] = sqrt
- score0 = safe_call(functions[sol[0]], evaluation, scores[sol[0]].get('nans'))
- score1 = safe_call(functions[sol[1]], evaluation, scores[sol[1]].get('nans'))
+ score0 = safe_call(functions[sol[0]], evaluation, scores[sol[0]].get("nans"))
+ score1 = safe_call(functions[sol[1]], evaluation, scores[sol[1]].get("nans"))
if score0 is None or score1 is None:
return
adjust_evaluation(evaluation)
- result = solutions[sol].evaluate({**evaluation,
- **{sol[0]: score0,
- sol[1]: score1}})
+ result = solutions[sol].evaluate({**evaluation, **{sol[0]: score0, sol[1]: score1}})
- solvable, failed = check_tptn(evaluation['tp'], evaluation['tn'], result)
+ solvable, failed = check_tptn(evaluation["tp"], evaluation["tn"], result)
assert solvable or failed
+
def test_solution_object():
"""
Testing the solution abstraction
"""
- sol = Solution(solution={'tp': {'expression': 'p*sens', 'symbols': ['p', 'sens']}},
- conditions=[{'expression': 'p', 'symbols': ['p'],
- 'mode': 'non-negative', 'depth': 1}])
+ sol = Solution(
+ solution={"tp": {"expression": "p*sens", "symbols": ["p", "sens"]}},
+ conditions=[
+ {"expression": "p", "symbols": ["p"], "mode": "non-negative", "depth": 1}
+ ],
+ )
assert len(sol.to_dict()) == 2
- res = sol.evaluate({'p': 3, 'sens': 1})
+ res = sol.evaluate({"p": 3, "sens": 1})
assert len(res) > 0
@@ -151,26 +172,31 @@ def test_solution_object():
assert not sol.check_non_zero(1)
+
def test_solution_non_negatives():
"""
Testing the non-negative base detection in the solution
"""
- sol = Solution({'tp': {'expression': 'p', 'symbols': ['p']}},
- conditions=[{'expression': 'p', 'symbols': ['p'],
- 'mode': 'non-negative', 'depth': 1}])
+ sol = Solution(
+ {"tp": {"expression": "p", "symbols": ["p"]}},
+ conditions=[
+ {"expression": "p", "symbols": ["p"], "mode": "non-negative", "depth": 1}
+ ],
+ )
+
+ res = sol.evaluate({"p": Interval(-2, -1)})
- res = sol.evaluate({'p': Interval(-2, -1)})
+ assert res["message"] == "negative base"
- assert res['message'] == 'negative base'
+ res = sol.evaluate({"p": IntervalUnion([Interval(-2, -1)])})
- res = sol.evaluate({'p': IntervalUnion([Interval(-2, -1)])})
+ assert res["message"] == "negative base"
- assert res['message'] == 'negative base'
+ res = sol.evaluate({"p": -1})
- res = sol.evaluate({'p': -1})
+ assert res["message"] == "negative base"
- assert res['message'] == 'negative base'
def test_solutions_to_dict():
"""
diff --git a/tests/individual/test_problem_generator.py b/tests/individual/test_problem_generator.py
index 9a04c13..2148175 100644
--- a/tests/individual/test_problem_generator.py
+++ b/tests/individual/test_problem_generator.py
@@ -2,42 +2,54 @@
This module tests the problem generator
"""
+import pytest
+
import numpy as np
-from mlscorecheck.individual import (generate_problems,
- calculate_scores,
- generate_problem_and_scores)
+from mlscorecheck.individual import (
+ generate_problems,
+ generate_problem_and_scores,
+ generate_multiclass_dataset,
+ sample_multiclass_dataset,
+)
from mlscorecheck.core import round_scores
+from mlscorecheck.scores import calculate_scores
+
def test_generate_problem_and_scores():
"""
Testing the joint problem and score sampling
"""
- _, scores = generate_problem_and_scores(random_state=5,
- rounding_decimals=3)
+ _, scores = generate_problem_and_scores(random_state=5, rounding_decimals=3)
assert len(scores) > 0
- _, scores = generate_problem_and_scores(random_state=5,
- rounding_decimals=3,
- score_subset=['acc'])
+ _, scores = generate_problem_and_scores(
+ random_state=5, rounding_decimals=3, score_subset=["acc"]
+ )
assert len(scores) == 1
+
def test_round_scores():
"""
Testing the rounding of scores
"""
- assert round_scores({'dummy': 0.12345}) == {'dummy': 0.12345}
- assert round_scores({'dummy': 0.12345}, rounding_decimals=2)['dummy'] == np.round(0.12345, 2)
+ assert round_scores({"dummy": 0.12345}) == {"dummy": 0.12345}
+ assert round_scores({"dummy": 0.12345}, rounding_decimals=2)["dummy"] == np.round(
+ 0.12345, 2
+ )
+
def test_calculate_all_scores():
"""
Testing the calculation of all scores
"""
- scores = calculate_scores({'p': 10, 'tp': 5, 'n': 20, 'tn': 15,
- 'beta_positive': 2, 'beta_negative': 2})
- assert scores['acc'] == 20/30
+ scores = calculate_scores(
+ {"p": 10, "tp": 5, "n": 20, "tn": 15, "beta_positive": 2, "beta_negative": 2}
+ )
+ assert scores["acc"] == 20 / 30
+
def test_generate_problems():
"""
@@ -50,16 +62,46 @@ def test_generate_problems():
assert len(evaluation) == 5
evaluation, _ = generate_problems(add_complements=True)
- assert 'fn' in evaluation
+ assert "fn" in evaluation
+
+ evaluation, _ = generate_problems(zeros=["tp"])
+ assert evaluation["tp"] == 0
+
+ evaluation, _ = generate_problems(zeros=["tn"])
+ assert evaluation["tn"] == 0
+
+ evaluation, _ = generate_problems(zeros=["fp"])
+ assert evaluation["tn"] == evaluation["n"]
+
+ evaluation, _ = generate_problems(zeros=["fn"])
+ assert evaluation["tp"] == evaluation["p"]
+
+
+def test_generate_multiclass_dataset():
+ """
+ Testing the generation of a multiclass dataset
+ """
+
+ dataset = generate_multiclass_dataset(random_state=5, max_n_classes=5)
+
+ assert len(dataset) <= 5
+ assert all(item > 0 for item in dataset.values())
+
+
+@pytest.mark.parametrize("random_seed", list(range(10)))
+def test_sample_multiclass_dataset(random_seed):
+ """
+ Testing the sampling of a multiclass dataset
+
+ Args:
+ random_seed (int): the random seed to be used
+ """
- evaluation, _ = generate_problems(zeros=['tp'])
- assert evaluation['tp'] == 0
+ dataset = generate_multiclass_dataset(random_state=random_seed)
- evaluation, _ = generate_problems(zeros=['tn'])
- assert evaluation['tn'] == 0
+ sample = sample_multiclass_dataset(dataset=dataset, random_state=random_seed)
- evaluation, _ = generate_problems(zeros=['fp'])
- assert evaluation['tn'] == evaluation['n']
+ assert len(dataset) == len(sample)
+ sums = np.sum(sample, axis=1)
- evaluation, _ = generate_problems(zeros=['fn'])
- assert evaluation['tp'] == evaluation['p']
+ assert np.all(sums == np.array(list(dataset.values())))
diff --git a/tests/individual/test_tptn_solutions.py b/tests/individual/test_tptn_solutions.py
index 9e28e32..a2aaf77 100644
--- a/tests/individual/test_tptn_solutions.py
+++ b/tests/individual/test_tptn_solutions.py
@@ -7,18 +7,32 @@
import numpy as np
from mlscorecheck.core import safe_call
-from mlscorecheck.individual import (tptn_solutions, is_applicable_tptn,
- generate_1_problem, calculate_scores)
+from mlscorecheck.individual import (
+ tptn_solutions,
+ is_applicable_tptn,
+ generate_1_problem,
+)
+from mlscorecheck.scores import calculate_scores
-@pytest.mark.parametrize('figure', ['tp', 'tn'])
+
+@pytest.mark.parametrize("figure", ["tp", "tn"])
@pytest.mark.parametrize("score", list(tptn_solutions.keys()))
-@pytest.mark.parametrize("zeros", [[], ['tp'], ['tn'], ['fp'], ['fn'], ['tp', 'fp'],
- ['tn', 'fn'], ['tn', 'tp'], ['fn', 'fp']])
+@pytest.mark.parametrize(
+ "zeros",
+ [
+ [],
+ ["tp"],
+ ["tn"],
+ ["fp"],
+ ["fn"],
+ ["tp", "fp"],
+ ["tn", "fn"],
+ ["tn", "tp"],
+ ["fn", "fp"],
+ ],
+)
@pytest.mark.parametrize("random_state", [3, 5, 7, 11])
-def test_tptn_solutions(figure: str,
- score: str,
- zeros: list,
- random_state: int):
+def test_tptn_solutions(figure: str, score: str, zeros: list, random_state: int):
"""
Testing the tp-tn solutions
@@ -28,15 +42,17 @@ def test_tptn_solutions(figure: str,
zeros (list(str)): the list of figures to set to zero
random_state (int): the seed of the random state
"""
- evaluation, _ = generate_1_problem(zeros=zeros,
- add_complements=True,
- random_state=random_state)
- evaluation['beta_positive'] = 2
- evaluation['beta_negative'] = 2
+ evaluation, _ = generate_1_problem(
+ zeros=zeros, add_complements=True, random_state=random_state
+ )
+ evaluation["beta_positive"] = 2
+ evaluation["beta_negative"] = 2
scores = calculate_scores(evaluation)
- if not (scores[score] is not None and is_applicable_tptn(score, scores[score], figure)):
+ if not (
+ scores[score] is not None and is_applicable_tptn(score, scores[score], figure)
+ ):
return
sol = tptn_solutions[score][figure]
@@ -45,21 +61,26 @@ def test_tptn_solutions(figure: str,
val = safe_call(sol, scores | evaluation)
- if val is None or (isinstance(val, list)
- and (len(val) == 0 or all(value is None for value in val))):
+ if val is None or (
+ isinstance(val, list) and (len(val) == 0 or all(value is None for value in val))
+ ):
return
val = np.array(val)
assert np.any(np.abs(evaluation[figure] - val) < 1e-8)
-special_values = {'ppv': [1, 0],
- 'npv': [1, 0],
- 'fbp': [5, 0],
- 'f1p': [2, 0],
- 'fbn': [0, 5],
- 'f1n': [2, 0]}
-@pytest.mark.parametrize('figure', ['tp', 'tn'])
+special_values = {
+ "ppv": [1, 0],
+ "npv": [1, 0],
+ "fbp": [5, 0],
+ "f1p": [2, 0],
+ "fbn": [0, 5],
+ "f1n": [2, 0],
+}
+
+
+@pytest.mark.parametrize("figure", ["tp", "tn"])
@pytest.mark.parametrize("score", list(tptn_solutions.keys()))
@pytest.mark.parametrize("random_state", [3, 5, 7])
def test_tptn_solutions_failure(figure: str, score: str, random_state: int):
@@ -71,11 +92,11 @@ def test_tptn_solutions_failure(figure: str, score: str, random_state: int):
score (str): the name of the score to test the solution for
random_state (int): the seed of the random state
"""
- evaluation, _ = generate_1_problem(zeros=[],
- add_complements=True,
- random_state=random_state)
- evaluation['beta_positive'] = 2
- evaluation['beta_negative'] = 2
+ evaluation, _ = generate_1_problem(
+ zeros=[], add_complements=True, random_state=random_state
+ )
+ evaluation["beta_positive"] = 2
+ evaluation["beta_negative"] = 2
scores = calculate_scores(evaluation)
@@ -89,8 +110,10 @@ def test_tptn_solutions_failure(figure: str, score: str, random_state: int):
val = safe_call(sol, scores | evaluation)
- if val is None or (isinstance(val, list)
- and (len(val) == 0 or all(value is None for value in val))):
+ if val is None or (
+ isinstance(val, list)
+ and (len(val) == 0 or all(value is None for value in val))
+ ):
assert True
return
@@ -98,14 +121,15 @@ def test_tptn_solutions_failure(figure: str, score: str, random_state: int):
assert np.all(np.abs(evaluation[figure] - val) > 1e-8)
scores[score] = 0
- evaluation['n'] = 0
- evaluation['p'] = 0
- evaluation['tn'] = 0
- evaluation['tp'] = 0
+ evaluation["n"] = 0
+ evaluation["p"] = 0
+ evaluation["tn"] = 0
+ evaluation["tp"] = 0
val = safe_call(sol, scores | evaluation)
- if val is None or (isinstance(val, list)
- and (len(val) == 0 or all(value is None for value in val))):
+ if val is None or (
+ isinstance(val, list) and (len(val) == 0 or all(value is None for value in val))
+ ):
assert True
return
diff --git a/tests/individual/test_utils.py b/tests/individual/test_utils.py
index 23d48ab..e0c05d1 100644
--- a/tests/individual/test_utils.py
+++ b/tests/individual/test_utils.py
@@ -3,16 +3,17 @@
"""
from mlscorecheck.core import NUMERICAL_TOLERANCE
-from mlscorecheck.individual import (create_intervals,
- create_problems_2,
- load_solutions,
- Interval,
- resolve_aliases_and_complements,
- is_less_than_zero,
- unify_results,
- IntervalUnion)
-from mlscorecheck.scores import (score_functions_with_solutions,
- score_specifications)
+from mlscorecheck.individual import (
+ create_intervals,
+ create_problems_2,
+ load_solutions,
+ Interval,
+ resolve_aliases_and_complements,
+ is_less_than_zero,
+ unify_results,
+ IntervalUnion,
+)
+from mlscorecheck.scores import score_functions_with_solutions, score_specifications
scores = score_specifications
functions = score_functions_with_solutions
@@ -20,42 +21,49 @@
random_seeds = [5]
+
def test_resolve_aliases_and_complements():
"""
Resolve the score aliases and complemnets
"""
- assert resolve_aliases_and_complements({'err': 0.4, 'f1': 0.8, 'sens': 0.3})\
- == {'acc': 0.6, 'f1p': 0.8, 'sens': 0.3}
+ assert resolve_aliases_and_complements({"err": 0.4, "f1": 0.8, "sens": 0.3}) == {
+ "acc": 0.6,
+ "f1p": 0.8,
+ "sens": 0.3,
+ }
+
def test_create_intervals():
"""
Testing the create intervals function
"""
- scores_ = resolve_aliases_and_complements({'acc': 0.6, 'tpr': 0.4, 'fnr': 0.6})
+ scores_ = resolve_aliases_and_complements({"acc": 0.6, "tpr": 0.4, "fnr": 0.6})
intervals = create_intervals(scores_, eps=1e-4)
- assert abs(intervals['acc'].lower_bound - 0.5999) <= 1.1 * NUMERICAL_TOLERANCE
- assert abs(intervals['acc'].upper_bound - 0.6001) <= 1.1 * NUMERICAL_TOLERANCE
- assert abs(intervals['sens'].lower_bound - 0.3999) <= 1.1 * NUMERICAL_TOLERANCE
- assert abs(intervals['sens'].upper_bound - 0.4001) <= 1.1 * NUMERICAL_TOLERANCE
+ assert abs(intervals["acc"].lower_bound - 0.5999) <= 1.1 * NUMERICAL_TOLERANCE
+ assert abs(intervals["acc"].upper_bound - 0.6001) <= 1.1 * NUMERICAL_TOLERANCE
+ assert abs(intervals["sens"].lower_bound - 0.3999) <= 1.1 * NUMERICAL_TOLERANCE
+ assert abs(intervals["sens"].upper_bound - 0.4001) <= 1.1 * NUMERICAL_TOLERANCE
- scores_ = resolve_aliases_and_complements({'fnr': 0.6})
+ scores_ = resolve_aliases_and_complements({"fnr": 0.6})
intervals = create_intervals(scores_, eps=1e-4)
- assert abs(intervals['sens'].lower_bound - 0.3999) <= 1.1 * NUMERICAL_TOLERANCE
- assert abs(intervals['sens'].upper_bound - 0.4001) <= 1.1 * NUMERICAL_TOLERANCE
+ assert abs(intervals["sens"].lower_bound - 0.3999) <= 1.1 * NUMERICAL_TOLERANCE
+ assert abs(intervals["sens"].upper_bound - 0.4001) <= 1.1 * NUMERICAL_TOLERANCE
+
+ intervals = create_intervals({"kappa": 0.5}, eps=1e-4)
+ assert abs(intervals["kappa"].lower_bound - 0.4999) <= 1.1 * NUMERICAL_TOLERANCE
+ assert abs(intervals["kappa"].upper_bound - 0.5001) <= 1.1 * NUMERICAL_TOLERANCE
- intervals = create_intervals({'kappa': 0.5}, eps=1e-4)
- assert abs(intervals['kappa'].lower_bound - 0.4999) <= 1.1 * NUMERICAL_TOLERANCE
- assert abs(intervals['kappa'].upper_bound - 0.5001) <= 1.1 * NUMERICAL_TOLERANCE
def test_create_problems_2():
"""
Testing the create problems function
"""
- problems = create_problems_2(['acc', 'sens', 'spec', 'ppv'])
+ problems = create_problems_2(["acc", "sens", "spec", "ppv"])
+
+ assert len(problems) == 6 * 2
- assert len(problems) == 6*2
def test_is_less_than_zero():
"""
@@ -65,6 +73,7 @@ def test_is_less_than_zero():
assert is_less_than_zero(Interval(-4, -3))
assert is_less_than_zero(IntervalUnion([Interval(-4, -3)]))
+
def test_unify_results():
"""
Testing the unification of results
@@ -74,5 +83,9 @@ def test_unify_results():
assert unify_results([None, None]) is None
assert unify_results([2, None]) == [2]
assert unify_results([Interval(2, 3)]) == IntervalUnion([Interval(2, 3)])
- assert unify_results([IntervalUnion([Interval(2, 3)])]) == IntervalUnion([Interval(2, 3)])
- assert unify_results([Interval(2, 3), 5]) == IntervalUnion([Interval(2, 3), Interval(5, 5)])
+ assert unify_results([IntervalUnion([Interval(2, 3)])]) == IntervalUnion(
+ [Interval(2, 3)]
+ )
+ assert unify_results([Interval(2, 3), 5]) == IntervalUnion(
+ [Interval(2, 3), Interval(5, 5)]
+ )
diff --git a/tests/scores/test_calculate_scores.py b/tests/scores/test_calculate_scores.py
new file mode 100644
index 0000000..b391676
--- /dev/null
+++ b/tests/scores/test_calculate_scores.py
@@ -0,0 +1,56 @@
+"""
+This module tests the score calculation capabilities
+"""
+
+import pytest
+
+import numpy as np
+
+from mlscorecheck.scores import (
+ calculate_scores,
+ calculate_scores_for_lp,
+ calculate_multiclass_scores,
+)
+from mlscorecheck.scores import score_functions_with_solutions
+
+
+def test_calculate_scores():
+ """
+ Testing the score calculation
+ """
+
+ scores = calculate_scores(
+ {"p": 40, "n": 20, "tp": 34, "tn": 13, "beta_positive": 2, "beta_negative": 2}
+ )
+ assert len(scores) == len(score_functions_with_solutions)
+
+
+def test_calculate_scores_for_lp():
+ """
+ Testing the score calculation for linear programming
+ """
+ scores = calculate_scores_for_lp({"p": 40, "n": 20, "tp": 34, "tn": 13})
+ assert len(scores) == 4
+
+ scores = calculate_scores_for_lp({"p": 0, "n": 20, "tp": 0, "tn": 13})
+ assert len(scores) == 2
+
+ scores = calculate_scores_for_lp({"p": 4, "n": 0, "tp": 3, "tn": 0})
+ assert len(scores) == 2
+
+
+@pytest.mark.parametrize("average", ["micro", "macro", "weighted"])
+def test_calculate_multiclass_scores(average):
+ """
+ Testing the calculation of multiclass scores
+
+ Args:
+ average (str): the mode of averaging
+ """
+ confm = np.array([[5, 8, 3], [3, 10, 2], [2, 4, 11]])
+ scores = calculate_multiclass_scores(
+ confusion_matrix=confm,
+ additional_symbols={"beta_positive": 2, "beta_negative": 2},
+ average=average,
+ )
+ assert len(scores) > 0
diff --git a/tests/scores/test_multiclass_scores.py b/tests/scores/test_multiclass_scores.py
new file mode 100644
index 0000000..d79e46f
--- /dev/null
+++ b/tests/scores/test_multiclass_scores.py
@@ -0,0 +1,98 @@
+"""
+This module tests the multiclass scores
+"""
+
+import pytest
+
+import numpy as np
+
+from sklearn.metrics import precision_recall_fscore_support
+
+from mlscorecheck.core import safe_call
+from mlscorecheck.scores import multiclass_score_map, multiclass_score
+from mlscorecheck.individual import (
+ generate_multiclass_dataset,
+ sample_multiclass_dataset,
+ create_confusion_matrix,
+)
+
+
+@pytest.mark.parametrize("score", list(multiclass_score_map.keys()))
+@pytest.mark.parametrize("average", ["micro", "macro", "weighted"])
+@pytest.mark.parametrize("random_seed", list(range(5)))
+def test_multiclass_scores(score, average, random_seed):
+ """
+ Testing a multiclass score
+
+ Args:
+ score (str): the score to test
+ average (str): the averaging to be used
+ random_seed (int): the random seed to be used
+ """
+ random_state = np.random.RandomState(random_seed)
+ dataset = generate_multiclass_dataset(random_state=random_seed)
+ matrix = sample_multiclass_dataset(dataset=dataset, random_state=random_seed)
+
+ params = {
+ "confusion_matrix": matrix,
+ "beta_positive": 2,
+ "beta_negative": 2,
+ "average": average,
+ }
+
+ score_matrix = safe_call(multiclass_score_map[score], params)
+
+ assert isinstance(score_matrix, float)
+
+ permutation_mask = list(range(len(matrix)))
+ random_state.shuffle(permutation_mask)
+
+ shuffled_matrix = matrix[permutation_mask][:, permutation_mask]
+
+ params = {
+ "confusion_matrix": shuffled_matrix,
+ "beta_positive": 2,
+ "beta_negative": 2,
+ "average": average,
+ }
+
+ score_shuffled = safe_call(multiclass_score_map[score], params)
+
+ assert abs(score_matrix - score_shuffled) < 1e-8
+
+
+@pytest.mark.parametrize("average", ["micro", "macro", "weighted"])
+@pytest.mark.parametrize("random_seed", list(range(5)))
+def test_compare_to_sklearn(average, random_seed):
+ """
+ Comparing sensitivity, positive predictive value and f1 to sklearn
+
+ Args:
+ average (str): the averaging to be used
+ random_seed (int): the random seed to be used
+ """
+
+ random_state = np.random.RandomState(random_seed)
+ y_true = random_state.randint(0, 5, size=100)
+ y_pred = random_state.randint(0, 5, size=100)
+
+ sample = create_confusion_matrix(y_true, y_pred)
+
+ ppv, sens, f1p, _ = precision_recall_fscore_support(y_true, y_pred, average=average)
+
+ sens_score = multiclass_score_map["sens"](confusion_matrix=sample, average=average)
+ ppv_score = multiclass_score_map["ppv"](confusion_matrix=sample, average=average)
+ f1_score = multiclass_score_map["f1p"](confusion_matrix=sample, average=average)
+
+ assert abs(sens - sens_score) < 1e-8
+ assert abs(ppv - ppv_score) < 1e-8
+ assert abs(f1p - f1_score) < 1e-8
+
+
+def test_exception():
+ """
+ Testing the exception throwing
+ """
+
+ with pytest.raises(ValueError):
+ multiclass_score(None, None, "dummy")
diff --git a/tests/scores/test_scores.py b/tests/scores/test_scores.py
index efe0ea0..dd2096c 100644
--- a/tests/scores/test_scores.py
+++ b/tests/scores/test_scores.py
@@ -8,30 +8,38 @@
from mlscorecheck.core import safe_call, safe_eval
from mlscorecheck.individual import generate_1_problem
-from mlscorecheck.scores import (score_functions_with_complements,
- score_functions_standardized_with_complements,
- score_specifications,
- score_function_aliases)
+from mlscorecheck.scores import (
+ score_functions_with_complements,
+ score_functions_standardized_with_complements,
+ score_specifications,
+ score_function_aliases,
+)
functions = score_functions_with_complements
functions_standardized = score_functions_standardized_with_complements
scores = score_specifications
aliases = score_function_aliases
-short_formula_scores = {key: score for key, score in scores.items() if 'args_short' in score}
-complement_scores = {key: score for key, score in scores.items() if 'complement' in score}
+short_formula_scores = {
+ key: score for key, score in scores.items() if "args_short" in score
+}
+complement_scores = {
+ key: score for key, score in scores.items() if "complement" in score
+}
-evaluation, problem = generate_1_problem(random_state=5,
- add_complements=True)
-evaluation['beta_positive'] = 2
-evaluation['beta_negative'] = 2
+evaluation, problem = generate_1_problem(random_state=5, add_complements=True)
+evaluation["beta_positive"] = 2
+evaluation["beta_negative"] = 2
for key in scores:
- evaluation[key] = functions[key](**{arg: evaluation[arg] for arg in scores[key]['args']})
+ evaluation[key] = functions[key](
+ **{arg: evaluation[arg] for arg in scores[key]["args"]}
+ )
for key, val in aliases.items():
evaluation[key] = evaluation[val]
+
@pytest.mark.parametrize("score", list(scores.keys()))
def test_score_and_standardized(score: str):
"""
@@ -46,6 +54,7 @@ def test_score_and_standardized(score: str):
assert abs(value - value_standard) < 1e-8
+
@pytest.mark.parametrize("score", list(scores.keys()))
def test_score_multiplication_invariance(score: str):
"""
@@ -55,17 +64,18 @@ def test_score_multiplication_invariance(score: str):
score (str): the score to test
"""
evaluation_multiplied = {**evaluation}
- evaluation_multiplied['tp'] = evaluation['tp'] * 5
- evaluation_multiplied['tn'] = evaluation['tn'] * 5
- evaluation_multiplied['fp'] = evaluation['fp'] * 5
- evaluation_multiplied['fn'] = evaluation['fn'] * 5
- evaluation_multiplied['p'] = evaluation['p'] * 5
- evaluation_multiplied['n'] = evaluation['n'] * 5
+ evaluation_multiplied["tp"] = evaluation["tp"] * 5
+ evaluation_multiplied["tn"] = evaluation["tn"] * 5
+ evaluation_multiplied["fp"] = evaluation["fp"] * 5
+ evaluation_multiplied["fn"] = evaluation["fn"] * 5
+ evaluation_multiplied["p"] = evaluation["p"] * 5
+ evaluation_multiplied["n"] = evaluation["n"] * 5
value = safe_call(functions[score], evaluation)
value_mult = safe_call(functions[score], evaluation_multiplied)
assert abs(value - value_mult) < 1e-8
+
@pytest.mark.parametrize("score", list(short_formula_scores.keys()))
def test_short_formulas(score: str):
"""
@@ -76,12 +86,13 @@ def test_short_formulas(score: str):
"""
value = safe_call(functions[score], evaluation)
- short_formula = short_formula_scores[score]['formula_short']
+ short_formula = short_formula_scores[score]["formula_short"]
- value_short = safe_eval(short_formula, evaluation | {'sqrt': math.sqrt})
+ value_short = safe_eval(short_formula, evaluation | {"sqrt": math.sqrt})
assert abs(value - value_short) < 1e-8
+
@pytest.mark.parametrize("score", list(complement_scores.keys()))
def test_short_complements(score: str):
"""
@@ -93,7 +104,7 @@ def test_short_complements(score: str):
value = safe_call(functions[score], evaluation)
- comp_score = scores[score]['complement']
+ comp_score = scores[score]["complement"]
comp_value = safe_call(functions[comp_score], evaluation)
assert abs(1 - (value + comp_value)) < 1e-8
diff --git a/tests/symbolic/test_algebra.py b/tests/symbolic/test_algebra.py
index ec4b743..c408321 100644
--- a/tests/symbolic/test_algebra.py
+++ b/tests/symbolic/test_algebra.py
@@ -7,7 +7,8 @@
from mlscorecheck.symbolic import symbolic_toolkits
from mlscorecheck.symbolic import SympyAlgebra, SageAlgebra, Symbols
-@pytest.mark.skipif('sympy' not in symbolic_toolkits, reason='sympy not installed')
+
+@pytest.mark.skipif("sympy" not in symbolic_toolkits, reason="sympy not installed")
def test_sympy_algebra():
"""
This module tests the algebra base class
@@ -15,14 +16,14 @@ def test_sympy_algebra():
alg = SympyAlgebra()
- assert alg.create_symbol('tp') is not None
+ assert alg.create_symbol("tp") is not None
- assert alg.create_symbol('tp', lower_bound=0, upper_bound=1) is not None
+ assert alg.create_symbol("tp", lower_bound=0, upper_bound=1) is not None
- tp = alg.create_symbol('tp')
- p = alg.create_symbol('p')
+ tp = alg.create_symbol("tp")
+ p = alg.create_symbol("p")
- assert alg.num_denom(tp/p) == (tp, p)
+ assert alg.num_denom(tp / p) == (tp, p)
assert alg.simplify(p) == p
@@ -33,27 +34,30 @@ def test_sympy_algebra():
assert alg.args(p + tp) == {p, tp}
assert alg.is_trivial(None)
- assert alg.is_trivial(alg.simplify(p/p))
+ assert alg.is_trivial(alg.simplify(p / p))
assert not alg.is_trivial(p)
assert alg.is_root(tp**0.5)
assert not alg.is_root(tp)
- assert sorted(map(str, list(alg.operands(tp + p**2)))) == sorted(map(str, [tp, p**2]))
+ assert sorted(map(str, list(alg.operands(tp + p**2)))) == sorted(
+ map(str, [tp, p**2])
+ )
- assert sorted(map(str, alg.free_symbols(tp + p))) == sorted(map(str, ['p', 'tp']))
+ assert sorted(map(str, alg.free_symbols(tp + p))) == sorted(map(str, ["p", "tp"]))
assert alg.is_power(tp**2)
- assert alg.is_division(tp/p)
+ assert alg.is_division(tp / p)
- assert alg.is_division(p/tp)
+ assert alg.is_division(p / tp)
- assert alg.is_division(1/p)
+ assert alg.is_division(1 / p)
assert not alg.is_division(p)
-@pytest.mark.skipif('sage' not in symbolic_toolkits, reason='sage not installed')
+
+@pytest.mark.skipif("sage" not in symbolic_toolkits, reason="sage not installed")
def test_sage_algebra():
"""
This module tests the algebra base class
@@ -61,20 +65,20 @@ def test_sage_algebra():
alg = SageAlgebra()
- assert alg.create_symbol('tp') is not None
+ assert alg.create_symbol("tp") is not None
- assert alg.create_symbol('x', nonnegative=True) is not None
- assert alg.create_symbol('y', positive=True) is not None
- assert alg.create_symbol('z', negative=True) is not None
- assert alg.create_symbol('a', nonpositive=True) is not None
- assert alg.create_symbol('b', real=True) is not None
+ assert alg.create_symbol("x", nonnegative=True) is not None
+ assert alg.create_symbol("y", positive=True) is not None
+ assert alg.create_symbol("z", negative=True) is not None
+ assert alg.create_symbol("a", nonpositive=True) is not None
+ assert alg.create_symbol("b", real=True) is not None
- assert alg.create_symbol('tp', lower_bound=0, upper_bound=1) is not None
+ assert alg.create_symbol("tp", lower_bound=0, upper_bound=1) is not None
- tp = alg.create_symbol('tp')
- p = alg.create_symbol('p')
+ tp = alg.create_symbol("tp")
+ p = alg.create_symbol("p")
- assert alg.num_denom(tp/p) == (tp, p)
+ assert alg.num_denom(tp / p) == (tp, p)
assert alg.simplify(p) == p
@@ -85,44 +89,48 @@ def test_sage_algebra():
assert alg.args(p + tp) == {p, tp}
assert alg.is_trivial(None)
- assert alg.is_trivial(alg.simplify(p/p))
+ assert alg.is_trivial(alg.simplify(p / p))
assert not alg.is_trivial(p)
assert alg.is_root(tp**0.5)
assert not alg.is_root(tp)
- assert sorted(map(str, list(alg.operands(tp + p**2)))) == sorted(map(str, [tp, p**2]))
+ assert sorted(map(str, list(alg.operands(tp + p**2)))) == sorted(
+ map(str, [tp, p**2])
+ )
- assert sorted(map(str, alg.free_symbols(tp + p))) == sorted(map(str, ['p', 'tp']))
+ assert sorted(map(str, alg.free_symbols(tp + p))) == sorted(map(str, ["p", "tp"]))
assert alg.is_power(tp**2)
- assert alg.is_division(tp/p)
+ assert alg.is_division(tp / p)
- assert alg.is_division(p/tp)
+ assert alg.is_division(p / tp)
- assert alg.is_division(1/p)
+ assert alg.is_division(1 / p)
assert not alg.is_division(p)
-@pytest.mark.skipif('sympy' not in symbolic_toolkits, reason='sympy not installed')
+
+@pytest.mark.skipif("sympy" not in symbolic_toolkits, reason="sympy not installed")
def test_symbols_sympy():
"""
Testing the base symbols with sympy
"""
- symbols = Symbols(algebraic_system='sympy')
+ symbols = Symbols(algebraic_system="sympy")
assert len(symbols.to_dict()) == 7
assert symbols.get_algebra() is not None
-@pytest.mark.skipif('sage' not in symbolic_toolkits, reason='sage not installed')
+
+@pytest.mark.skipif("sage" not in symbolic_toolkits, reason="sage not installed")
def test_symbols_sage():
"""
Testing the base symbols with sympy
"""
- symbols = Symbols(algebraic_system='sage')
+ symbols = Symbols(algebraic_system="sage")
assert len(symbols.to_dict()) == 7
assert symbols.get_algebra() is not None
diff --git a/tests/symbolic/test_availability.py b/tests/symbolic/test_availability.py
index 0d932cc..462cf15 100644
--- a/tests/symbolic/test_availability.py
+++ b/tests/symbolic/test_availability.py
@@ -4,17 +4,19 @@
from mlscorecheck.symbolic import check_importability, get_symbolic_toolkit
+
def test_test_importability():
"""
Testing the importability function
"""
- assert check_importability('itertools') == 'itertools'
- assert check_importability('asdfasdfasdfasdfasdf') is None
+ assert check_importability("itertools") == "itertools"
+ assert check_importability("asdfasdfasdfasdfasdf") is None
+
def test_get_symbolic_toolkit():
"""
Testing the get_symbolic_toolkit function
"""
- assert get_symbolic_toolkit() in [None, 'sympy', 'sage']
+ assert get_symbolic_toolkit() in [None, "sympy", "sage"]
diff --git a/tests/symbolic/test_score_objects.py b/tests/symbolic/test_score_objects.py
index cefdb70..37539f1 100644
--- a/tests/symbolic/test_score_objects.py
+++ b/tests/symbolic/test_score_objects.py
@@ -4,45 +4,51 @@
import pytest
-from mlscorecheck.symbolic import (get_base_objects,
- get_symbolic_toolkit,
- get_all_objects,
- get_objects_without_complements,
- Score)
+from mlscorecheck.symbolic import (
+ get_base_objects,
+ get_symbolic_toolkit,
+ get_all_objects,
+ get_objects_without_complements,
+ Score,
+)
symbolic_toolkit = get_symbolic_toolkit()
-@pytest.mark.skipif(symbolic_toolkit is None, reason='no symbolic toolkit available')
+
+@pytest.mark.skipif(symbolic_toolkit is None, reason="no symbolic toolkit available")
def test_base_objects():
"""
Testing the instantiation of the base objects
"""
assert len(get_base_objects(symbolic_toolkit)) > 0
-@pytest.mark.skipif(symbolic_toolkit is None, reason='no symbolic toolkit available')
+
+@pytest.mark.skipif(symbolic_toolkit is None, reason="no symbolic toolkit available")
def test_all_objects():
"""
Testing the instantiation of the base objects
"""
assert len(get_all_objects(symbolic_toolkit)) > 0
-@pytest.mark.skipif(symbolic_toolkit is None, reason='no symbolic toolkit available')
+
+@pytest.mark.skipif(symbolic_toolkit is None, reason="no symbolic toolkit available")
def test_no_complement_objects():
"""
Testing the instantiation of the base objects
"""
assert len(get_objects_without_complements(symbolic_toolkit)) > 0
-@pytest.mark.skipif(symbolic_toolkit is None, reason='no symbolic toolkit available')
+
+@pytest.mark.skipif(symbolic_toolkit is None, reason="no symbolic toolkit available")
def test_serialization():
"""
Testing the serialization
"""
base_objects = get_base_objects(symbolic_toolkit)
- serialized = base_objects['acc'].to_dict()
+ serialized = base_objects["acc"].to_dict()
- score = Score(symbols=base_objects['acc'].symbols, **serialized)
+ score = Score(symbols=base_objects["acc"].symbols, **serialized)
assert score is not None
- assert score.get_algebra() == base_objects['acc'].symbols.algebra
+ assert score.get_algebra() == base_objects["acc"].symbols.algebra
diff --git a/tests/symbolic/test_solver.py b/tests/symbolic/test_solver.py
index 9dee33c..a54d6ad 100644
--- a/tests/symbolic/test_solver.py
+++ b/tests/symbolic/test_solver.py
@@ -6,64 +6,69 @@
from mlscorecheck.symbolic import ProblemSolver, Symbols
from mlscorecheck.symbolic import get_symbolic_toolkit, get_all_objects
-from mlscorecheck.symbolic import _collect_denominators_and_bases, collect_denominators_and_bases
+from mlscorecheck.symbolic import (
+ _collect_denominators_and_bases,
+ collect_denominators_and_bases,
+)
from mlscorecheck.symbolic import check_recurrent_solution
symbolic_toolkit = get_symbolic_toolkit()
-@pytest.mark.skipif(symbolic_toolkit is None, reason='no symbolic toolkit available')
+
+@pytest.mark.skipif(symbolic_toolkit is None, reason="no symbolic toolkit available")
def test_problem_solver():
"""
Testing the problem solver
"""
base_scores = get_all_objects(symbolic_toolkit)
- prob_sol = ProblemSolver(base_scores['sens'], base_scores['spec'])
+ prob_sol = ProblemSolver(base_scores["sens"], base_scores["spec"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['sens'], base_scores['acc'])
+ prob_sol = ProblemSolver(base_scores["sens"], base_scores["acc"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['acc'], base_scores['sens'])
+ prob_sol = ProblemSolver(base_scores["acc"], base_scores["sens"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['acc'], base_scores['spec'])
+ prob_sol = ProblemSolver(base_scores["acc"], base_scores["spec"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['spec'], base_scores['acc'])
+ prob_sol = ProblemSolver(base_scores["spec"], base_scores["acc"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['acc'], base_scores['f1p'])
+ prob_sol = ProblemSolver(base_scores["acc"], base_scores["f1p"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['fm'], base_scores['ppv'])
+ prob_sol = ProblemSolver(base_scores["fm"], base_scores["ppv"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['ppv'], base_scores['fm'])
+ prob_sol = ProblemSolver(base_scores["ppv"], base_scores["fm"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
- prob_sol = ProblemSolver(base_scores['ppv'], base_scores['gm'])
+ prob_sol = ProblemSolver(base_scores["ppv"], base_scores["gm"])
prob_sol.solve()
prob_sol.edge_cases()
assert prob_sol.get_solution() is not None
-@pytest.mark.skipif(symbolic_toolkit is None, reason='no symbolic toolkit available')
+
+@pytest.mark.skipif(symbolic_toolkit is None, reason="no symbolic toolkit available")
def test_root_in_edge_cases():
"""
Testing the detection of a root
@@ -73,25 +78,22 @@ def test_root_in_edge_cases():
denoms = []
bases = []
- _collect_denominators_and_bases(symbols.tp**0.5,
- denoms,
- bases,
- symbols.algebra,
- depth=1)
+ _collect_denominators_and_bases(
+ symbols.tp**0.5, denoms, bases, symbols.algebra, depth=1
+ )
assert bases
denoms = []
bases = []
- _collect_denominators_and_bases((symbols.tp + symbols.tn)**0.5,
- denoms,
- bases,
- symbols.algebra,
- depth=1)
+ _collect_denominators_and_bases(
+ (symbols.tp + symbols.tn) ** 0.5, denoms, bases, symbols.algebra, depth=1
+ )
assert bases
-@pytest.mark.skipif(symbolic_toolkit is None, reason='no symbolic toolkit available')
+
+@pytest.mark.skipif(symbolic_toolkit is None, reason="no symbolic toolkit available")
def test_depths():
"""
Testing the proper use of depths
@@ -102,7 +104,7 @@ def test_depths():
tn = symbols.tn
sin = symbols.algebra.algebra.sin
- expr = sin((8 + tp / (1 + tn))**2) / sin(tp/(1 + tn))
+ expr = sin((8 + tp / (1 + tn)) ** 2) / sin(tp / (1 + tn))
denoms, bases = collect_denominators_and_bases(expr, symbols.algebra)
assert denoms
@@ -112,8 +114,9 @@ def test_depths():
denoms, bases = collect_denominators_and_bases(expr, symbols.algebra)
assert bases
+
def test_recurrent_solution():
"""
Testing the recurrent solution test
"""
- assert check_recurrent_solution('a', ['a'])
+ assert check_recurrent_solution("a", ["a"])