Skip to content

Latest commit

 

History

History
523 lines (504 loc) · 12.1 KB

README.md

File metadata and controls

523 lines (504 loc) · 12.1 KB

Towards Correctness Checking of MPI Partitioned Communication in MUST - Artifact

This repository contains the artifact with the test cases for the paper Correctness Checking of MPI+OpenMP Applications Using Vector Clocks in MUST. The test cases used are in the tests/ folder and are grouped by MPI-CorrBench (COBE) and our own test suite, as well as correct and erroneous test cases.

Flaky Test Cases

Some test cases are not uniquely classified. For the counter-based MPI-Serialized analysis, the error has to manifest to be detected, meaning any of the erroneous thread-level test cases could be classified as TP or FN. This includes the four test cases in our own test suite:

  • ThreadLevel_serialized_error.cpp
  • ThreadLevel_serialized_few_tasks_error.cpp
  • ThreadLevel_serialized_many_tasks_error.cpp
  • ThreadLevel_serialized_sections_error.cpp

And the single test case from MPI-CorrBench:

  • wrong_threading_level_3.c

The ThreadLevel_serialized_many_tasks_error.cpp test case performs many concurrent MPI calls and is more consistently classified as TP. The other four test cases are not consistently classified as TP or FN.

False Positive Test Case'

The test case own_checks/ConComm/correct/concurrent_communicator_transitive.cpp is a correct test for the concurrent collectives that use the same communicator error class, that uses MPI synchronization to synchronize the threads of the first process transitively. Since the clock-based approach currently does not track the MPI synchronization it also does not recognize this transitive synchronization and reports this test case as erroneous even though the execution of the collectives that use the same communicator are not concurrent, and hence clock-based approach classifies this test case as (FP). MUST does not have a counter-based approach for this error class and hence classifies this test case as TN.

Classification Results

  • TP: True Positive (correct-alert)
  • FP: False Positive (false-alert)
  • TN: True Negative (error-free)
  • FN: False Negative (missing-alert)
  • A: Accuracy
Analyses Test Suite Total TP FP TN FN A
Counter-based Own 30 1 0 17 12 0.60
Clock-based Own 30 13 1 16 0 0.97
Counter-based COBE 20 0 0 10 10 0.50
Clock-based COBE 20 10 0 10 0 1.00

Detailed Results

Test Suite Test Error class Type Counter-based Clock-based
COBE deadlock_probe_2 Probe Correct TN TN
COBE probe Probe Correct TN TN
COBE two_collectives Collective Correct TN TN
COBE two_collectives_2 Collective Correct TN TN
COBE two_collectives_3 Collective Correct TN TN
COBE two_collectives_4 Collective Correct TN TN
COBE two_collectives_5 Collective Correct TN TN
COBE two_collectives_6 Collective Correct TN TN
COBE two_collectives_7 Collective Correct TN TN
COBE two_collectives Thread-serialized Correct TN TN
COBE deadlock_probe_2 Probe Erroneous FN TP
COBE probe Probe Erroneous FN TP
COBE two_collectives Collectives Erroneous FN TP
COBE two_collectives_2 Collectives Erroneous FN TP
COBE two_collectives_3 Collectives Erroneous FN TP
COBE two_collectives_4 Collectives Erroneous FN TP
COBE two_collectives_5 Collectives Erroneous FN TP
COBE two_collectives_6 Collectives Erroneous FN TP
COBE two_collectives_7 Collectives Erroneous FN TP
COBE wrong_threading_level_3 Thread-serialized Erroneous FN TP
Own concurrent_recv Receive Correct TN TN
Own concurrent_communicator Collective Correct TN TN
Own concurrent_communicator_for_dynamic Collective Correct TN TN
Own concurrent_communicator_for_nowait Collective Correct TN TN
Own concurrent_communicator_sections Collective Correct TN TN
Own concurrent_communicator_single Collective Correct TN TN
Own concurrent_communicator_task_nesting Collective Correct TN TN
Own concurrent_communicator_tasking_depend Collective Correct TN TN
Own concurrent_communicator_tasking_locks Collective Correct TN TN
Own concurrent_communicator_transitive Collective Correct TN FP
Own concurrent_recv_probe Probe Correct TN TN
Own concurrent_recv_probe_any_tag Probe Correct TN TN
Own ThreadLevel_serialized Thread-serialized Correct TN TN
Own ThreadLevel_serialized_single Thread-serialized Correct TN TN
Own ThreadLevel_serialized_tasking_depend Thread-serialized Correct TN TN
Own ThreadLevel_serialized_tasking_locks Thread-serialized Correct TN TN
Own ThreadLevel_serialized_taskwait Thread-serialized Correct TN TN
Own concurrent_recv_any_tag_error Receive Erroneous FN TP
Own concurrent_recv_error Receive Erroneous FN TP
Own concurrent_communicator_error Collective Erroneous FN TP
Own concurrent_communicator_for_dynamic_nowait_error Collective Erroneous FN TP
Own concurrent_communicator_sections_error Collective Erroneous FN TP
Own concurrent_communicator_task_nesting_error Collective Erroneous FN TP
Own concurrent_communicator_tasking_error Collective Erroneous FN TP
Own concurrent_recv_probe_any_tag_error Probe Erroneous FN TP
Own concurrent_recv_probe_error Probe Erroneous FN TP
Own ThreadLevel_serialized_error Thread-serialized Erroneous Flaky (mostly FN) TP
Own ThreadLevel_serialized_few_tasks_error Thread-serialized Erroneous Flaky (mostly FN) TP
Own ThreadLevel_serialized_many_tasks_error Thread-serialized Erroneous TP (inherently flaky) TP
Own ThreadLevel_serialized_sections_error Thread-serialized Erroneous Flaky (mostly FN) TP

Building Docker Container

The docker container installs MPICH v4.0.2, Clang v15.0.6, MUST, and LLVM lit. To build the container, run in the root directory:

docker build -t must-omp-clock .

Running Tests

To run all test cases for the counter-based and clock-based analyses, run:

docker run -it must-omp-clock ./tests/run_tests.sh

(Make sure that run_tests.sh uses LF line endings and not CRLF, otherwise Linux will provide the error message "No such file or directory")

To start a specific test case, run:

docker run -e LIT_MUST_VERSION=<analysis-type> -it must-omp-clock lit -j1 -a <path-to-test>