diff --git a/.github/workflows/forge-specs.yml b/.github/workflows/forge-specs.yml index 0871067..b0ade72 100644 --- a/.github/workflows/forge-specs.yml +++ b/.github/workflows/forge-specs.yml @@ -1,7 +1,7 @@ on: push -name: Build +name: Macgonuts CI jobs: - Build-Task: + Build: runs-on: ubuntu-latest steps: - name: Install basic tools @@ -9,6 +9,9 @@ jobs: run: | sudo apt-get install git sudo apt-get install gcc-9 + sudo apt-get update + sudo apt-get install perl + sudo apt-get install lcov - name: Install Hefesto shell: bash run: | @@ -21,6 +24,15 @@ jobs: sudo chown -R runner /usr/local/share/hefesto cd ../.. rm -rf hefesto + - name: Install lcov-generator + shell: bash + run: | + git clone https://github.com/rafael-santiago/helios + cd helios + sudo -E hefesto --install=lcov-generator + sudo chown -R runner /usr/local/share/hefesto + cd ../ + rm -rf helios - name: Clone project repo uses: actions/checkout@v3 with: @@ -29,5 +41,15 @@ jobs: shell: bash run: | cd src - sudo -E hefesto - + sudo -E hefesto --coverage + - name: Tar coverage report + shell: bash + run: | + sudo tar -cvf /home/libmacgonuts-coverage.tar src/reports/macgonuts-static-lib + sudo chown runner /home/libmacgonuts-coverage.tar + - name: Upload LCOV results + uses: actions/upload-artifact@v2 + with: + name: libmacgonuts-coverage-report + path: /home/libmacgonuts-coverage.tar + retention-days: 7 diff --git a/.gitignore b/.gitignore index 6590ff4..872b932 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ **/bin/* *.Forgefile-* *.o/ +coverage.info diff --git a/README.md b/README.md index b678731..2effe85 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,10 @@ -# Macgonuts ![ci-status](https://github.com/rafael-santiago/macgonuts/actions/workflows/forge-specs.yml/badge.svg) +# Macgonuts ![ci-status](https://github.com/rafael-santiago/macgonuts/actions/workflows/forge-specs.yml/badge.svg) ![c](https://img.shields.io/badge/is_what_we_speak-black?logo=c&logoColor=white&style=plastic) ![suckless](https://img.shields.io/badge/is_what_we_seek_to_follow-black?logo=suckless&logoColor=white&style=plastic) + +![linux-function-coverage](https://img.shields.io/badge/function_coverage-0%25-red?logo=linux&logoColor=white&style=plastic) ![linux-line-coverage](https://img.shields.io/badge/line_coverage-0%25-red?logo=linux&logoColor=white&style=plastic) + +![freebsd-function-coverage](https://img.shields.io/badge/function_coverage-0%25-red?logo=freebsd&logoColor=white&style=plastic) ![freebsd-line-coverage](https://img.shields.io/badge/function_coverage-0%25-red?logo=freebsd&logoColor=white&style=plastic) + +--- ``Macgonuts`` is an ``ARP/NDP`` swiss army knife to make ``MAC addresses`` going nuts on networks around! diff --git a/doc/BUILD.md b/doc/BUILD.md index 73a4bac..fe58c3c 100644 --- a/doc/BUILD.md +++ b/doc/BUILD.md @@ -12,6 +12,7 @@ fresh ``macgonuts`` binary to get your stuff done, you can give ``the low-cost b - [Installing Hefesto](#installing-hefesto) - [The low-cost build](#the-low-cost-build) - [The developer's build](#developers-build) + - [Extracting code coverage](#extracting-code-coverage) - [Installing the command line tool](#installing-the-command-line-tool) ## Getting newest macgonuts source code revision @@ -56,6 +57,17 @@ you@somewhere-over-the-rainbow:~/hefesto/src# logout (redo login and you done) ``` +Now you need to install some conveniences for code coverage extractions, so you need to clone `Helios` +and install `lcov-generator`: + +``` +you@somewhere-over-the-rainbow:~# git clone https://github.com/rafael-santiago/helios +you@somewhere-over-the-rainbow:~# cd helios +you@somewhere-over-the-rainbow:~/helios# hefesto --install=lcov-generator +you@somewhere-over-the-rainbow:~/helios# cd .. +you@somewhere-over-the-rainbow:~# rm -rf helios +``` + You can also run the script ``get-hefesto.sh`` into ``src`` folder of ``Macgonuts``. [``Back``](#topics) @@ -143,6 +155,36 @@ remembering you that your code is actually working and that ``TDD`` matters. :ra [``Back``](#topics) +### Extracting code coverage + +``Macgonuts`` build gives support for code coverage extraction, it support ``gcov`` or ``llvm-cov``. You also need to +have ``lcov`` well-installed more on that [here](https://github.com/linux-test-project/lcov). + +By using ``Hefesto`` we can easily extract the code coverage of ``Macgonuts`` by invoking ``Hefesto`` as follows: + +``` +you@somewhere-over-the-rainbow:~/macgonuts/src# hefesto --coverage +``` + +By default the report will be generated under ``src/reports`` directory. If you want to specify a directory to generate +the reports you can pass the option ``--genhtml-outpath=`` option: + +``` +you@somewhere-over-the-rainbow:~/macgonuts/src# hefesto --coverage \ +> --genhtml-outpath=/mnt/tdd/rocks +``` + +By design we are only extracting code coverage from ``libmacgnuts`` (the main project under ``src``). +The ``cmd-tool`` is pretty hard for unit testing since it would involve run all attacks that this tool +implements in form of commands (a.k.a tasks) from the github actions' runner. Sincerely, it would be not +easy to do from a rather ``restricted-docker-velotrol-like`` [sic] environment. So, *C'est la vie!* + +> - Wait. What does *"velotrol"* is?! + +Well, a image will make you understand my point much better, [look](https://duckduckgo.com/?q=velotrol&t=h_&iax=images&ia=images)! :rofl: + +[``Back``](#topics) + ## Installing the command line tool Having ``Hefesto`` well installed all you need is move to ``src`` toplevel subdirectory and run the following: diff --git a/src/Forgefile.hsl b/src/Forgefile.hsl index 78db51d..2e1b68a 100644 --- a/src/Forgefile.hsl +++ b/src/Forgefile.hsl @@ -71,6 +71,18 @@ macgonuts-static-lib.epilogue() { $subprojects.add_item("test"); $subprojects.add_item("cmd"); if (build_projects($subprojects) == 0) { + var option type list; + $option = hefesto.sys.get_option("coverage"); + if ($option.count() > 0) { + var report_path type string; + $report_path = hefesto.sys.make_path(get_coverage_report_dir(), "index.html"); + var fcov type string; + $fcov = get_function_coverage($report_path); + var lcov type string; + $lcov = get_line_coverage($report_path); + hefesto.sys.echo("INFO: Code coverage : [ functions = " + $fcov + " % / lines = " + $lcov + " % ]\n"); + do_break_when_low_coverage($fcov, $lcov); + } hefesto.sys.echo("INFO: Done.\n"); } else { hefesto.project.abort(1); diff --git a/src/build/toolsets.hsl b/src/build/toolsets.hsl index 1724d30..253ac2d 100644 --- a/src/build/toolsets.hsl +++ b/src/build/toolsets.hsl @@ -9,7 +9,9 @@ include ~/toolsets/gcc/gcc-lib.hsl include ~/toolsets/clang/clang-app.hsl include ~/toolsets/clang/clang-lib.hsl include ~/toolsets/common/utils/lang/c/dependency_scanner.hsl +include ~/toolsets/common/utils/lang/c/lcov.hsl include ~/fsutil.hsl +include ~/conv.hsl function installer() : result type none { var option type list; @@ -51,6 +53,20 @@ function runtests(binary type string, args type string) : result type none { if (hefesto.sys.run($binary + " " + $args) != 0) { hefesto.project.abort(1); } + var option type list; + $option = hefesto.sys.get_option("coverage"); + if ($option.count() > 0) { + var obj_output_dir type string; + $option = hefesto.sys.get_option("obj-output-dir"); + if ($option.count() > 0) { + $obj_output_dir = $option.item(0); + } else { + $obj_output_dir = hefesto.sys.pwd(); + } + if (generate_lcov_report($obj_output_dir) != 0) { + hefesto.project.abort(1); + } + } } function set_rootdir(change_to type string) : result type none { @@ -171,6 +187,22 @@ local function build_accacia() : result type int { result $exit_code; } +function get_coverage_report_dir() : result type string { + var report_path type string; + var option type list; + $option = hefesto.sys.get_option("genhtml-outpath"); + var genhtml_outpath type string; + if ($option.count() == 0) { + $report_path = hefesto.sys.make_path(get_rootdir(), + hefesto.sys.make_path("/reports/", hefesto.project.name())); + } else { + $report_path = $option.item(0); + $report_path = hefesto.sys.make_path($report_path, + hefesto.project.name()); + } + result $report_path; +} + local function build_submodule(subdir type string) : result type int { var oldcwd type string; $oldcwd = hefesto.sys.pwd(); @@ -189,6 +221,43 @@ local function build_submodule(subdir type string) : result type int { $build_options.replace("--libraries=.* ", ""); $build_options.replace("--ldflags=.* ", ""); + var coverage type list; + $coverage = hefesto.sys.get_option("coverage"); + var report_path type string; + + var projects2cov type list; + # INFO(Rafael): Add to this list all projects relevant to extract coverage info. + # + # By now we are only extracting coverage info from libmacgonuts. + # The cmd-tool is rather difficult for unit testing since it depends + # on promoting all attacks that its tasks implements. Automating the + # execution of all them into a docker-velotrol-based-environment [sic] + # would be a quixotic task. Even so, if you want to, good luck! + $projects2cov.add_item("macgonuts-static-lib"); + + if ($coverage.count() > 0 + && $subdir == "test") { + var subproject type string; + $subproject = $subdir; + $subproject.replace("/", "-"); + $coverage = hefesto.sys.get_option("genhtml-outpath"); + var genhtml_outpath type string; + $report_path = get_coverage_report_dir(); + if ($coverage.count() > 0) { + $report_path = $coverage.item(0); + $build_options.replace("--genhtml-outpath=.* ", ""); + } + if ($projects2cov.index_of(hefesto.project.name()) > -1) { + $genhtml_outpath = " --genhtml-outpath=" + $report_path; + } + $build_options = $build_options + + " --gcda-search-path=.o," + hefesto.sys.make_path($oldcwd, ".o") + " " + + " --lcov-remove-patterns=*_tests.c,*src/test* " + + " --genhtml-rendering-options=--legend" + + $genhtml_outpath; + + } + var exit_code type int; $exit_code = hefesto.sys.run("hefesto " + $build_options); @@ -197,6 +266,89 @@ local function build_submodule(subdir type string) : result type int { result $exit_code; } +function do_break_when_low_coverage(function_coverage type string, + line_coverage type string) : result type none { + var low_nr type int; + $low_nr = is_low_cov($function_coverage); + if ($low_nr > 0) { + hefesto.sys.echo("ERROR: Low function coverage detected (it must be >= 75) : " + + $function_coverage + "%\n"); + } + $low_nr = $low_nr + is_low_cov($line_coverage); + if ($low_nr > 0) { + hefesto.sys.echo("ERROR: Low line coverage detected (it must be >= 75) : " + + $line_coverage + "%\n"); + } + if ($low_nr > 0) { + hefesto.project.abort(1); + } +} + +local function is_low_cov(coverage type string) : result type int { + var dec type string; + $dec = $coverage; + $dec.replace("\\..*$", ""); + result (str2int($dec) < 75); +} + +function get_function_coverage(report_path type string) : result type string { + result get_coverage_result($report_path, ""); +} + +function get_line_coverage(report_path type string) : result type string { + result get_coverage_result($report_path, ""); +} + +function get_function_coverage_badge(value type string) : result type string { + result get_coverage_badge($value, "function coverage"); +} + +function get_line_coverage_badge(value type string) : result type string { + result get_coverage_badge($value, "line coverage"); +} + +local function get_coverage_badge(value type string, prefix type string) : result type string { + var uri type string; + $prefix.replace(" ", "_"); + $uri = "https://img.shields.io/badge/" + $prefix + "-" + $value + "%25"; + var dec type string; + $dec = $value; + $dec.replace("\\.*$", ""); + var d type int; + $d = str2int($dec); + if ($d >= 90) { + $uri = $uri + "-lime"; + } else if ($d >= 75) { + $uri = $uri + "-yellow"; + } else { + $uri = $uri + "-red"; + } + $uri = $uri + "?logo=" + hefesto.sys.os_name() + "&logoColor=white&style=for-the-badge"; + result $uri; +} + +local function get_coverage_result(report_path type string, pattern type string) : result type string { + var report_data type list; + $report_data = hefesto.sys.lines_from_file($report_path, ".*"); + if ($report_data.count() == 0) { + result "NaN"; + } + var r type int; + $r = 0; + while ($r < $report_data.count()) { + var report_line type string; + $report_line = $report_data.item($r); + if ($report_line.match($pattern) == 1 && ($r + 3) < $report_data.count()) { + $report_line = $report_data.item($r + 3); + $report_line.replace(".*\">", ""); + $report_line.replace(" %$", ""); + result $report_line; + } + $r = $r + 1; + } + result "NaN"; +} + local function build_cutest() : result type int { var oldcwd type string; $oldcwd = hefesto.sys.pwd();