Metric Comparison + API Updates

[donn]

• openlane.common
  • Created new function get_latest_file, that finds the latest file with a certain filename in a specific folder
  • Created new Filter object aggregating the functionality of wildcards
  • Created new module openlane.common.metrics to aggregate functions dealing with metrics, replacing previous file
    • __main__ file of this module has three commands, compare, compare-multiple, compare-main:
      • compare: Compares two metrics files and generates an output summary
      • compare-multiple: Compares two folders of metrics files and generates an output summary
      • compare-main: Compares a folder of metric files with metrics fetched from efabless/openlane-metrics and generates an output summary
    • New Metric object created, encapsulating the metric name, its aggregator, and some other functionality related to evaluating changes in metrics
      • Also allowed some metrics to exclude some groups from aggregation, e.g. iter for routing metrics
    • Created "library" of Metrics of the various metrics we aggregated so far
      • aggregate_metrics now uses the metric library for aggregators as a default value
    • Updated parsing of metrics to more closely match https://github.com/ieee-ceda-datc/datc-rdf-Metrics4ML: the format for modifiers was wrong
• openlane.state
  • Created __main__ file that returns the latest state for a run directory (using get_latest_file, with an options to extract the metrics thereof for a file (useful for scripts)
• openlane.config.Config:
  • Updated refg:: to return the glob string verbatim if the glob returns no results
• openlane.config.Macro:
  • Created new method from_state that automatically creates a Macro object from a State object
• Created new object, ScopedFile, which is a file that's deleted as soon as it's deconstructed
• KLayout.*:
  • Unified generation of commandline arguments using .get_cli_args()
  • Updated scripts to use %OL_METRIC_I instead of having to use other communication channels
  • Made sure all paths passed to KLayout commands are absolute because KLayout loves to not use CWD for relative paths
• Magic.*:
  • Fixed bug when MAGICRC is a relative path
• Netgen.LVS, Checker.LVS:
  • Updated metric names for grammatical consistency
• Added httpx to replace requests as the dependency for HTTP functions

CI and Testing

• Added function in CI to test every PR against the current main function
• Added pytest-xdist to dev dependencies, so multiple tests can be run in parallel

---

Depends on efabless/openlane2-step-unit-tests#21

[kareefardi]

Couple general additional comments:

1- design__lint_errors__count -> design__lint_error__count where error is without an s.
2- I think any change in a critical variable should be reported regardless of value higher_is_better. As such, table verbosity CRITICAL will always display critical changes.
3- How to apply filter?

python3 -m openlane.common.metrics compare ./designs/spm/runs/run_1/final/metrics.json ./designs/spm/runs/run_2/final/metrics.json --table-out diff.out --filter 'design__lint_errors__count'

and

python3 -m openlane.common.metrics compare ./designs/spm/runs/run_1/final/metrics.json ./designs/spm/runs/run_2/final/metrics.json --table-out diff.out 

produce the same diff.out file.
4- I think a CLI argument for decimal points would be a nice addition.
5- deltas in openlane.common.metrics compare are displayed at the end of the file. I think they should be at the beginning of the file.

[kareefardi]

why would that be an option ?

[donn]

If you just want the highlights and not the table

[donn]

@kareefardi I can't say I agree with every change to a "critical" metric being critical. For example-- slack increasing is not "critical" as in demanding attention, but slack decreasing very much is.

What this tells me is that we're not aligned on the definition of critical: my definition of critical was "a change impacting the manufacturability of a design." Any change in I/O pin count, for example, is critical (as evidenced by it having no higher_is_better property) as there is no ordinary circumstance in which that would change-- but there are some metrics where only negative changes impact the manufacturability of the design. Meanwhile, OpenROAD rsz getting better at its job would increase slacks across all designs and, under this definition, would not add immense noise to the reports without cause for concern.

How did you define critical?

[donn]

@kareefardi Bot issue seems to be that the current HEAD of master doesn't have metrics uploaded. Simplest solution would be to just merge this PR.