Implement ONEAPI_DEVICE_SELECTOR in UR #740

Wee-Free-Scot · 2023-07-21T16:56:15Z

Add urDeviceGetSelected and implement to return only those devices specified by the ONEAPI_DEVICE_SELECTOR (for the platform given as input).

Add urPlatformGetSelected and implement to return only those platforms specified by the ONEAPI_DEVICE_SELECTOR.

Intended to fix issue: #220

pbalcer · 2023-07-24T09:05:31Z

With the introduction of config in #681 we have several functions in the API that are loader-only and are excluded from the ddi. Should this be the same? If so, it could also be a good idea to create a separate loader-specific .yml spec file and improve the way the entry point it would contain are excluded from the ddi (right now the scripts look whether the name contains the word 'loader'...).

Wee-Free-Scot · 2023-07-31T13:37:19Z

@pbalcer what are the criteria for loader-only?

ONEAPI_DEVICE_SELECTOR is general to all the devices exposed by UR and applies to all backends -- so it seems incorrect to limit it so that sometimes it takes effect (UR with loader) and other times it does not (UR without loader). Does that make sense?

pbalcer · 2023-07-31T13:54:40Z

Using adapters standalone is not a supported usage model. I think (@kbenzie ?). If it is, it's currently broken (e.g., the loader config symbols would be missing, adapter libraries should only export the DDI symbols).

My understanding is that we want to implement this functionality at the loader level precisely so that it applies uniformly to all downstream adapters. If we treat the urPlatformGetSelected and urDeviceGetSelected just like all other APIs, each adapter would have to look at the environment variable individually, and then we'd have to stitch that together at the loader layer - that seems more complicated.

kbenzie · 2023-07-31T14:17:12Z

Using adapters standalone is not a supported usage model. I think (@kbenzie ?).

That is correct.

My understanding is that we want to implement this functionality at the loader level precisely so that it applies uniformly to all downstream adapters. If we treat the urPlatformGetSelected and urDeviceGetSelected just like all other APIs, each adapter would have to look at the environment variable individually, and then we'd have to stitch that together at the loader layer - that seems more complicated.

This matches my expectations.

Wee-Free-Scot · 2023-08-02T13:48:37Z

@pbalcer

create a separate loader-specific .yml spec file and improve the way the entry point it would contain are excluded from the ddi

Is that something I could/should do as part of this PR or does that add another dependency?

For now, in this PR, should I just insert "loader" into the API names?

pbalcer · 2023-08-02T14:01:28Z

@pbalcer

create a separate loader-specific .yml spec file and improve the way the entry point it would contain are excluded from the ddi

Is that something I could/should do as part of this PR or does that add another dependency?

For now, in this PR, should I just insert "loader" into the API names?

No, if everyone agrees that this makes sense, I can do that refactoring in a separate PR.

pbalcer · 2023-08-03T09:31:13Z

Done in #770

include/ur_api.h

Wee-Free-Scot · 2023-12-08T19:10:25Z

source/ur_api.cpp

+ ///< pNumDevices will be updated with the total number of selected devices
+ ///< available for the given platform.
+) {
+ ur_result_t result = UR_RESULT_SUCCESS;


What is going on here? Is this just a hangover from the previous non-loader version?

this file is just a sanity-check whether the header file compiles and is valid C. We should remove it.

CMakeLists.txt

kbenzie · 2023-12-11T15:45:06Z

source/loader/ur_lib.cpp

+ std::regex validation_pattern("^("
+ "\\*" // C++ escape for \, regex escape for literal '*'
+ "|"
+ "cpu" // ensure case-insenitive, when using
+ "|"
+ "gpu" // ensure case-insenitive, when using
+ "|"
+ "fpga" // ensure case-insenitive, when using
+ "|"
+ "[[:digit:]]+" // '<num>'
+ "|"
+ "[[:digit:]]+\\.[[:digit:]]+" // '<num>.<num>'
+ "|"
+ "[[:digit:]]+\\.\\*" // '<num>.*.*'
+ "|"
+ "\\*\\.\\*" // C++ and regex escapes, literal '*.*'
+ "|"
+ "[[:digit:]]+\\.[[:digit:]]+\\.[[:digit:]]+" // '<num>.<num>.<num>'
+ "|"
+ "[[:digit:]]+\\.[[:digit:]]+\\.\\*" // '<num>.<num>.*'
+ "|"
+ "[[:digit:]]+\\.\\*\\.\\*" // '<num>.*.*'
+ "|"
+ "\\*\\.\\*\\.\\*" // C++ and regex escapes, literal '*.*.*'
+ ")$", std::regex_constants::icase);


We recently removed use of std::regex from the OpenCL adapter in #1039 due to profiling with PIN showed it to produce incredibly innefficient code. That was only for the OpenCL adapter, this would be on the critical path which makes me a bit uncomfortable.

That (don't use std::regex) is good to know, thanks.

Currently, that regex pattern is only referenced in this comment:
line 491 // TODO -- use regex validation_pattern to catch all other syntax errors in the ODS string

Without this catch-all, some syntax errors will make it into the code that assumes there are no syntax errors, which is not great.

I would not expect that this code would ever be on the critical path of a performance critical loop. I would not expect users to query the list of devices (directly or indirectly) in a performance critical situation. Do you anticipate user code that does this?

To mitigate such user code (if it is a design goal), the parsing of the env var (including creating and using the regex) could be done only once and the outcome could be cached in the loader. The current code does not implement that, but it could.

Currently, that regex pattern is only referenced in this comment:
line 491 // TODO -- use regex validation_pattern to catch all other syntax errors in the ODS string

Is the regex only intended to be used during validation or as part of the parsing of the ODS string?

To mitigate such user code (if it is a design goal), the parsing of the env var (including creating and using the regex) could be done only once and the outcome could be cached in the loader. The current code does not implement that, but it could.

I think this could be acceptable.

codecov-commenter · 2023-12-12T18:51:17Z

Codecov Report

Attention: 486 lines in your changes are missing coverage. Please review.

Comparison is base (be849b2) 15.33% compared to head (60373fa) 15.12%.
Report is 4 commits behind head on main.

Files	Patch %	Lines
source/loader/ur_lib.cpp	0.00%	275 Missing ⚠️
test/conformance/device/urDeviceGetSelected.cpp	0.00%	175 Missing ⚠️
include/ur_print.hpp	0.00%	28 Missing ⚠️
source/loader/ur_libapi.cpp	0.00%	4 Missing ⚠️
source/loader/ur_print.cpp	0.00%	4 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #740      +/-   ##
==========================================
- Coverage   15.33%   15.12%   -0.22%     
==========================================
  Files         241      242       +1     
  Lines       34820    35306     +486     
  Branches     3989     4044      +55     
==========================================
- Hits         5340     5339       -1     
- Misses      29429    29916     +487     
  Partials       51       51

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fabiomestre · 2024-01-05T17:58:15Z

test/conformance/device/urDeviceGetSelected.cpp

@@ -0,0 +1,249 @@
+// Copyright (C) 2022-2023 Intel Corporation


Suggested change

// Copyright (C) 2022-2023 Intel Corporation

// Copyright (C) 2024 Intel Corporation

fabiomestre · 2024-01-05T17:59:37Z

Should we document the expected behaviour of the environment variable in the UR specification? I have concerns that someone trying to implement an adapter for UR wouldn't be able to find all the information needed to implement urDeviceGetSelected() from the UR spec alone.

Also have some concerns about the naming. I think that UR should strive to be agnostic regarding the upstream runtimes. Adding support for a variable with "ONEAPI" in the name goes against that in my opinion. I think we could easily map ONEAPI_DEVICE_SELECTOR to a UR specific variable while keeping the same syntax for simplicity. However, I know there have been some discussions about the naming of ONEAPI_DEVICE_SELECTOR before, so there might be a good reason not to do this.

kbenzie · 2024-01-08T10:42:19Z

Also have some concerns about the naming. I think that UR should strive to be agnostic regarding the upstream runtimes. Adding support for a variable with "ONEAPI" in the name goes against that in my opinion. I think we could easily map ONEAPI_DEVICE_SELECTOR to a UR specific variable while keeping the same syntax for simplicity. However, I know there have been some discussions about the naming of ONEAPI_DEVICE_SELECTOR before, so there might be a good reason not to do this.

This has come up a few times, the naming is not going to change since its a product stack naming choice and applies to all of oneAPI, not just UR. Also note that this repo exists in the oneapi-src GitHub org, so its not out of place here.

Wee-Free-Scot · 2024-01-08T17:04:44Z

I have concerns that someone trying to implement an adapter for UR wouldn't be able to find all the information needed to implement urDeviceGetSelected() from the UR spec alone.

No adapter implements this API -- it is loader only. The implementation in loader uses urDeviceGet from the adapter(s), then collates and selects the information that it will pass on to the client.

kbenzie · 2024-01-17T15:13:23Z

I thought which occurred to me recently @Wee-Free-Scot. When we merge this, how would it interact with the implementation of ONEAPI_DEVICE_SELECTOR which already exists in the SYCL RT?

Would it not matter because the SYCL RT isn't using the UR loader yet?

Wee-Free-Scot · 2024-01-17T16:31:48Z

There are at least two "wouldn't matter" arguments:

SYCL RT does not use URT yet, so the new API will never get called.
URT provides both urDeviceGet (unadulterated list of devices passed through from the underlying adapter) and urDeviceGetSelected (modified list of devices applying ONEAPI_DEVICE_SELECTOR), so future versions of SYCL RT can chose when to call the new API.

I would expect an incremental approach to implementation usage in SYCL RT:

current -- gets list of devices without URT, runs SYCL RT code to apply ODS
intermediate -- gets list of devices from urDeviceGet, runs SYCL RT code to apply ODS, regression test vs, (1)
target -- gets list of devices from urDeviceGetSelected, bypasses SYCL RT code to apply ODS, regression test vs, (2)

kbenzie · 2024-01-17T17:00:24Z

Makes sense. Do we have regression tests for 2. intermediate/3. target at the moment?

Wee-Free-Scot · 2024-01-17T17:08:14Z

Some of the UR tests that I created in this PR do a regression test of sorts: they use urDeviceGet and urDeviceGetSelected with ODS set such that I should/shouldn't get the same output. It's not complete coverage, but it hits some expected use-cases.

In terms of regression testing for SYCL RT, we'd need to involve the appropriate people when we know who will do the implementation of future implementations (2) and (3).

kbenzie · 2024-01-17T17:10:29Z

In terms of regression testing for SYCL RT, we'd need to involve the appropriate people when we know who will do the implementation of future implementations (2) and (3).

Given my team will be responsible for replacing PI with UR in the SYCL RT, that could be us.

kbenzie · 2024-01-17T17:29:05Z

I'm testing this in my system which has both an NVIDIA and an AMD GPU. The topology looks like this:

.
└── adapters
    ├── 0 hip
    │   └── platforms
    │       └── 0
    │           └── devices
    │               └── 0 AMD Radeon RX 6700 XT
    └── 1 cuda
        └── platforms
            └── 0
                └── devices
                    └── 0 NVIDIA GeForce RTX 3060 Ti

I've modified the urinfo tool to use urDeviceGetSelected instead of urDeviceGet for getting quick results. This is without ONEAPI_DEVICE_SELECTOR set, both GPU's are listed as expected:

$ build-r/bin/urinfo
DEBUG: getenv_to_map parsed env var and failed to produce a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and failed to produce a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and failed to produce a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and failed to produce a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
[adapter(0,hip):platform(0):device(0,gpu)] AMD HIP BACKEND, AMD Radeon RX 6700 XT gfx1031 [HIP 60032.83]
[adapter(1,cuda):platform(0):device(0,gpu)] NVIDIA CUDA BACKEND, NVIDIA GeForce RTX 3060 Ti 8.6 [CUDA 12.2]

And when set to cuda:gpu:

$ ONEAPI_DEVICE_SELECTOR=cuda:gpu build-r/bin/urinfo
ninja: Entering directory `build-r'
ninja: no work to do.
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
WARNING: ignoring term with irrelevant backend
error: urDeviceGetSelected(platform, UR_DEVICE_TYPE_ALL, 0, nullptr, &numDevices) failed: UR_RESULT_ERROR_INVALID_VALUE

And set to hip:gpu:

$ ONEAPI_DEVICE_SELECTOR=hip:gpu build-r/bin/urinfo
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
WARNING: ignoring term with irrelevant backend
error: urDeviceGetSelected(platform, UR_DEVICE_TYPE_ALL, 0, nullptr, &numDevices) failed: UR_RESULT_ERROR_INVALID_VALUE

What I think is happening here is these GPU's are provided by different adapters but the currently implementation of urDeviceGetSelected is only considering the devices within the platform it is given and thus only the devices in a single adapter.

Wee-Free-Scot · 2024-01-17T17:41:57Z

Only looking for devices matching both the input parameters and the ODS env var is the intent of the code.
I was expecting the new API to be a drop-in replacement for the existing UR function, which takes a platform handle.
I was anticipating that the caller/client would already have code for loading each adapter and calling urDeviceGet using each of the platform handles in turn.
If platform==level_zero and ODS==cuda:gpu then the list of selected devices is supposed to be empty.
The error looks wrong -- it should return an empty list, i.e. numDevices:=0 and return value UR_SUCCESS.

kbenzie · 2024-01-17T17:55:11Z

The error looks wrong -- it should return an empty list, i.e. numDevices:=0 and return value UR_SUCCESS.

Okay, I made this change:

benie@bench git diff source/loader/ur_lib.cpp
diff --git a/source/loader/ur_lib.cpp b/source/loader/ur_lib.cpp
index 17489efc..64a0f3bb 100644
--- a/source/loader/ur_lib.cpp
+++ b/source/loader/ur_lib.cpp
@@ -564,7 +564,7 @@ ur_result_t urDeviceGetSelected(ur_platform_handle_t hPlatform,

     if (acceptDeviceList.size() == 0 && discardDeviceList.size() == 0) {
         // nothing in env var was understood as a valid term
-        return UR_RESULT_ERROR_INVALID_VALUE;
+        return UR_RESULT_SUCCESS;
     } else if (acceptDeviceList.size() == 0) {
         // no accept terms were understood, but at least one discard term was
         // we are magnanimous to the user when there were bad/ignored accept terms

And am now getting these results:

$ ONEAPI_DEVICE_SELECTOR=cuda:gpu build-run urinfo
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
WARNING: ignoring term with irrelevant backend
No devices found platform 0.
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
[adapter(1,cuda):platform(0):device(0,gpu)] NVIDIA CUDA BACKEND, NVIDIA GeForce RTX 3060 Ti 8.6 [CUDA 12.2]

$ ONEAPI_DEVICE_SELECTOR=hip:gpu buildr-/bin/urinfo
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
DEBUG: size of acceptDeviceList = 1
DEBUG: size of discardDeviceList = 0
DEBUG: In ApplyFilter, if block case 1, matches = 1
DEBUG: getenv_to_map parsed env var and produced a map
DEBUG: termType isAcceptFilter
WARNING: ignoring term with irrelevant backend
No devices found platform 0.
[adapter(0,hip):platform(0):device(0,gpu)] AMD HIP BACKEND, AMD Radeon RX 6700 XT gfx1031 [HIP 60032.83]

Which is looking much better.

Wee-Free-Scot · 2024-01-17T17:59:54Z

The ODS=cuda:gpu run looks good, but the ODS=hip:gpu run gives contradictory output "No devices found platform 0" followed immediately by details of the 1 device in that zero-length list! Without the debug messages, it looks fine, but how it gets the answer is probably significant.

Wee-Free-Scot · 2024-01-17T18:02:18Z

Is there threading involved here? Are the output lines only partially-ordered? Does it check platform not-0 before it checks platform 0 and then mislabel one of the two final output lines?

kbenzie · 2024-01-17T18:04:33Z

The ODS=cuda:gpu run looks good, but the ODS=hip:gpu run gives contradictory output "No devices found platform 0" followed immediately by details of the 1 device in that zero-length list! Without the debug messages, it looks fine, but how it gets the answer is probably significant.

They both print No devices found platform 0. but at different points, that's coming from urinfo so would need to update what it prints.

Is there threading involved here? Are the output lines only partially-ordered? Does it check platform not-0 before it checks platform 0 and then mislabel one of the two final output lines?

There's no threading, its this function urinfo::app::enumerateDevices() but using urDevicesGetSelected instead of urDevicesGet.

Now that oneapi-src#740 is merged, actually use `urDeviceGetSelected` in the `urinfo` tool to mirror the behaviour of the `sycl-ls` tool.

Wee-Free-Scot marked this pull request as draft July 21, 2023 16:56

Wee-Free-Scot added the enhancement New feature or request label Jul 21, 2023

Wee-Free-Scot mentioned this pull request Jul 24, 2023

Provide an Implementation of ONEAPI_DEVICE_SELECTOR #220

Closed

Wee-Free-Scot force-pushed the Issue220_Add-support-for-ODS branch 2 times, most recently from 999fcbb to 8907ddb Compare August 3, 2023 17:35

jandres742 reviewed Sep 22, 2023

View reviewed changes

include/ur_api.h Show resolved Hide resolved

Wee-Free-Scot force-pushed the Issue220_Add-support-for-ODS branch from 093dca6 to 22578d2 Compare December 8, 2023 19:04

Wee-Free-Scot commented Dec 8, 2023

View reviewed changes

pbalcer reviewed Dec 11, 2023

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

kbenzie reviewed Dec 11, 2023

View reviewed changes

Wee-Free-Scot marked this pull request as ready for review January 5, 2024 11:19

Wee-Free-Scot requested a review from a team as a code owner January 5, 2024 11:19

Wee-Free-Scot self-assigned this Jan 5, 2024

Wee-Free-Scot requested review from kbenzie, jandres742 and pbalcer and removed request for jandres742 January 5, 2024 12:11

fabiomestre reviewed Jan 5, 2024

View reviewed changes

kbenzie force-pushed the Issue220_Add-support-for-ODS branch from 4c07be8 to 60373fa Compare February 13, 2024 11:34

Wee-Free-Scot and others added 13 commits February 13, 2024 16:15

Alpha version of code for ODS via adapters

b517076

Fix-ups for first batch of unit tests

a44f811

Fix strange capture warning/error

6d25c1b

Fix std::memcpy error

7fd776b

Fix code formatting mistakes

01ba391

Fix more code format mistakes

085d21b

Adding tests for ODS function urDeviceGetSelected

588966f

Fix explict path for helpers.h

e246aba

Apply code formatting patches

2e01ec1

Use URT logger for output messages

d57f79a

Squashing two CI/CD errors

49b09e0

Generate code with required clang-format version

085f991

kbenzie force-pushed the Issue220_Add-support-for-ODS branch from 60373fa to 085f991 Compare February 14, 2024 11:12

kbenzie approved these changes Feb 14, 2024

View reviewed changes

kbenzie merged commit 4e69cc6 into oneapi-src:main Feb 22, 2024
52 checks passed

kbenzie added a commit to kbenzie/unified-runtime that referenced this pull request Feb 23, 2024

[urinfo] Use urDeviceGetSelected

4957bd0

Now that oneapi-src#740 is merged, actually use `urDeviceGetSelected` in the `urinfo` tool to mirror the behaviour of the `sycl-ls` tool.

kbenzie mentioned this pull request Feb 27, 2024

[urinfo] Use urDeviceGetSelected #1389

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement ONEAPI_DEVICE_SELECTOR in UR #740

Implement ONEAPI_DEVICE_SELECTOR in UR #740

Wee-Free-Scot commented Jul 21, 2023 •

edited

Loading

pbalcer commented Jul 24, 2023

Wee-Free-Scot commented Jul 31, 2023

pbalcer commented Jul 31, 2023 •

edited

Loading

kbenzie commented Jul 31, 2023

Wee-Free-Scot commented Aug 2, 2023

pbalcer commented Aug 2, 2023

pbalcer commented Aug 3, 2023

Wee-Free-Scot Dec 8, 2023

pbalcer Dec 11, 2023

kbenzie Dec 11, 2023

Wee-Free-Scot Dec 11, 2023

kbenzie Dec 12, 2023

codecov-commenter commented Dec 12, 2023 •

edited

Loading

fabiomestre Jan 5, 2024

fabiomestre commented Jan 5, 2024

kbenzie commented Jan 8, 2024

Wee-Free-Scot commented Jan 8, 2024

kbenzie commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

kbenzie commented Jan 17, 2024 •

edited

Loading

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

		@@ -0,0 +1,249 @@
		// Copyright (C) 2022-2023 Intel Corporation

	// Copyright (C) 2022-2023 Intel Corporation
	// Copyright (C) 2024 Intel Corporation

Implement ONEAPI_DEVICE_SELECTOR in UR #740

Implement ONEAPI_DEVICE_SELECTOR in UR #740

Conversation

Wee-Free-Scot commented Jul 21, 2023 • edited Loading

pbalcer commented Jul 24, 2023

Wee-Free-Scot commented Jul 31, 2023

pbalcer commented Jul 31, 2023 • edited Loading

kbenzie commented Jul 31, 2023

Wee-Free-Scot commented Aug 2, 2023

pbalcer commented Aug 2, 2023

pbalcer commented Aug 3, 2023

Wee-Free-Scot Dec 8, 2023

Choose a reason for hiding this comment

pbalcer Dec 11, 2023

Choose a reason for hiding this comment

kbenzie Dec 11, 2023

Choose a reason for hiding this comment

Wee-Free-Scot Dec 11, 2023

Choose a reason for hiding this comment

kbenzie Dec 12, 2023

Choose a reason for hiding this comment

codecov-commenter commented Dec 12, 2023 • edited Loading

Codecov Report

fabiomestre Jan 5, 2024

Choose a reason for hiding this comment

fabiomestre commented Jan 5, 2024

kbenzie commented Jan 8, 2024

Wee-Free-Scot commented Jan 8, 2024

kbenzie commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

kbenzie commented Jan 17, 2024 • edited Loading

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

Wee-Free-Scot commented Jan 17, 2024

kbenzie commented Jan 17, 2024

Wee-Free-Scot commented Jul 21, 2023 •

edited

Loading

pbalcer commented Jul 31, 2023 •

edited

Loading

codecov-commenter commented Dec 12, 2023 •

edited

Loading

kbenzie commented Jan 17, 2024 •

edited

Loading