Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Formatting in extensions #255

Merged
merged 24 commits into from
Oct 4, 2024
Merged

Formatting in extensions #255

merged 24 commits into from
Oct 4, 2024

Conversation

willtebbutt
Copy link
Member

@willtebbutt willtebbutt commented Sep 13, 2024

For reasons that I'm not entirely clear on, I had really poorly formatted basically all of the package extensions. This PR sorts that out. I'll merge when CI passes. Since there are literally no changes to functionality, I won't bother changing the patch version.

edit: this isn't urgent, so I'm going to try and figure out how to sort coverage out before merging this. I imagine we'll be doing more GPU stuff in the comparatively near future, so now is a good time to get it sorted.

Copy link

codecov bot commented Sep 13, 2024

Codecov Report

Attention: Patch coverage is 96.42857% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
ext/MooncakeCUDAExt.jl 96.00% 1 Missing ⚠️
Files with missing lines Coverage Δ
ext/MooncakeJETExt.jl 100.00% <100.00%> (ø)
ext/MooncakeLogDensityProblemsADExt.jl 83.33% <ø> (ø)
src/test_utils.jl 91.98% <100.00%> (ø)
ext/MooncakeCUDAExt.jl 96.00% <96.00%> (+96.00%) ⬆️

Copy link
Contributor

github-actions bot commented Sep 13, 2024

Performance Ratio:
Ratio of time to compute gradient and time to compute function.
Warning: results are very approximate! See here for more context.

┌────────────────────────────┬──────────┬─────────┬─────────────┬─────────┐
│                      Label │ Mooncake │  Zygote │ ReverseDiff │  Enzyme │
│                     String │   String │  String │      String │  String │
├────────────────────────────┼──────────┼─────────┼─────────────┼─────────┤
│                   sum_1000 │    120.0 │    1.11 │        6.11 │    2.11 │
│                  _sum_1000 │     7.88 │  1380.0 │        34.3 │   0.106 │
│               sum_sin_1000 │     2.54 │     1.6 │        10.9 │    1.01 │
│              _sum_sin_1000 │     3.19 │   332.0 │        17.8 │    1.51 │
│                   kron_sum │     58.1 │     9.8 │       189.0 │    11.3 │
│              kron_view_sum │     76.1 │    10.5 │       217.0 │    10.9 │
│      naive_map_sin_cos_exp │     3.16 │ missing │        9.84 │    2.81 │
│            map_sin_cos_exp │     4.69 │    1.77 │        7.59 │    3.45 │
│      broadcast_sin_cos_exp │     4.42 │    2.58 │        1.71 │    2.87 │
│                 simple_mlp │     8.64 │    3.29 │        12.3 │    3.21 │
│                     gp_lml │     14.7 │    4.42 │     missing │ missing │
│ turing_broadcast_benchmark │     7.24 │ missing │        31.7 │ missing │
│         large_single_block │      3.9 │  4140.0 │        29.8 │    2.24 │
└────────────────────────────┴──────────┴─────────┴─────────────┴─────────┘

@willtebbutt
Copy link
Member Author

@maleadt any idea how I can get make my codecov token available in buildkite? I don't think I had codecov turned on when you installed buildkite in this org, so maybe there's something else that we need to do to enable coverage?

@maleadt
Copy link

maleadt commented Sep 14, 2024

You need to add the token as an encrypted secret to the pipeline: https://github.com/JuliaGPU/buildkite?tab=readme-ov-file#using-secrets

.buildkite/pipeline.yml Outdated Show resolved Hide resolved
.buildkite/pipeline.yml Outdated Show resolved Hide resolved
@willtebbutt
Copy link
Member Author

Thanks for the pointer @maleadt -- I've updated .pipeline.yml but it doesn't appear to be working as expected. Can you see anything that I'm doing obviously wrong?

@maleadt
Copy link

maleadt commented Sep 15, 2024

Our logic to determine if this is a third-party PR seems to fail: https://github.com/JuliaGPU/buildkite/blob/0dbc7b3422d53d770b8e26d79cbada81a0e6d5ce/image/hooks/environment#L5-L8

BUILDKITE_PULL_REQUEST_REPO="https://github.com/compintell/Tapir.jl.git"
BUILDKITE_REPO="https://github.com/withbayes/Tapir.jl.git"

Was this repository renamed?

@willtebbutt
Copy link
Member Author

willtebbutt commented Sep 16, 2024

Ah, yes -- the org was renamed from withbayes to compintell

@maleadt
Copy link

maleadt commented Sep 16, 2024

I updated the repository, and now it does successfully decrypt the token:

2024-09-16 09:49:28 CEST	🔑 Decrypting secrets
2024-09-16 09:49:28 CEST	Found secret for CODECOV_TOKEN
2024-09-16 09:49:28 CEST	# CODECOV_TOKEN added

@willtebbutt
Copy link
Member Author

Thanks for the assistance with this @maleadt -- strangely it doesn't appear to be the case that the result of running the buildkite CI is showing up with the other CI, and is no longer showing up as one of the CI runs at the bottom of this page. Any idea what might be going on?

(Thanks so much for your help with this. I really appreciate it!)

@maleadt
Copy link

maleadt commented Sep 16, 2024

strangely it doesn't appear to be the case that the result of running the buildkite CI is showing up with the other CI, and is no longer showing up as one of the CI runs at the bottom of this page. Any idea what might be going on?

image

I assume that with the repository transfer, it's now part of an org that doesn't have the buildkite app yet. So you're back to step one in: https://github.com/JuliaGPU/buildkite?tab=readme-ov-file#steps-for-administrators

If you can make me temporarily an admin to the org, I can add the integration again.

@willtebbutt
Copy link
Member Author

@yebai could you please make @maleadt an owner of the org briefly, so that this can be resolved?

@yebai
Copy link
Contributor

yebai commented Sep 16, 2024

Done.

@maleadt
Copy link

maleadt commented Sep 16, 2024

Thanks, you can remove me from the org.

The integration was set-up already, so I'm not really sure what's up here. I'll take another look, and if not I'll remove and re-create the pipeline if that's OK.

@maleadt
Copy link

maleadt commented Sep 16, 2024

Turns out the Buildkite integration was still tied to the previous org name, so had to be reinstalled. I've also re-created the pipeline; please push another commit, it should hopefully all work now.

@willtebbutt
Copy link
Member Author

Hmmm it now seems to be the case that the tests are running, codecov is doing something during buildkite's run, but the results don't seem to contribute the the coverage (it's still sitting at zero -- see here.

@maleadt does everything look okay on your side of things now? I don't seem to be able to look at the complete CI logs on buildkite, so I'm not sure if there any some warnings I'm missing, or something like that.

@maleadt
Copy link

maleadt commented Sep 17, 2024

2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_folder: Searching src for .jl files...
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/Tapir.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/Tapir.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/chain_rules_macro.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/chain_rules_macro.jl.3350233.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/chain_rules_macro.jl.3351887.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/codual.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/codual.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/fwds_rvs_data.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/fwds_rvs_data.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interface.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/interface.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_folder: Searching src/interpreter for .jl files...
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interpreter/abstract_interpretation.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/abstract_interpretation.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/abstract_interpretation.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interpreter/bbcode.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/bbcode.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interpreter/contexts.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/contexts.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/contexts.jl.3351887.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/contexts.jl.3352323.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/contexts.jl.3353543.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/contexts.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interpreter/ir_normalisation.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/ir_normalisation.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interpreter/ir_utils.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/ir_utils.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interpreter/s2s_reverse_mode_ad.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/s2s_reverse_mode_ad.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/interpreter/zero_like_rdata.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/interpreter/zero_like_rdata.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_folder: Searching src/rrules for .jl files...
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/avoiding_non_differentiable_code.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/avoiding_non_differentiable_code.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/blas.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/rrules/blas.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/builtins.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/rrules/builtins.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/fastmath.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/fastmath.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/foreigncall.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/foreigncall.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/iddict.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/iddict.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/lapack.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/lapack.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/low_level_maths.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/low_level_maths.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/misc.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/misc.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/new.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/new.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/rrules/tasks.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/rrules/tasks.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/safe_mode.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/safe_mode.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/stack.jl
2024-09-16 13:46:55 CEST	┌ Info: CoverageTools.process_cov: Coverage file(s) for src/stack.jl do not exist.
2024-09-16 13:46:55 CEST	└ Assuming file has no coverage.
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/tangents.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/tangents.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/test_utils.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/test_utils.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_file: Detecting coverage for src/utils.jl
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/utils.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: CoverageTools.process_cov: processing src/utils.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/Tapir.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/chain_rules_macro.jl.3350233.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/chain_rules_macro.jl.3351887.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/codual.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/fwds_rvs_data.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/abstract_interpretation.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/abstract_interpretation.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/bbcode.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/contexts.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/contexts.jl.3351887.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/contexts.jl.3352323.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/contexts.jl.3353543.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/contexts.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/ir_normalisation.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/ir_utils.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/s2s_reverse_mode_ad.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/interpreter/zero_like_rdata.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/rrules/blas.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/rrules/builtins.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/safe_mode.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/tangents.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/test_utils.jl.3357948.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/utils.jl.3349178.cov
2024-09-16 13:46:55 CEST	[ Info: Removing src/utils.jl.3357948.cov
2024-09-16 13:46:56 CEST	[ Info: Submitting data to Codecov...

This does not report coverage for ext/, because the dirs argument defaults to only src: https://github.com/JuliaCI/julia-coverage-buildkite-plugin/blob/e16e9ebce8bdc3f485317938edbbc33dd5f17356/hooks/post-command#L7-L16

@willtebbutt willtebbutt reopened this Sep 19, 2024
@willtebbutt
Copy link
Member Author

willtebbutt commented Sep 19, 2024

@maleadt seemingly this still isn't working quite correctly -- the coverage for ext/MooncakeCUDAExt.jl remains at 0% (other coverage changes are there because I've had to comment out some code to get CI to pass during the name change process).

Some things to note:

  1. I've run coverage locally, and I'm seeing around 85% coverage of ext/MooncakeCUDAExt.jl,
  2. the package name has now changed to Mooncake.jl, but decryption of the secret seems to be working fine,
  3. the coverage for other items in the ext directory seem to be fine, and are handled in my .github/workflows/CI.yml runner, which suggests that the problem is specific to the buildkite script, and
  4. The .buildkite/pipeline.yml runner has been updated to ask for coverage of both src and ext.

Is there anything else obvious that you can think of that might be causing the lack of coverage?

@maleadt
Copy link

maleadt commented Sep 19, 2024

Coverage info is being detected and submitted:

2024-09-19 12:24:34 CEST	[ Info: CoverageTools.process_file: Detecting coverage for ext/MooncakeCUDAExt.jl
2024-09-19 12:24:34 CEST	[ Info: CoverageTools.process_cov: processing ext/MooncakeCUDAExt.jl.1970102.cov

Not sure what the issue is. Maybe you could try adding some buildite-agent commands to upload the coverage files from CI for local inspection.

@willtebbutt
Copy link
Member Author

willtebbutt commented Sep 19, 2024

Thanks for taking a look.

The total number of files being uploaded to codecov is 10 -- see here for the latest commit. Although it looks like something has changed in the latest run, because it's now refusing to decrypt the secrets again (apologies for if the package rename messed this up).

Either way, I agree with you that it looks like somehow the coverage report is getting lost in transit between the buildkite runner and wherever codecov hosts the results.

edit: I'll take a look at the buildkite-agent commands, as you suggest.

@maleadt
Copy link

maleadt commented Sep 19, 2024

Although it looks like something has changed in the latest run, because it's now refusing to decrypt the secrets again (apologies for if the package rename messed this up).

Yes, it did. I'll probably have to recreate the pipeline again.

@willtebbutt
Copy link
Member Author

@maleadt when you get a minute, would you mind recreating the pipeline? I'm keen to start tackling this again.

@maleadt
Copy link

maleadt commented Oct 3, 2024

I did that right after my previous comment, and Buildkite reveals that the Codecov secret is successfully decrypted. Or is there anything else missing?

@willtebbutt
Copy link
Member Author

🤦 sorry -- I was looking at the old tapir-dot-jl run, rather than Mooncake. Thanks for sorting that!

@willtebbutt
Copy link
Member Author

Hurrah! We have non-zero coverage from the buildkite run. I can now resolve the low coverage number, and then merge this. Thanks again for your assistance @maleadt

@willtebbutt
Copy link
Member Author

Only change since CI last passed is a patch bump, so I'm going to merge.

@willtebbutt willtebbutt merged commit 02fc1f5 into main Oct 4, 2024
13 of 15 checks passed
@willtebbutt willtebbutt deleted the wct/extension-formatting branch October 4, 2024 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants