[Bug]: Seeing differences on local vs CI runs for the formatters

[Geethree]

What happened?

As the title states, starting to pull my hair our locally. Currently I have my format targets configured like..

load("@aspect_rules_lint//format:defs.bzl", "format_multirun", "format_test")

format_multirun(
    name = "format",
    go = "@multitool//tools/gofumpt",
    protocol_buffer = "@rules_buf_toolchains//:buf",
    starlark = "@buildifier_prebuilt//:buildifier",
    terraform = "@multitool//tools/terraform",
    yaml = "@multitool//tools/yamlfmt",
)

format_test(
    name = "test",
    size = "small",
    go = "@multitool//tools/gofumpt",
    no_sandbox = True,
    protocol_buffer = "@rules_buf_toolchains//:buf",
    starlark = "@buildifier_prebuilt//:buildifier",
    workspace = "//:.gitlab-ci.yml",
    yaml = "@multitool//tools/yamlfmt",
)

Locally on ubuntu 22.04 and on MacOS a current commit fails as such:

spencer@gspencer-mlt autobots % bazel test tools/format/... --test_output=summary
INFO: Analyzed 16 targets (0 packages loaded, 0 targets configured).
FAIL: //tools/format:test_Starlark_with_buildifier (see /private/var/tmp/_bazel_gspencer/26b182f6d79bbfc06c198e7527c526e0/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/tools/format/test_Starlark_with_buildifier/test.log)
FAIL: //tools/format:test_YAML_with_yamlfmt (see /private/var/tmp/_bazel_gspencer/26b182f6d79bbfc06c198e7527c526e0/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/tools/format/test_YAML_with_yamlfmt/test.log)
INFO: Found 12 targets and 4 test targets...
INFO: Elapsed time: 0.312s, Critical Path: 0.22s
INFO: 5 processes: 8 local.
INFO: Build completed, 2 tests FAILED, 5 total actions
//tools/format:test_Go_with_gofmt                                        PASSED in 0.1s
//tools/format:test_Protocol_Buffer_with_buf                             PASSED in 0.1s
//tools/format:test_Starlark_with_buildifier                             FAILED in 0.1s
  /private/var/tmp/_bazel_gspencer/26b182f6d79bbfc06c198e7527c526e0/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/tools/format/test_Starlark_with_buildifier/test.log
//tools/format:test_YAML_with_yamlfmt                                    FAILED in 0.1s
  /private/var/tmp/_bazel_gspencer/26b182f6d79bbfc06c198e7527c526e0/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/tools/format/test_YAML_with_yamlfmt/test.log

Executed 4 out of 4 tests: 2 tests pass and 2 fail locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

Summary here as it is just formatting yaml files and a BUILD file. *

In CI this isn't the case.. as all targets pass

Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Analyzing: 16 targets (1 packages loaded, 0 targets configured)
Analyzing: 16 targets (1 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Analyzing: 16 targets (16 packages loaded, 1217 targets configured)
[1 / 1] checking cached actions
Analyzing: 16 targets (21 packages loaded, 1257 targets configured)
[1 / 17] checking cached actions
Analyzing: 16 targets (23 packages loaded, 3545 targets configured)
[64 / 69] 4 / 4 tests; [Prepa] Creating symlink bazel-out/k8-fastbuild/bin/external/rules_multitool~~multitool~multitool/tools/terraform/terraform ... (2 actions, 0 running)
INFO: Analyzed 16 targets (23 packages loaded, 3545 targets configured).
INFO: Found 12 targets and 4 test targets...
INFO: Elapsed time: 5.028s, Critical Path: 0.74s
INFO: 76 processes: 68 internal, 8 local.
INFO: Build completed successfully, 76 total actions
//tools/format:test_Go_with_gofmt                                        PASSED in 0.3s
//tools/format:test_Protocol_Buffer_with_buf                             PASSED in 0.1s
//tools/format:test_Starlark_with_buildifier                             PASSED in 0.5s
//tools/format:test_YAML_with_yamlfmt                                    PASSED in 0.1s
Executed 4 out of 4 tests: 4 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

CI here is running in gitlab inside of the gcr.io/bazel-public/bazel:7.1.0 container image.

I'm not really sure of how to go about debugging this..

Version

bazel version
Bazelisk version: development
Build label: 7.1.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Thu Mar 21 18:08:59 2024 (1711044539)
Build timestamp: 1711044539
Build timestamp as int: 1711044539

bazel run @buildifier_prebuilt//:buildifier -- -version
buildifier version: 6.4.0
buildifier scm revision: 433ea8554e82832e4fa4bdd530ca471564071511

bazel run @multitool//tools/yamlfmt -- -version
yamlfmt 0.13.0 (dd8547d)

How to reproduce

No response

Any other information?

No response

[alexeagle]

As documented on https://github.com/aspect-build/rules_lint/blob/main/docs/format.md#format_test - the format_test rule is not recommended because it's impossible for it to be hermetic when you've set no_sandbox = True. Maybe on your CI you're getting a remote cache hit so it's not actually running the formatter on the files?
I'd recommend having a separate CI step to assert the changed files are formatted. Our https://aspect.build/workflows product includes this.