Remote debug heartbeat

[gsantoro]

What does this PR do?

Changes to Tiltfile + dockerfile + manifests to remote debug Heartbeat similarly to how it works already for Metricbeat and Filebeat.

Why is it important?

I found absolutely necessity to be able to remote debug heartbeat and similar beats to investigate bugs

Checklist

None of the below applies since it is a dev tool

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Instructions at #32326

Related issues

• Relates Feature/tilt 2 #32326
• Used by issue Heartbeat fails to add a new monitor if host port differs from service port #34214

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @gsantoro? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-13T13:45:48.353+0000

• Duration: 70 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	25235
Skipped	1954
Total	27189

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[gsantoro]

/test

[cmacknz]

The failure in CI is:

[2023-01-11T09:58:36.531Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]

[2023-01-11T09:58:36.632Z] fatal: detected dubious ownership in repository at '/go/src/github.com/elastic/beats'

[2023-01-11T09:58:36.632Z] To add an exception for this directory, call:

[2023-01-11T09:58:36.632Z] 

[2023-01-11T09:58:36.632Z] 	git config --global --add safe.directory /go/src/github.com/elastic/beats

[2023-01-11T09:58:36.632Z] Error: template: inline:1:128: executing "inline" at <commit>: error calling commit: running "git rev-parse HEAD" failed with exit code 128

[2023-01-11T09:58:36.632Z] failed to expand template '-s -X github.com/elastic/beats/v7/libbeat/version.buildTime={{ date }} -X github.com/elastic/beats/v7/libbeat/version.commit={{ commit }}'

[2023-01-11T09:58:36.632Z] github.com/elastic/beats/v7/dev-tools/mage.expandTemplate

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/dev-tools/mage/common.go:102

[2023-01-11T09:58:36.632Z] github.com/elastic/beats/v7/dev-tools/mage.Expand

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/dev-tools/mage/common.go:58

[2023-01-11T09:58:36.632Z] github.com/elastic/beats/v7/dev-tools/mage.MustExpand

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/dev-tools/mage/common.go:64

[2023-01-11T09:58:36.632Z] github.com/elastic/beats/v7/dev-tools/mage.Build

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/dev-tools/mage/build.go:180

[2023-01-11T09:58:36.632Z] github.com/elastic/beats/v7/dev-tools/mage.GolangCrossBuild

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/dev-tools/mage/build.go:133

[2023-01-11T09:58:36.632Z] github.com/elastic/beats/v7/dev-tools/mage/target/build.GolangCrossBuild

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/dev-tools/mage/target/build/build.go:38

[2023-01-11T09:58:36.632Z] main.main.func35

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/metricbeat/mage_output_file.go:1216

[2023-01-11T09:58:36.632Z] main.main.func12.1

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/src/github.com/elastic/beats/metricbeat/mage_output_file.go:316

[2023-01-11T09:58:36.632Z] runtime.goexit

[2023-01-11T09:58:36.632Z] 	/var/lib/jenkins/workspace/PR-34224-3-18390e45-3611-4430-9588-e80b89f1e8bd/.gvm/versions/go1.18.9.linux.amd64/src/runtime/asm_amd64.s:1571

[2023-01-11T09:58:36.632Z] Error: failed building for darwin/amd64: exit status 

Never seen this, but Stackoverflow suggests the ownership of the new directory might be wrong: https://stackoverflow.com/questions/72978485/git-submodule-update-failed-with-fatal-detected-dubious-ownership-in-repositor

[gsantoro]

@cmacknz I am not sure how to fix this though.
If I understand the error and the stackoverflow I should run that command git config --global --add safe.directory /go/src/github.com/elastic/beats to fix it. But since that's a global variable the change will be reflected on my laptop and not on the CI. I had run that command on my laptop and there is no change to commit to my PR.

I have checked the ownership of the new folder on my machine but I can't see anything wrong with it.

Any other suggestions?

[cmacknz]

@v1v it looks like elastic/golang-crossbuild#232 is hitting us now

[cmacknz]

The build error should be fixed now if you merge/rebase with main.

[gsantoro]

@cmacknz Thanks, I saw the commit to main by chance and this PR seems to be building correctly now. Thanks