chore: vendor more OpenTOFU code

[t0yv0]

Changes:

• dynamic/internal/shim is gone in favor of more files in pkg/vendored/opentofu
• vendored more files from opentofu to accomplish this
• simpler structure of go modules under dynamic/* - two modules removed
• is dynamic/go.mod still needed as a separate module?

I think it fixes #2668 or it should do that because the resulting solution bundles vendored opentofu code that rewrites the generated protobuf files to use the Pulumi bridge specific copy.

Fixes #2668

Concerns:

• do acceptance tests on dynamic bridged providers pass - yes they do
• looks like dynamic still depends on opentofu indirectly, which has a different version from the vendored copy; perhaps benign but may be suspicious - this has been fixed so the code does not depend on opentofu proper yet
• vendored code is patched to remove provisioners; this is likely not yet used in the bridge

[t0yv0]

Hmm this is a little suspect, likely we're not shadow-rewriting all imports in pkg/vendored/opentofu.

[t0yv0]

Removed this. Fully using vendored (renamed) packages. No direct dependency on opentofu.

[t0yv0]

Hmm this is interesting, this approach is not quite right, deep inside debugging TestLoadProvider I get an issue.

Deep inside the call stack, gRPC is unhappy with:

    Message "unknown service tfplugin5_pulumi.Provider"

[t0yv0]

I think I figured that one out, we need to be more careful with how protoc is invoked.

[iwahbe]

I'm ok with vendoring, but I'd like us to re-examine the process by which we vendor. I have two specific questions:

• Can we link the new code in the pkg/vendored/README.md?
• Can we standardize on a single generate.go script (instead of one per vendored directory)?

Once we do this, I don't see any reason to keep the dynamic/go.mod anymore.

[t0yv0]

Streamlined vendor folder layout a bit. Single generate.go now.

[t0yv0]

One reason to keep the module around is scoping this redirect:

	// Use OpenTofu's fork of hcl (copied from OpenTofu).
	github.com/hashicorp/hcl/v2 => github.com/opentofu/hcl/v2 v2.0.0-20240416130056-03228b26f391

I think hashicorp/hcl is used in non-dynamic bridge code already and I wonder if the replacement will cause any unwanted changes there, e.g. does it need to be there. We'll need test covering why it's there.

[t0yv0]

I think I'll postpone removing dynamic/go.mod to a subsequent PR. Focusing on getting tests to run here for now.

[t0yv0]

make: *** [test] Error 1
        	Error Trace:	/Users/runner/work/pulumi-terraform-bridge/pulumi-terraform-bridge/dynamic/internal/shim/run/loader_test.go:42
        	Error:      	Received unexpected error:
        	            	provider registry.opentofu.org/hashicorp/tls 4.0.4 is not available for darwin_arm64
        	Test:       	TestLoadProvider/registry

Surprising because I think I can pass it locally? Perhaps platform specific issues?

[t0yv0]

Fixed an issue with an over-eager patch that disabled code when vendoring. Let us see if this passes now.

[t0yv0]

t.Parallel() is not compatible with t.Setenv

[VenelinMartinov]

I think it's fine to remove t.Parallel from tests which need t.Setenv but you need to ignore the linter with //nolint:paralleltest

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.64%. Comparing base (917f85c) to head (456ab96).
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2718   +/-   ##
=======================================
  Coverage   69.64%   69.64%           
=======================================
  Files         301      301           
  Lines       38726    38726           
=======================================
  Hits        26969    26969           
  Misses      10240    10240           
  Partials     1517     1517           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[t0yv0]

The linter rule you mention @VenelinMartinov is not enabled as yet.

[t0yv0]

Unfortunately Windows is not happy:

=== NAME  TestLoadProvider
    testing.go:1231: TempDir RemoveAll cleanup: remove C:\Users\RUNNER~1\AppData\Local\Temp\TestLoadProvider3788457442\001\registry.opentofu.org\hashicorp\tls\4.0.4\windows_amd64\terraform-provider-tls.exe: Access is denied.
--- FAIL: TestLoadProvider (4.84s)

[t0yv0]

Ah no the linter is there. Just it does not fire under make lint for some reason.

Error: dynamic/internal/shim/run/loader_test.go:35:1: Function TestLoadProvider missing the call to method parallel (paralleltest)
func TestLoadProvider(t *testing.T) {
^

[VenelinMartinov]

Could it be a difference in local tool versions? Mikhail hit this recently. AFAIK CI runs make lint

[t0yv0]

Worked this morning. Fixing.