Download upstream docs for dynamically bridged provider

[guineveresaenger]

This pull request adds logic that enables us to generate resource docs for a dynamically bridged provider using the upstream dependency.

Usage for remote providers:
pulumi package get-schema terraform-provider <registry address> <version> fullDocs=<true|false>.

Adds a parameterized arg field to pulumi package get-schema terraform-provider <registry address> <version> called fullDocs, which when set to true will instruct the bridge to git clone --depth 1 -b <version> <terraform provider github repo> <local dir for dynamic docs>. This allows us to keep the exact same docs logic we have established. The shallow clone targets the exact doc version we need. (*)

For this, we infer the github repo from the OpenTofu org name. It is based on the assumption that registry.opentofu.org/org/foo is based on a provider that lives at github.com/org/terraform-provider-foo. OpenTofu says their protocol follows that of the HashiCorp Terraform Registry which requires an org/user and a provider name of the format terraform-provider-foo and we have historical evidence that GitHub is the most commonly used source host for terraform providers. See also opentofu/registry#1337.

Usage for local providers:
pulumi package get-schema terraform-provider <local-path> upstreamRepoPath=<localPath>. Here, the local docs path gets read directly into upstreamRepoPath.

The one thing that is perhaps missing here is to expand the remote args to allow to take a source repo as an additional argument. However, our internal use case is primarily one of automation, so this is not an immediate need.

(*)Cloning docs for the AWS terraform provider at v5.70.0 has the following time performance:

git clone --depth 1 -b v5.70.0  awsDir  
2.06s user 
1.62s system 
34% cpu 
10.713 total

Fixes #2607

[iwahbe]

I left a couple of nits, but I have 1 more major piece of design feedback:

Instead of introducing XLoadUpstreamRepoForDocs and then threading the logic into the docs generator, could we download the docs during the GetSchema call directly in dynamic/main.go and then just set UpstreamRepoPath? That should accomplish the same effect without adding more logic to the docs generator.

[codecov]

Codecov Report

Attention: Patch coverage is 47.05882% with 18 lines in your changes missing coverage. Please review.

Project coverage is 69.53%. Comparing base (143b3ee) to head (5df05c7).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/tfgen/generate.go	18.75%	13 Missing ⚠️
pkg/pf/tfgen/main.go	0.00%	4 Missing ⚠️
pkg/tfgen/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2664   +/-   ##
=======================================
  Coverage   69.53%   69.53%           
=======================================
  Files         301      301           
  Lines       38646    38655    +9     
=======================================
+ Hits        26871    26878    +7     
- Misses      10254    10257    +3     
+ Partials     1521     1520    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[guineveresaenger]

This may not be the way we want to do this, but I changed the function signature of Generate() and UnstableGenerateFromSchema to return a package schema so that we can access it from an outside library.

Alternatives:

• Implementing a separate MakeSchemaWithExamples(generator) function in generate.go that takes a pre-created Generator as an argument and returns a package schema for consumption by dynamic
• Generating the "lightweight" schema first, as before, and then making a separate call to UnstableGenerateFromSchema.

Unfortunately, adding extra options to the GeneratorOpts only gets us halfway - we must call Generate() or its parts somewhere in dynamic.

I've gone with changing the function signature to avoid writing yet another generator function with a narrowly scoped purpose, but can definitely be convinced otherwise.

[iwahbe]

There are currently two argument styles added in this PR:

pulumi package get-schema terraform-provider ./terraform-provider-local upstreamRepoPath=./local
pulumi package get-schema terraform-provider my/provider fullDocs

One is positional, but the other is flag based. Can we standardize? Personally, I'd prefer flags. That would make it:

pulumi package get-schema terraform-provider ./terraform-provider-local upstreamRepoPath=./local
pulumi package get-schema terraform-provider my/provider docs=full

[iwahbe]

Do we need this bump, or is it incidental?

[guineveresaenger]

We need this bump, otherwise the test won't pass.

[iwahbe]

Why are we defining a local struct only to instantiate one copy? Same with the subtest.

[guineveresaenger]

Why are we defining a local struct only to instantiate one copy?

• Test information at a glance
• More easily add subtests in the future if needed
• Allows us to run individual tests by name when using an integrated IDE test framework
• Makes the name of subtest far more obvious for running it in the terminal

Is there a downside here?

[iwahbe]

Verbosity and the chance of rot. It's an abstraction we aren't using.

[iwahbe]

Let's be consistent with upstreamRepoPath for local providers.

Suggested change

	return Args{}, fmt.Errorf("expected to be parameterized by 1-3 arguments: <name> [version] [fullDocs]")
	return Args{}, fmt.Errorf("expected to be parameterized by 1-3 arguments: <name> [version] [fullDocs=<true|false>]")

[iwahbe]

This is a regression.

[pulumi-bot]

This PR has been shipped in release v3.98.0.