PDK should support integration testing

[scotje]

From @jcbollinger on April 18, 2018 16:48

PDK seems to be tightly focused on supporting the development of individual modules, and it is very useful in that area, but individual modules are not end of the story. Site configurations are built of many modules employed together. It is combinatorically difficult to test modules against every constellation of supported versions of all of their dependencies, and in practice, it is not done. It is even more difficult to anticipate all the forms and and combinations of site data that might be presented to modules. PDK does not help with those things, and it cannot reasonably be expected to do.

But what PDK could and should do is support testing against the specific constellation of modules, site manifest, and site data that a user happens to want to deploy at their particular site. From an rspec-puppet perspective, I picture this being set up as a framework for running a collection of host tests against (a clone of) the site's actual whole configuration as an integrated unit.

Copied from original issue: #481

[bmjen]

Sounds like this feature is related to puppetlabs/pdk-planning#10 and can be addressed by the planned control-repo support.

[jcbollinger]

Could you elaborate on that a bit, @bmjen ? I'm not devoting much attention to the control-repo stuff, so I'm having trouble evaluating your assertion. I am at least a little dubious, however, at least as to PDK control-repo support providing for integration testing for sites that don't use a control repo. I also want to be sure to distinguish between stipulating that a particular collection of module versions work properly together and testing that they do. The latter is what this feature request is about.

[cdenneen]

Even though the site/profile and site/role are technically modules I would like to see the testing be moved to the control-repo itself. spec/classes/{profile,role}/ as needed. Allowing to test how these "modules" come together with the assets in .fixtures.yml or even better Puppetfile would make this a lot cleaner, catch duplicate declarations, circular dependencies, etc and also allow for testing with litmus for acceptance testing.