Parse deferred templates twice

Currently it is not possible to have a template file.epp ```puppet <%- | Stdlib::Port $port, String[1] $password, | %> port <%= $port %> password <%= $password %> ``` and run ```puppet file{'/tmp/junk': content => stdlib::deferrable_epp('module/file.epp', { 'port' => '1234', pass => Deferred('secrets::get',['mysecret'])}), } ``` since the deferred template substitution will fail: ``` Error: Failed to apply catalog: Evaluation Error: Resource type not found: Stdlib::Port (file: inlined-epp-text, line: 2, column: 3) ``` due to Stdlib::Port not being available on the agent node. This change now parses the EPP twice. The first pass will reduce the template to: ```puppet port = 1234 password <%= $password %> ``` and this simpler template will be passed in deferred mode. Note the original template type for password must accept the intermediate generated value of `<%= $password %>` which is typically case for a secret password.
puppetlabs · Apr 15, 2024 · 60fd443 · 60fd443
1 parent d871c4d
commit 60fd443
Show file tree

Hide file tree

Showing 4 changed files with 188 additions and 15 deletions.
diff --git a/REFERENCE.md b/REFERENCE.md
@@ -3173,18 +3173,38 @@ Type: Puppet Language
 This function returns either a rendered template or a deferred function to render at runtime.
 If any of the values in the variables hash are deferred, then the template will be deferred.
 
-Note: this function requires all parameters to be explicitly passed in. It cannot expect to
-use facts, class variables, and other variables in scope. This is because when deferred, we
-have to explicitly pass the entire scope to the client.
-
-#### `stdlib::deferrable_epp(String $template, Hash $variables)`
+Note: In the case where at least some of the values are deferred and preparse is `true` the template
+      is parsed twice:
+ The first parse will evalute any parameters in the template that do not have deferred values.
+ The second parse will run deferred and evaluate only the remaining deferred parameters. Consequently
+ any parameters to be deferred must accept a String[1] in original template so as to accept the value
+ "<%= $variable_with_deferred_value %>" on the first parse.
+
+ @param template template location - identical to epp function template location.
+ @param variables parameters to pass into the template - some of which may have deferred values.
+ @param preparse
+     If `true` the epp template will be parsed twice, once normally and then a second time deferred.
+     It may be nescessary to set `preparse` `false` when deferred values are somethig other than
+     a string
+
+#### `stdlib::deferrable_epp(String $template, Hash $variables, Boolean $preparse = true)`
 
 This function returns either a rendered template or a deferred function to render at runtime.
 If any of the values in the variables hash are deferred, then the template will be deferred.
 
-Note: this function requires all parameters to be explicitly passed in. It cannot expect to
-use facts, class variables, and other variables in scope. This is because when deferred, we
-have to explicitly pass the entire scope to the client.
+Note: In the case where at least some of the values are deferred and preparse is `true` the template
+      is parsed twice:
+ The first parse will evalute any parameters in the template that do not have deferred values.
+ The second parse will run deferred and evaluate only the remaining deferred parameters. Consequently
+ any parameters to be deferred must accept a String[1] in original template so as to accept the value
+ "<%= $variable_with_deferred_value %>" on the first parse.
+
+ @param template template location - identical to epp function template location.
+ @param variables parameters to pass into the template - some of which may have deferred values.
+ @param preparse
+     If `true` the epp template will be parsed twice, once normally and then a second time deferred.
+     It may be nescessary to set `preparse` `false` when deferred values are somethig other than
+     a string
 
 Returns: `Variant[String, Sensitive[String], Deferred]`
 
@@ -3200,6 +3220,12 @@ Data type: `Hash`
 
 
 
+##### `preparse`
+
+Data type: `Boolean`
+
+
+
 ### <a name="stdlib--end_with"></a>`stdlib::end_with`
 
 Type: Ruby 4.x API

diff --git a/functions/deferrable_epp.pp b/functions/deferrable_epp.pp
@@ -1,15 +1,39 @@
 # This function returns either a rendered template or a deferred function to render at runtime.
 # If any of the values in the variables hash are deferred, then the template will be deferred.
 #
-# Note: this function requires all parameters to be explicitly passed in. It cannot expect to
-# use facts, class variables, and other variables in scope. This is because when deferred, we
-# have to explicitly pass the entire scope to the client.
+# Note: In the case where at least some of the values are deferred and preparse is `true` the template
+#       is parsed twice:
+#  The first parse will evalute any parameters in the template that do not have deferred values.
+#  The second parse will run deferred and evaluate only the remaining deferred parameters. Consequently
+#  any parameters to be deferred must accept a String[1] in original template so as to accept the value
+#  "<%= $variable_with_deferred_value %>" on the first parse.
 #
-function stdlib::deferrable_epp(String $template, Hash $variables) >> Variant[String, Sensitive[String], Deferred] {
+#  @param template template location - identical to epp function template location.
+#  @param variables parameters to pass into the template - some of which may have deferred values.
+#  @param preparse
+#      If `true` the epp template will be parsed twice, once normally and then a second time deferred.
+#      It may be nescessary to set `preparse` `false` when deferred values are somethig other than
+#      a string
+#
+function stdlib::deferrable_epp(String $template, Hash $variables, Boolean $preparse = true) >> Variant[String, Sensitive[String], Deferred] {
   if $variables.stdlib::nested_values.any |$value| { $value.is_a(Deferred) } {
+    if $preparse {
+      $_variables_escaped = $variables.map | $_var , $_value | {
+        if $_value.is_a(Deferred) {
+          { $_var => "<%= \$${_var} %>" }
+        } else {
+          { $_var => $_value }
+        }
+      }.reduce | $_memo, $_kv | { $_memo + $_kv }
+
+      $_template = inline_epp(find_template($template).file,$_variables_escaped)
+    } else {
+      $_template = find_template($template).file
+    }
+
     Deferred(
       'inline_epp',
-      [find_template($template).file, $variables],
+      [$_template, $variables],
     )
   }
   else {

diff --git a/spec/acceptance/stdlib_deferrable_epp_spec.rb b/spec/acceptance/stdlib_deferrable_epp_spec.rb
@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+require 'spec_helper_acceptance'
+
+describe 'stdlib::deferable_epp function' do
+  let(:testfile) { (os[:family] == 'windows') ? 'C:\\test.epp' : '/tmp/test.epp' }
+
+  before(:all) do
+    apply_manifest(<<-MANIFEST)
+      $_epp = @(EPP)
+      <%- |
+      Stdlib::Port $port,
+      String[1] $password,
+      | -%>
+      port=<%= $port %>
+      password=<%= $password %>"
+      | EPP
+      $_testfile = $facts['os']['family'] ? {
+        'windows' => 'C:\\test.epp',
+        default => '/tmp/test.epp',
+      }
+
+      file{ $_testfile:
+        ensure  => file,
+        content => $_epp,
+      }
+    MANIFEST
+  end
+
+  before(:each) do
+    rm_testfile = <<-MANIFEST
+      $_testfile = $facts['os']['family'] ? {
+        'windows' => 'C:\\test.epp',
+        default => '/tmp/test.epp',
+      }
+      file { "${_testfile}.rendered":
+        ensure  => absent,
+      }
+    MANIFEST
+    apply_manifest(rm_testfile)
+  end
+
+  context 'with no deferred values' do
+    let(:pp) do
+      <<-MANIFEST
+        $_testfile = $facts['os']['family'] ? {
+          'windows' => 'C:\\test.epp',
+          default => '/tmp/test.epp',
+        }
+
+        file{ "${_testfile}.rendered":
+          ensure  => file,
+          content => stdlib::deferrable_epp(
+            $_testfile,
+            {'port' => 1234, 'password' => 'top_secret'}
+          ),
+        }
+      MANIFEST
+    end
+
+    it 'applies manifest, generates file' do
+      idempotent_apply(pp)
+      expect(file("#{testfile}.rendered")).to be_file
+      expect(file("#{testfile}.rendered").content).to match(%r{port=1234})
+      expect(file("#{testfile}.rendered").content).to match(%r{password=top_secret})
+    end
+  end
+
+  context 'with deferred values' do
+    let(:pp) do
+      <<-MANIFEST
+        $_testfile = $facts['os']['family'] ? {
+          'windows' => 'C:\\test.epp',
+          default => '/tmp/test.epp',
+        }
+
+        file{ "${_testfile}.rendered":
+          ensure  => file,
+          content => stdlib::deferrable_epp(
+            $_testfile,
+            {'port' => 1234, 'password' => Deferred('inline_epp',['<%= $secret_password %>',{'secret_password' => 'so_secret'}])},
+          ),
+        }
+      MANIFEST
+    end
+
+    it 'applies manifest, generates file' do
+      idempotent_apply(pp)
+      expect(file("#{testfile}.rendered")).to be_file
+      expect(file("#{testfile}.rendered").content).to match(%r{port=1234})
+      expect(file("#{testfile}.rendered").content).to match(%r{password=so_secret})
+    end
+  end
+end
diff --git a/spec/functions/stdlib_deferrable_epp_spec.rb b/spec/functions/stdlib_deferrable_epp_spec.rb
@@ -24,8 +24,37 @@
 
     it {
       foo = Puppet::Pops::Types::TypeFactory.deferred.create('join', [1, 2, 3])
-      # This kind_of matcher requires https://github.com/puppetlabs/rspec-puppet/pull/24
-      expect(subject).to run.with_params('mymod/template.epp', { 'foo' => foo }) # .and_return(kind_of Puppet::Pops::Types::PuppetObject)
+      expect(subject).to run.with_params('mymod/template.epp', { 'foo' => foo }).and_return(kind_of(Puppet::Pops::Types::PuppetObject))
+    }
+  end
+
+  context 'defers rendering with mixed deferred and undeferred input' do
+    let(:pre_condition) do
+      <<~END
+        function epp($str, $data) { fail("should not have invoked epp()") }
+        function find_template($str) { return "path" }
+        function file($path) { return "foo: <%= foo %>, bar: <%= bar %>" }
+      END
+    end
+
+    it {
+      foo = Puppet::Pops::Types::TypeFactory.deferred.create('join', [1, 2, 3])
+      expect(subject).to run.with_params('mymod/template.epp', { 'foo' => foo, 'bar' => 'xyz' }).and_return(kind_of(Puppet::Pops::Types::PuppetObject))
+    }
+  end
+
+  context 'defers rendering with mixed deferred and undeferred input and preparse false' do
+    let(:pre_condition) do
+      <<~END
+        function epp($str, $data) { fail("should not have invoked epp()") }
+        function find_template($str) { return "path" }
+        function file($path) { return "foo: <%= foo %>, bar: <%= bar %>" }
+      END
+    end
+
+    it {
+      foo = Puppet::Pops::Types::TypeFactory.deferred.create('join', [1, 2, 3])
+      expect(subject).to run.with_params('mymod/template.epp', { 'foo' => foo, 'bar' => 'xyz' }, false).and_return(kind_of(Puppet::Pops::Types::PuppetObject))
     }
   end
 end