feat: flag to raise on missing file

The new `salt_tower.raise_on_missing_files` config flag can be enabled
to raise an error whenever a file to be included cannot be found.

CHANGELOG.md

@@ -1,85 +1,122 @@
 # Changelog
+
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+
+- Enabling `salt_tower.raise_on_missing_files` will raise an error when an included file does not exist
+
 ## [1.8.2] - 2021-12-03
+
 ### Fixed
+
 - Trying to release a versioned documentation
 
 ## [1.8.1] - 2021-12-03
+
 ### Fixed
+
 - Trying to release a versioned documentation
 
 ## [1.8.0] - 2021-12-03
+
 ### Added
+
 - [Alternative directory include mode](https://jgraichen.github.io/salt-tower/v1.8.2/configuration/#include_directory_mode)
 
 ## [1.7.0] - 2021-04-03
+
 ### Fixed
+
 - Python package loader for Salt 3003
 - Calling conventions to have more salt renderer lookup files relative to current template (e.g. JINJA `include`) (#11)
 
 ### Added
+
 - Tower compatibility with Salt 3003 (#18)
 - Experimental flag to have some renderers (e.g. JINJA) lookup files in `pillar_roots` (#11)
 
 ## [1.6.0] - 2021-03-09
+
 ### Added
+
 - Add require option to tower.get
 - Experimental: Support custom context in template injected `render` function
 - Deep scrubbing of merge strategies (`__`) from merged pillar data
 
 ## [1.5.2] - 2021-02-10
+
 ### Fixed
+
 - Compatibility issues with Salt 2017
 
 ## [1.5.1] - 2021-02-10
+
 ### Fixed
+
 - Missing symlink to new filter renderer from `_renderers` directory for gitfs deployments
 
 ## [1.5.0] - 2021-02-10
+
 ### Added
+
 - `filter` renderer returning only a matching subset from a dataset (e.g. YAML)
 
 ## [1.4.0] - 2021-01-29
+
 ### Added
+
 - Support for Python 3.8 and 3.9, and Salt 3001 and 3002
 
 ## [1.3.1] - 2020-05-11
+
 ### Fixed
+
 - Fix mutation of context variables when rendering multiple files
 
 ## [1.3.0] - 2020-05-11
+
 ### Added
+
 - Added `merge-last`, `merge-first`, `remove` and `overwrite` merge strategies for dictionaries
 - Improve logging source and exception for rendering errors (#10)
 - Extended `render` context function to support relative lookup
 
 ### Fixed
+
 - Fix `render` context function to use correct `basedir` variable
 - Add `basedir` variable when rendering top file
 
 ## [1.2.0] - 2020-01-31
+
 ### Added
+
 - Provide `basedir` context variable for absolute includes, e.g. `!include {{ basedir }}/files/config.ini`
 - Support loading pillar and renderer modules via gitfs and sync runner
 
 ## [1.1.0] - 2019-03-10
+
 ### Added
+
 - Pillar include supports relative includes using only `./` and `../`
 - Compatible with new salt 2019.2
 
 ## 1.0.0 - 2018-11-16
+
 ### Added
+
 - First version of ext pillar
   - First version of ext pillar
   - Yamlet and Text renderer
 
-[Unreleased]: https://github.com/jgraichen/salt-tower/compare/v1.8.0...HEAD
+[Unreleased]: https://github.com/jgraichen/salt-tower/compare/v1.8.2...HEAD
+[1.8.2]: https://github.com/jgraichen/salt-tower/compare/v1.8.1...v1.8.2
+[1.8.1]: https://github.com/jgraichen/salt-tower/compare/v1.8.0...v1.8.1
 [1.8.0]: https://github.com/jgraichen/salt-tower/compare/v1.7.0...v1.8.0
 [1.7.0]: https://github.com/jgraichen/salt-tower/compare/v1.6.0...v1.7.0
 [1.6.0]: https://github.com/jgraichen/salt-tower/compare/v1.5.2...v1.6.0

docs/configuration.md

@@ -87,9 +87,13 @@ The `all-sls` mode can be quite useful if you like to split up e.g. your roles,
 
     The servers will now a all share the same configuration, but will have different cipher settings depending on their operating system version.
 
+### salt_tower.raise_on_missing_files
+
+Enabling the `salt_tower.raise_on_missing_files` flag will have salt tower raise an error if any included file from `tower.sls` or any `include` list cannot be found. If the include statement is a pattern, that does not match any files, an error will be raised too.
+
 ### salt_tower.unstable_enable_saltenv
 
-The `salt_tower.unstable_enable_saltenv` flags modifies some flags passed to the salt rendering pipeline. The salt renderers behave differently if they think they are rendering a state file or a pillar file. Including a file, such as with JINJA `import_yaml` will look up files in either `file_roots` or `pillar_roots`. Unfortunately, this behavior cannot be customized by plugins.
+The `salt_tower.unstable_enable_saltenv` flag modifies some options passed to the salt rendering pipeline. The salt renderers behave differently if they think they are rendering a state file or a pillar file. Including a file, such as with JINJA `import_yaml` will look up files in either `file_roots` or `pillar_roots`. Unfortunately, this behavior cannot be customized by plugins.
 
 By setting `salt_tower.unstable_enable_saltenv` to `True`, Salt Tower will pass additional flags to the rendering pipeline, indicating that pillars are rendered. Engines, such as JINJA, will now look up files in `pillar_roots`. This can work well, when you Salt Tower base directory is the same as the pillar root directory.
 

pyproject.toml

@@ -47,6 +47,7 @@ dev = [
     "pylint==2.13.8",
     "pytest~=7.0.0",
     "tox~=3.23.0",
+    "jinja2<3.0", # not support by salt
 ]
 
 [tool.pytest.ini_options]

salt_tower/pillar/tower.py

@@ -10,9 +10,9 @@
 import logging
 import os
 import string
-
+from copyreg import constructor
 from glob import glob
-from typing import Any, Dict, TYPE_CHECKING
+from typing import TYPE_CHECKING, Any, Dict
 
 import salt.loader
 import salt.minion
@@ -224,7 +224,12 @@ def lookup(self, item, base=None, cwd=None):
         return []
 
     def load(self, item, base=None, cwd=None):
-        for file in self.lookup(item, base, cwd):
+        matches = self.lookup(item, base, cwd)
+
+        if not matches and __opts__.get("salt_tower.raise_on_missing_files"):
+            raise FileNotFoundError(f"File not found: {item}")
+
+        for file in matches:
             self._load_file(file, base)
 
     def _load_file(self, file, base=None):

salt_tower/renderers/filter.py

@@ -1,16 +1,16 @@
 # -*- coding: utf-8 -*-
 # pylint: disable=missing-docstring
 """
-The ``filter`` renderers takes a dataset and only returns the first matching
-value. It can match globs on grains and pillar values.
+The ``filter`` renderers takes a dataset and only returns the first
+matching value. It can match globs on grains and pillar values.
 
-The input must be a parsed dictionary, for example from the YAML renderer. The
-first key is uses as a pattern to match the value from grains or pillar. Shell
-like globs are supported as the ``fnmatch`` function is used to check the
-patterns.
+The input must be a parsed dictionary, for example from the YAML
+renderer. The first key is uses as a pattern to match the value from
+grains or pillar. Shell like globs are supported as the ``fnmatch``
+function is used to check the patterns.
 
-The ``default`` option can provide a string used as the value if the grain or
-pillar key does not exist.
+The ``default`` option can provide a string used as the value if the
+grain or pillar key does not exist.
 
 Example: Default matching uses ``minion_id``:
 
@@ -46,7 +46,6 @@
 
 import logging
 import shlex
-
 from fnmatch import fnmatch
 
 try:

salt_tower/renderers/text.py

@@ -3,8 +3,8 @@
 """
 A text renderer for processing and loading text blobs.
 
-It is mainly developed to be used with file loaded in tower pillars and to be
-used with the yamlet renderers include tag:
+It is mainly developed to be used with file loaded in tower pillars and
+to be used with the yamlet renderers include tag:
 
 .. code-block:: yaml
 
@@ -18,8 +18,8 @@
     The pillar key content.
 
 
-The following example will render a text file and
-strip whitespace from both ends:
+The following example will render a text file and strip whitespace from
+both ends:
 
 .. code-block:: text
 
@@ -28,7 +28,8 @@
     A short text message
 
 
-The renderer is further able to wrap the text blob into a nested dictionary:
+The renderer is further able to wrap the text blob into a nested
+dictionary:
 
 .. code-block:: text
 

salt_tower/renderers/yamlet.py

@@ -17,14 +17,12 @@
 import io
 import os
 
-from yaml.constructor import ConstructorError
-from yaml.nodes import ScalarNode, MappingNode
-
 import salt.loader
 import salt.template
-
-from salt.utils.yamlloader import SaltYamlSafeLoader, load
 from salt.utils.odict import OrderedDict
+from salt.utils.yamlloader import SaltYamlSafeLoader, load
+from yaml.constructor import ConstructorError
+from yaml.nodes import MappingNode, ScalarNode
 
 try:
     from salt.utils.files import fopen