Cron management

[vanosg]

I think we can do cron management a little bit better, and as a result of changes here, better support the program in other locations as well.

Global settings file

To begin, I propose creating a ~/settings/.brewpisettings file to hold common variables we already repeatedly ask for in the install and upgrade process (and other places as well). By placing this file in a standard location, the multiple brewpi scripts could reference (from any repo) without having to import an entire module from a differenet repo, or hack-ily parse through other code. It would hold:

• webPath - path to brewpi web install location
• scriptPath - path to brewpi-script install location
• Hexfile version - current version of the hex file loaded on Arduino
• board version - model of Arduino board being used
• modules present to be loaded - in this case, brewpi.py, wifichecker.sh, and maybe bubblecounter.whatever
• stdout logging path - path to stdout log file
• stderr logging path - path to stderr log file
  ... and anything else that we reuse I haven't though of yet.

At first, I thought about pushing the entire brewpi settings file currently being used to /etc/brewpi, but I think this will cause significant issues with configuration management as updates are made to the config file, or the user changes it. Instead, I suggest creating a sym-link in /etc/brewpi/settings to wherever the settings file is installed. This way, any script can universally reference /etc/brewpi/settings to get its important data, no matter where the user has installed brewpi. This additionally allows us to make changes to the config file locally via the git repo without needing to update a file outside the repo.

Module requirements

A module will take one of four arguments from stdin when run:

• null: Run the module
• install: install the module
• cron: return the cron line that should be used in cron.d
• remove: optional, but good to have it in the plan

Installing a module

For cronUpdate.sh , I strongly disagree with putting the install code for each module into cronUpdate.sh .

The goal of modularity is pushing as much code specific to a module, out to the module itself. Think of how aptitude updates run, or installing a python module- the install code for each new package is not contained within the main program- you merely pass an install argument to the code it has downloaded, and the package takes care of the rest. For wifichecker, its only a few lines, but as modules become more advanced, so too may become the install code. You don't want to carry around that amount of code for modules some users may never use (and especially if the modules eventually get pushed to their own repo).

Instead, when called by cronUpdate.sh with the 'install' argument (and root privs), the module will take care of adding itself to cron.d, updating its entry in the .settings file as 'true', and moving files anywhere they may need to be. There will also need to be an install path variable for each entry

Tracking/Updating modules

To update modules, updateCron cycles through the list of installed modules in .settings, checks which vars are true, and compares (version number? md5 hash? cron line?) of a module and, if different, prompts to update.
-OR-
cycle through each installed module and just overwrite the cron file with each new entry (starting with brewpi)

I don't want to be prompted to install old modules I already said no to!

updateCron has a 'master' list of all potential modules within it. (This is the only module-specific code located in updateCron) When it scans .settings to update, any entries that exist within the master list but not the .settings file will cause the user to be prompted to install that module. If they choose 'yes', the module is installed, and an entry is made to the .settings file with moduleName = True. If the user choose not to install, an entry is made to .settings with moduleName = False, and that module is not prompted to the user for install again on future updates.

updateCron psuedocode

read in all modules listed in .settings file
retrieve current cron lines from each installed module
compare to installed cron.d lines, replace if necessary (or straight overwrite everytime)
prompt user to install any newly-added modules not seen before
restart cron
Profit!

Bottom line

I think this framework for modules / cron

• gives us a format that future modules should adhere to
• pushes as much unique/module-specific code out to the module as possible
• promotes code re-use by brewpi
• prevents extensive changes to 'core' code (updateCron.sh) as each module is added (only adding a variable name to check for, nothing extensive or bulky)
• enables commonly-used variables to be used from a static location
• don't need to create backward-compatibility checks for the 'new' cron file version

Lets discuss!

[elcojacobs]

There are a few things lumped together in this proposal, so I will address them separately:

Having one place to store settings

I agree that we should have a central default location to store settings, default being /etc/brewpi sounds good. Similarly, we should log to /var/log/brewpi.
Data can be logged to /var/brewpi.

Where I disagree is that we keep settings in the repositories. The setting files should NOT be part of the repository, as changes in them will cause merge conflicts. What should be in the repo are defaults to fall back on, when the settings are not defined in /etc/settings.

Another point of discussion is where to put the brewpi execuatables (everything in the git repo). /usr would make sense for example.

Another point of discussion is how we do this on Windows. We can use C:\ProgramData, C:\Users.. and also make it a required part of the install to assign a directory.

In all cases, we do need a file in the repository to point to these directories. But this file should not be in version control (it should be in .gitignore). The file can be generated at install and we can revert to defaults if it is missing.

Modules

Our module definition is vague. Right now, it seems to mean CRON entry. We are talking about replacing cron.d with daemontools, so we need to define this differently.
We will have applications that need to run continuously (services, like the brewpi script) and things that need to run periodically, like the wifi checker.

In the next version, we will have multiple scripts, databases, a webserver, etc.

installing modules

I will talk just about cron here, because I don't have enough knowledge about daemon tools yet. Hopefully some of the things will apply there too.

If modules are responsible for their install, independent of other files, this will lead to code repetition. Each of the modules would contain code to find itself in the cron job and to update this.
Therefore, I think letting the module return its preferred cron entry, but let external code mange it is a better design. So updateCron.sh could just ask each module for the cron entry, but other than that, the module does not manage anything.
One time setup things (like adding auto wlan0 to network interfaces in wifiChecker) can be part of the module setup, but these should be repeatable.

tracking/updating/managing modules

This splits in 4 parts:

• Which modules are available
• Which modules are installed
• Which modules are chosen not to be installed
• Updating existing modules

This is largely covered already with the recent change in the cron.d file format. The list of installed or explicitly not installed modules is stored in the cron file itself. Because we did not have a central place to store settings, I thought this would be the best location: closest to the actual entries.
When a module is chosen not to be installed, it is added to the list prepended with ~.
Which modules are available can be part of updateCron.sh

Part of this rewrite was also to put log paths in variables. This allows us not make the cron jobs system specific, so we can version control them just by comparing the line, even when the user specified different directories.

So I think the recent cron rewrite already brought a lot of the things on the wish list here. I would rather focus on the next generation code and daemon tools. I don't think moving the files (settings/data) is worth the hassle in the current code, but for the next generation we should do it right from the start.

[vanosg]

There's a lot covered. Let me hit one paragraph at a time.

Having one place to store settings

Agree with most in here- logging to /var/log/brepwi is a great idea. I don't like the idea of actually pushing a settings file to /etc/brewpi , because that makes it difficult for us to maintain. Having a dettings.default file we can add lines to via git, and a settings file that overrides that is a good way to balance user-mods and dev-management. I don't claim to have all the answers here, but I think keeping the settings file in the script dir, with a symblolic link to it in the /etc dir, and adding it to the .gitignore, (like we do now) provides flexibility for future changes, but not a big deal, either way is I guess essentially the same. Pick your poision.

Not sure we need to move executables out to /usr, again it makes it hard for us to maintain

I don't think we need to involve moves involving porting to Windows in this thread of discussion (at this point, atleast)

Modules

My thought is anything that is not part of core brewpi (ie, something that every user needs in order to run the brewpi fermentation chamber) should be a module. Bubble counting, wifi-checking, Mash-tun monitoring, etc). I don't think it should at all be related to if it has a cron entry or not. Maybe that's where a disconnect between these thought processes is, in my head a module doesn't necessarily mean something that is run by cron, just in many cases maybe it would be helpful. Defining what a module is so we're all on the same page will certainly help. At a bare minimum, however, a module should be able to execute by itself particularly for ease of testing purposes,

Installing modules

This is good point- regarding letting cronUpdate add the line to the cron file. Good thought and that will also minimize code re-usage. I do still think we should avoid having all the setup and install questions being run from cronUpdate, that should be module specific and used to either help generate the cron line or store some other settings in the global settings file for future need.

Tracking/managing/updating modules.

The re-write you just did does work. I just think the method that I am proposing will lead to less headaches in the future. Here is why I think that:

• You don't have to worry about comparing lines- this gets quickly cumbersome to maintain, having to backwards-support every change you have ever made in the cron file (THIS IS A BIG, BIG DEAL)
• Rarely ever would you use a cron file to track settings- it can be done, but it normally isn't. As a user, I want all my variables and settings to be maintained in a single, easy-to-find location
• The variables you are using in cron can also be used in numerous other places in the script. Why do I want to keep trying to re-find those variables across different files outside of cron (updater for instance), instead of knowing where I can read them consistently? Yes, storing them 'close to the entries' is good for now, but what about other files that want to use them? Let's just define the global settings file and use that, for not just modules but everything- scriptPath and wwwPath are reused and checked constantly throughout the tools repo, this solution would fix many things
• And again, regarding logs, in a global settings file the log paths are also placed there, eliminating the need to ever compare a line- its set and kept by the "user", and customizable to each system. Again, all in the same place, accessible by all programs.

Next Gen

If you don't care about working on it in the current codebase, then let me :P That will a) free you up to focus on the next gen codebase, and b) let this implementation give you an idea on what does and doesn't work for this method of managing modules. Worst case, the next gen codebase doesn't stabilize for another year (hope not, but it happens!) and you have a better management solution here to for the current userbase.

Like I said, the current re-write certainly works, I just think we can still improve upon it and use it to guide the creation of a module framework for the next gen of code.