Support long running cron tasks

[roelvanduijnhoven]

The best current way to work with cron, as documented here, has some down-sides for me. But, luckily, recent work with a separate environment file (#438) offers a great opportunity to improve cron! :)

The biggest downside of this approach is it will not work with long running cron task. As these will be aborted when a deploy hits. We make use of long running cron tasks quite a bit, and deploy quite a bit.

Cron as native thing

So: let's say we would add an option for a role to define a cronfile:

service: app
servers:
  some-role:
    hosts:
      - 34.35.36.37
    cronfile: config/crontab

Now: on deploy Kamal can create a /etc/cron.d/app-some-role file on the host. Kamal will wrap each line in the config/crontab file (extracted from the image) into something like this:

*/2 * * * * docker run --env-file ~/.kamal/env/roles/app-some-role.env some-image:6ec1f39 [original-command-here]

Now as a result the host system will nicely schedule cron :). Nothing is started twice. And long running task can simply finish.

Now: it's now as simple as that. The role options (like add-host) is for example not copied in this way. So we would need to find a way to mimic these settings as well.

One way to do that is that we could have Kamal write an executable ~/.kamal/run/roles/app-some-role file that will contain the whole command to run a new Docker container. In which case we can rewrite the cron to:

*/2 * * * * ~/.kamal/run/roles/app-some-role [original-command-here]

---

Note that is roughly the strategy that me and team have been using for cron for the last few years (some home-grown solution on top of Docker Swarm with some Ansible scripts that are retiring in favour of Kamal!). But that cron strategy specifically has been working great.

Could also be that I'm missing some simpler solutions that solves these problems I currently face :)!

(Note only now found out about https://github.com/basecamp/kamal/discussions/categories/ideas, should have used that probably. Sorry!)

[zealot128]

I stumbled upon the same issue - Using the recommended method of running a crond inside it's own Docker container worked apparently for simple workloads. But if you have a lot of cronjobs and deploy regularly, chances are, you are killing your cronjobs during running.

Even though most cronjobs should be performed idempotently and should handle a missing run and be short lived - and we usually try do adhere to these principles.
But still, killing a job in the middle of maintenance tasks might be not wanted, and switching to an in-process-solution, such as sidekiq-scheduler was not something we wanted to try first.

We are using whenever do generate the crontabs, and this method works:

1. wrap the job in a docker run with @roelvanduijnhoven great line, works:

Make sure to replace the role. We just run the cronjobs in one of the bg-worker's nodes, so we just steal the app-worker.env here.

# config/schedule.rb
revision = ENV['CI_COMMIT_SHORT_SHA'] || 'latest'

job_type :runner, "docker run -it --env-file ~/.kamal/env/roles/app-worker.env --rm registry.company.de/image/app:#{revision} bundle exec rails runner ':task' :output"

2. outside Kamal, update the crontab on the host like this:

#!/bin/bash
# chmod +x ./bin/update_cron

cron_host="10.11.12.13"
schedule_file="config/schedule.rb"
cron_image="registry.company.com/app/image:$CI_COMMIT_SHORT_SHA"

echo "Deploying cron job $cron_image to $cron_host"
ssh -t -i ./deploy/id_rsa root@$cron_host \
  "docker run -it $cron_image bundle exec whenever -f $schedule_file > /tmp/crontab && sed -i '/^\s*$/d' /tmp/crontab && crontab /tmp/crontab"

One can also put it into a Kamal post-deploy hook but for us, it's just one line more in the gitlab-ci.yml.

[jankeesvw]

I wanted to share a solution I encountered with getting Cron (with Whenever) to run correctly and how I eventually resolved it. After some trial and error, I discovered that the environment variables available to cron -f are not the same as those set when running the container. This caused some unexpected behavior and made it difficult to get rails tasks working as expected.

To solve this, I modified my deploy.yaml to pipe the environment variables into /etc/environment before starting Cron. This ensures that all necessary environment variables are accessible when Cron runs. Here is the updated section of my deploy.yaml:

cron:
  hosts:
    - 1.2.3.4
  cmd: bash -c "bundle exec whenever --update-crontab && env > /etc/environment && cron -f"

Additionally, I made some changes to my Dockerfile to handle the permissions for /etc/environment:

RUN touch /var/run/crond.pid && \
    chown rails:rails /var/run/crond.pid && \
    chown rails:rails /etc/environment

I hope this helps anyone who might encounter a similar issue in the future. It took me a while to figure this out, so hopefully, this can save someone else some time!

[frenkel]

I wanted to share a solution I encountered with getting Cron (with Whenever) to run correctly and how I eventually resolved it. After some trial and error, I discovered that the environment variables available to cron -f are not the same as those set when running the container. This caused some unexpected behavior and made it difficult to get rails tasks working as expected.

To solve this, I modified my deploy.yaml to pipe the environment variables into /etc/environment before starting Cron. This ensures that all necessary environment variables are accessible when Cron runs. Here is the updated section of my deploy.yaml:

cron:
  hosts:
    - 1.2.3.4
  cmd: bash -c "bundle exec whenever --update-crontab && env > /etc/environment && cron -f"

Additionally, I made some changes to my Dockerfile to handle the permissions for /etc/environment:

RUN touch /var/run/crond.pid && \
    chown rails:rails /var/run/crond.pid && \
    chown rails:rails /etc/environment

I hope this helps anyone who might encounter a similar issue in the future. It took me a while to figure this out, so hopefully, this can save someone else some time!

Thanks for sharing this. What kind of cron are you using? The one I use (from debian) gives an error when run as cron -f:
seteuid: Operation not permitted

[vladyio]

@frenkel Try this:

RUN touch /var/run/crond.pid && \
  chown rails:rails /var/run/crond.pid && \
  chown rails:rails /etc/environment && \
+ chmod u+s /usr/sbin/cron

Should help with the cron permissions issue.

[frenkel]

@vladyio thanks, but that is exactly what I want to prevent. I don't want to run it as root.

I've come up with a solution: use solid queue tasks as a replacement. Works much nicer!

[janosrusiczki]

While I appreciate @jankeesvw effort and response I found it too finicky. So I took @frenkel 's response as an inspiration and I ended up switching from whenever / cron to running my scheduled tasks via my job runner, which in my case is Sidekiq.

I just needed to add: https://github.com/sidekiq-scheduler/sidekiq-scheduler

I even spared some resources by not running a third container for cron as I was already running one for Sidekiq.