Allow running detached app commands and follow logs by container ID #906

aliismayilov · 2024-08-07T14:05:44Z

This is useful for long running rake tasks or scripts that can be run without having to keep an open connection to servers.

The command that starts a detached container exits immediately and the output leaves the container ID that can be used for following the logs or investigating them later. To support this, I've also extended the kamal logs command with --container-id option.

Example:

❯ kamal app exec --detach 'bin/rails hello'
Get most recent version available as an image...
Launching command with version latest from new container...
  INFO [21e97fa4] Running docker run --detach --network kamal --env SOLID_QUEUE_IN_PUMA="true" --env-file .kamal/apps/dummy_rails/env/roles/web.env --log-driver "awslogs" --log-opt awslogs-region="eu-central-1" --log-opt awslogs-group="/ec2/dummy-rails" --log-opt awslogs-force-flush-interval-seconds="1" --volume dummy_rails_storage:/rails/storage 233638847975.dkr.ecr.eu-central-1.amazonaws.com/dummy_rails:latest bin/rails hello on 3.77.153.204
  INFO [21e97fa4] Finished in 0.438 seconds with exit status 0 (successful).
App Host: 3.77.153.204
d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090

❯ kamal logs --follow --container-id=d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090
  INFO Following logs on 3.77.153.204...
  INFO ssh -t root@3.77.153.204 -p 22 'echo d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090 | xargs docker logs --timestamps --tail 10 --follow 2>&1'
2024-10-26T13:03:20.334253736Z logger says: 3
2024-10-26T13:03:21.334471050Z logger says: 4
2024-10-26T13:03:22.334622132Z logger says: 5
2024-10-26T13:03:23.334916155Z logger says: 6
2024-10-26T13:03:24.335143905Z logger says: 7
2024-10-26T13:03:25.335388754Z logger says: 8
2024-10-26T13:03:26.335653526Z logger says: 9
2024-10-26T13:03:27.335893494Z logger says: 10
2024-10-26T13:03:28.336115025Z logger says: 11
2024-10-26T13:03:29.336400979Z logger says: 12
2024-10-26T13:03:30.336642630Z logger says: 13
2024-10-26T13:03:31.336861355Z logger says: 14
2024-10-26T13:03:32.337858183Z logger says: 15
2024-10-26T13:03:33.338105717Z logger says: 16
2024-10-26T13:03:34.338330317Z logger says: 17
2024-10-26T13:03:35.338555287Z logger says: 18
2024-10-26T13:03:36.338794180Z logger says: 19
2024-10-26T13:03:37.339028060Z logger says: 20
Connection to 3.77.153.204 closed.
❯ kamal logs --container-id=d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090
  INFO Following logs on 3.77.153.204...
  INFO ssh -t root@3.77.153.204 -p 22 'echo d5bba7b800f054a217623428f0926585b90e075fdd46e368083323e706f1e090 | xargs docker logs --timestamps --tail 10 --follow 2>&1'
2024-10-26T13:03:28.336115025Z logger says: 11
2024-10-26T13:03:29.336400979Z logger says: 12
2024-10-26T13:03:30.336642630Z logger says: 13
2024-10-26T13:03:31.336861355Z logger says: 14
2024-10-26T13:03:32.337858183Z logger says: 15
2024-10-26T13:03:33.338105717Z logger says: 16
2024-10-26T13:03:34.338330317Z logger says: 17
2024-10-26T13:03:35.338555287Z logger says: 18
2024-10-26T13:03:36.338794180Z logger says: 19
2024-10-26T13:03:37.339028060Z logger says: 20
Connection to 3.77.153.204 closed.

To support logs inspection, I've also applied kamal logging config to all newly started app exec containers. This resolves #1154.

djmb

Hi @aliismayilov,

Sorry for the delay in getting to this! I think we just need some checks for invalid option combinations and we should be good to go!

lib/kamal/cli/app.rb

djmb · 2024-10-23T11:04:56Z

Also it looks like there are some test failures - CliAppTest#test_exec_detach and CommandsAppTest#test_execute_in_new_detached_container

aliismayilov · 2024-10-23T15:12:24Z

Hi @aliismayilov,
Sorry for the delay in getting to this! I think we just need some checks for invalid option combinations and we should be good to go!

Thanks for looking into this! I've added an incompatibility check. See if it makes sense.

Also it looks like there are some test failures - CliAppTest#test_exec_detach and CommandsAppTest#test_execute_in_new_detached_container

Not anymore! I must've missed them in the avalanche of log lines 😄

aliismayilov · 2024-10-23T19:25:54Z

I just realized that there will be no output from a detached container in the logs. I created a separate issue: #1154

aliismayilov · 2024-10-26T13:52:08Z

@djmb I've updated the pull request with more changes to support logs inspection for detached containers. In short:

all newly started app containers get kamal logging configuration applied
detached containers do not have --rm option. This allows inspection of detached containers even after they finished running
kamal logs gets a new option --container-id to inspect logs of a specific container

I've also updated the pull request description.

lib/kamal/commands/app/logging.rb

this is useful for long running rake tasks or scripts that can be run without having to keep open connection to the server. Example: ``` kamal app exec 'bin/rails db:backfill_task' --detach ```

aliismayilov · 2024-11-21T17:56:26Z

@djmb I've rebased the PR. There shouldn't be any conflicts or failures 😬

aliismayilov · 2024-11-23T20:57:50Z

lib/kamal/commands/app/logging.rb

+
+  private
+
+  def container_id_command(container_id)


TBH, I'm not happy about this approach, but I couldn't come up with a better approach that would allow passing literal container ID (via kamal command) and a command that outputs container ID (see: lib/kamal/cli/app/boot.rb)

aliismayilov · 2024-11-23T21:00:03Z

OK, I fixed the failing spec and to be sure, the CI passes on my ~~machine~~ account: https://github.com/aliismayilov/kamal/actions/runs/11990309351

aliismayilov force-pushed the detached-run branch from 687f08b to 1cac6e5 Compare September 3, 2024 08:02

aliismayilov force-pushed the detached-run branch from 1cac6e5 to d2edf8d Compare October 21, 2024 13:48

djmb requested changes Oct 23, 2024

View reviewed changes

lib/kamal/cli/app.rb Show resolved Hide resolved

aliismayilov force-pushed the detached-run branch 2 times, most recently from 6a21493 to a978ed2 Compare October 23, 2024 14:57

aliismayilov force-pushed the detached-run branch from a978ed2 to 765e441 Compare October 26, 2024 12:41

aliismayilov changed the title ~~Allow running detached app commands~~ Allow running detached app commands and follow logs by container ID Oct 26, 2024

aliismayilov marked this pull request as draft October 26, 2024 13:29

aliismayilov force-pushed the detached-run branch 3 times, most recently from fb1b4bf to bc049e7 Compare October 26, 2024 13:50

aliismayilov marked this pull request as ready for review October 26, 2024 13:50

djmb reviewed Nov 21, 2024

View reviewed changes

lib/kamal/commands/app/logging.rb Show resolved Hide resolved

aliismayilov added 6 commits November 21, 2024 18:05

Allow running detached app commands

dbe0c3a

this is useful for long running rake tasks or scripts that can be run without having to keep open connection to the server. Example: ``` kamal app exec 'bin/rails db:backfill_task' --detach ```

Make --detach incompatible with reuse or interactive

c662b8d

Enable logging on app exec new containers

1da882b

Follow logs of a specific container

183fe9e

Capture logs for specific container_id

75b44cd

Prefer dasherized notation

ac90ee0

aliismayilov force-pushed the detached-run branch from 091a836 to ac90ee0 Compare November 21, 2024 17:54

aliismayilov added 2 commits November 23, 2024 21:37

Pipe app container id

1cc5406

Allow running the CI manually

8a7843c

aliismayilov force-pushed the detached-run branch from e806242 to 8a7843c Compare November 23, 2024 20:46

aliismayilov commented Nov 23, 2024

View reviewed changes

djmb merged commit fbc4515 into basecamp:main Dec 2, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow running detached app commands and follow logs by container ID #906

Allow running detached app commands and follow logs by container ID #906

aliismayilov commented Aug 7, 2024 •

edited

Loading

djmb left a comment

djmb commented Oct 23, 2024

aliismayilov commented Oct 23, 2024

aliismayilov commented Oct 23, 2024

aliismayilov commented Oct 26, 2024 •

edited

Loading

aliismayilov commented Nov 21, 2024

aliismayilov Nov 23, 2024

aliismayilov commented Nov 23, 2024

Allow running detached app commands and follow logs by container ID #906

Allow running detached app commands and follow logs by container ID #906

Conversation

aliismayilov commented Aug 7, 2024 • edited Loading

djmb left a comment

Choose a reason for hiding this comment

djmb commented Oct 23, 2024

aliismayilov commented Oct 23, 2024

aliismayilov commented Oct 23, 2024

aliismayilov commented Oct 26, 2024 • edited Loading

aliismayilov commented Nov 21, 2024

aliismayilov Nov 23, 2024

Choose a reason for hiding this comment

aliismayilov commented Nov 23, 2024

aliismayilov commented Aug 7, 2024 •

edited

Loading

aliismayilov commented Oct 26, 2024 •

edited

Loading