How to ensure migrations to be run on a single web host

[igor-alexandrov]

We have several hosts for web. Is there currently a way to ensure that Rails migrations will run only on a single web host?

Currently we start our containers with docker-entrypoint file that is listed below.

#!/bin/bash -e

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

[schonert]

Would like to bump this up. Been looking around for suggestions and there's not a lot of info regarding handling migrations. Having a consolidated answer we can reference would be great!

We initially handled migrations in our entrypoint - however we later experienced some issues with longer running data migrations that involved fetching and syncing data with external an external service.

Ideally we could run this in the pre_deploy hooks - However, I'm not sure how to access and execute the script inside the new docker container.

For now we have to either run the long migrations manually after a deploy or do a kamal app exec -i "rails db:migrate:with_data" from inside our post deploy hook.

[fesilvajacquier]

I am arriving a bit late to the discussion. But here it goes.
I am trying to ensure that the migrations are run on only one host on deploy.
Does it make sense to tag the desired host (let's say with db) and to set the ENV variable DB_ROLE=1 at env.tags.db and use it to conditionally run the migration from the docker-entrypoint?

[nikhilbhatt]

@fesilvajacquier yes agree & I think tags are for this purpose only, to set env vars for a specific host.

how to set tags?
https://kamal-deploy.org/docs/configuration/environment-variables/#tags

Example

servers:
  - 172.0.0.1: db
  - 172.0.0.2
  - 172.0.0.3
  
env:
  clear:
    MYSQL_USER: app
  secret:
    - MYSQL_PASSWORD
  tags:
    db:
      MIGRATION: 1

and in docker-entrypoint we can use this MIGRATION env variable to run the migrations.

[djmb]

You should be able to do something like this in a pre-deploy hook I think:

VERSION=$KAMAL_VERSION kamal app exec -p "./bin/rails db:prepare"

If using destinations you'd need to make sure to pass the current one as well (you can get the current one in the KAMAL_DESTINATION env variable in hooks).

The -p will only run it on the primary server.

The ENV tag approach looks nice as well, but using a pre-deploy hook ensures the migrations are run before any of the new app containers are booted.

[emmceemoore]

FYI - I've noticed that adding something like this to the pre-deploy hook after the initial setup has already been performed seems to work OK. However, I run into errors when I attempt to kamal setup things from scratch.

...
Running the pre-deploy hook...
  INFO [c0a62ed5] Running /usr/bin/env .kamal/hooks/pre-deploy as some-user@localhost
  Finished all in 12.2 seconds
Releasing the deploy lock...
  Finished all in 162.7 seconds
  ERROR (Kamal::Cli::HookError): Hook `pre-deploy` failed:
.kamal/hooks/pre-deploy exit status: 256
.kamal/hooks/pre-deploy stdout: >>> PRE-DEPLOY...
- Running migrations...
Get most recent version available as an image...
Launching command with version latest-production from new container...
Running the pre-connect hook...
  INFO [cdca5ffe] Running /usr/bin/env .kamal/hooks/pre-connect as some-user@localhost
 DEBUG [cdca5ffe] Command: /usr/bin/env .kamal/hooks/pre-connect
 DEBUG [cdca5ffe]       >>> PRE-CONNECT...
  INFO [cdca5ffe] Finished in 0.009 seconds with exit status 0 (successful).
 DEBUG [bd6d6154] Running /usr/bin/env mkdir -p .kamal && echo [2024-11-04T22:59:45Z] [some-user@redacted.com] [production] Executed cmd './manage.sh migrate' on app version latest-production >> .kamal/some-website-production-audit.log on 192.168.1.210
 DEBUG [bd6d6154] Command: /usr/bin/env mkdir -p .kamal && echo [2024-11-04T22:59:45Z] [some-user@redacted.com] [production] Executed cmd './manage.sh migrate' on app version latest-production >> .kamal/some-website-production-audit.log
 DEBUG [bd6d6154] Finished in 0.357 seconds with exit status 0 (successful).
  INFO [65f3e607] Running docker run --rm --network kamal --env ENV_NAME="production" --env-file .kamal/apps/some-website-production/env/roles/web.env --cpus "1" --memory "2g" ghcr.io/redacted/some-website:latest-production ./manage.sh migrate on 192.168.1.210
 DEBUG [65f3e607] Command: docker run --rm --network kamal --env ENV_NAME="production" --env-file .kamal/apps/some-website-production/env/roles/web.env --cpus "1" --memory "2g" ghcr.io/redacted/some-website:latest-production ./manage.sh migrate
 DEBUG [65f3e607]       docker: open .kamal/apps/some-website-production/env/roles/web.env: no such file or directory.
See 'docker run --help'.
  ERROR (SSHKit::Command::Failed): Exception while executing on host 192.168.1.210: docker exit status: 125
docker stdout: Nothing written
docker stderr: docker: open .kamal/apps/some-website-production/env/roles/web.env: no such file or directory.
See 'docker run --help'.

/Users/some-user/.local/share/gem/gems/sshkit-1.23.2/lib/sshkit/command.rb:97:in `exit_status='
/Users/some-user/.local/share/gem/gems/sshkit-1.23.2/lib/sshkit/backends/netssh.rb:185:in `execute_command'
/Users/some-user/.local/share/gem/gems/sshkit-1.23.2/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
<internal:kernel>:90:in `tap'
/Users/some-user/.local/share/gem/gems/sshkit-1.23.2/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
/Users/some-user/.local/share/gem/gems/sshkit-1.23.2/lib/sshkit/backends/abstract.rb:66:in `capture'
/Users/some-user/.local/share/gem/gems/kamal-2.3.0/lib/kamal/sshkit_with_ext.rb:10:in `capture_with_info'
/Users/some-user/.local/share/gem/gems/kamal-2.3.0/lib/kamal/cli/app.rb:141:in `block (3 levels) in exec'
/Users/some-user/.local/share/gem/gems/kamal-2.3.0/lib/kamal/cli/app.rb:139:in `each'
/Users/some-user/.local/share/gem/gems/kamal-2.3.0/lib/kamal/cli/app.rb:139:in `block (2 levels) in exec'
/Users/some-user/.local/share/gem/gems/sshkit-1.23.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
/Users/some-user/.local/share/gem/gems/sshkit-1.23.2/lib/sshkit/backends/abstract.rb:31:in `run'
/Users/some-user/.local/share/gem/gems/kamal-2.3.0/lib/kamal/sshkit_with_ext.rb:118:in `block (2 levels) in execute'
.kamal/hooks/pre-deploy stderr: Nothing written

docker stderr: docker: open .kamal/apps/some-website-production/env/roles/web.env: no such file or directory.

It looks as though the web.env file doesn't get created till after the pre-deploy hook completes.

[dn]

@djmb I might be wrong here - but wouldn't that execute the migration on the 'old' primary - and not on the new just build image/primary to be?

To clarify - I'm also looking into a way to only run migration/some rake task once per deployment - before the rails server start preferable;

The only solution I have so far seems to use something like:

boot:
  limit: 1
  wait: 180

This slows down deployments a lot - but as long as everything finishs in 180s - I do not run into any errors. Is there a better way? Am I overlooking here something? It seems like everyone would run into this - as soon as you have more than 1 host.

[djmb]

@dn - kamal app exec -p will boot a new container and the VERSION=$KAMAL_VERSION should ensure it is using the version we have just built. Though I've not tested this works!

The issue with the env that @emmceemoore has pointed out might be a problem anyway. kamal app exec won't have access to the latest environment variables. We might need to add an option to it to push them (see also #1223)

[emmceemoore]

For me, running VERSION=$KAMAL_VERSION kamal app exec ... in the pre-deploy hook didn't seem to work:

  INFO [bfdf4bb2] Running /usr/bin/env .kamal/hooks/pre-deploy as ubuntu@localhost
 DEBUG [bfdf4bb2] Command: /usr/bin/env .kamal/hooks/pre-deploy
 DEBUG [bfdf4bb2] 	Get most recent version available as an image...
 DEBUG [bfdf4bb2] 	Launching command with version latest-production from new container...
 DEBUG [bfdf4bb2] 	Running the pre-connect hook...
 DEBUG [bfdf4bb2] 	  INFO [97a9104f] Running /usr/bin/env .kamal/hooks/pre-connect as ubuntu@localhost
 DEBUG [bfdf4bb2] 	 DEBUG [97a9104f] Command: /usr/bin/env .kamal/hooks/pre-connect
 DEBUG [bfdf4bb2] 	  INFO [97a9104f] Finished in 0.002 seconds with exit status 0 (successful).
 DEBUG [bfdf4bb2] 	 DEBUG [28b127c5] Running /usr/bin/env mkdir -p .kamal && echo [2025-01-29T15:32:30Z] [ubuntu] [production] Executed cmd './do-some-stuff.sh' on app version latest-production >> .kamal/my-website-production-audit.log on vm-production-01.example.com

(Notice the Launching command with version latest-production ... bit. It doesn't seem that the latest-$DESTINATION tag is pointing the new version at this point in the deployment.)

---

However, using the --version command line option (i.e. - kamal app exec --version "${KAMAL_VERSION}" ...) did work for me:

  INFO [28851627] Running /usr/bin/env .kamal/hooks/pre-deploy as ubuntu@localhost
 DEBUG [28851627] Command: /usr/bin/env .kamal/hooks/pre-deploy
 DEBUG [28851627] 	Launching command with version 4fb574c5b8fb02a0e81da31ab2d19655bb225fdf from new container...
 DEBUG [28851627] 	Running the pre-connect hook...
 DEBUG [28851627] 	  INFO [86dd0e47] Running /usr/bin/env .kamal/hooks/pre-connect as ubuntu@localhost
 DEBUG [28851627] 	 DEBUG [86dd0e47] Command: /usr/bin/env .kamal/hooks/pre-connect
 DEBUG [28851627] 	  INFO [86dd0e47] Finished in 0.002 seconds with exit status 0 (successful).
 DEBUG [28851627] 	 DEBUG [7c32bcc1] Running /usr/bin/env mkdir -p .kamal && echo [2025-01-29T18:57:47Z] [ubuntu] [production] Executed cmd './do-some-stuff.sh' on app version 4fb574c5b8fb02a0e81da31ab2d19655bb225fdf >> .kamal/my-website-production-audit.log on vm-production-01.example.com

(Notice that the actual version is used in lieu of the changing latest-$DESTINATION tag.)