[Kamal 2] Redirections to internal host?

[jmonteiro]

I have a Rails app running on two Raspberry Pis, deployed with Kamal v1. Those are made accessible from the internet through CloudFlare Tunnels and a CloudFlare Load Balancer.

The Kamal v1 and Rails apps are pretty simple. They are running on port 80.

On CloudFlare Zero Access, I have tunnels setup so each one is called as pi1.example.org and pi2.example.org, with an override for setting the HTTP host to example.org

The CloudFlare Load Balancer is mounted at example.org and each of the two servers are mapped as:

This worked fine on Kamal v1, however once migrating to Kamal v2 I started facing issues. Before, a redirection in Rails like a redirect_to new_user_path would return a Location: https://example.org/users/new HTTP header, but on Kamal 2 I'm getting Location: https://pi1.example.org/users/new (notice the incorrect host name), leading to a browser error as pi1.example.org is unaccessible from the internet.

My Kamal v2 deploy.yml file includes this:

proxy:
  ssl: false
  host: example.org
  healthcheck:
    path: /up

I've been trying a few approaches to fix this, but wasn't able to find a solution. I guess that's a misconfiguration from my part on the deploy.yml, as the CloudFlare section should be pretty standard.

Any ideas and suggestions on how I could fix this issue?

[thiagochirana]

so, did you manage to solve the problem?

I did but I didn't use load ballancer or anything else

[jmonteiro]

Unfortunately no, and I tried it with Kamal 2.4. Something in the stack (my best guess is kamal-proxy) is rewriting the Location host to reflect the instance hostname instead.

[jmonteiro]

Found the issue. Doing some investigation on this tonight, I found the reason why that's happening:

On Kamal v1 (with Traefik), the following headers are passed to Rails:

• SERVER_NAME: example.org
• HTTP_HOST: example.org
• HTTP_X_FORWARDED_HOST: example.org

On Kamal v2 (with Kamal-proxy), with the exact same application and environment, the following headers are passed to Rails:

• SERVER_NAME: example.org
• HTTP_HOST: example.org
• HTTP_X_FORWARDED_HOST: pi1.example.org ⭐️

Notice that the HTTP_X_FORWARDED_HOST header is different when using Kamal v2, and that is causing the incorrect redirection. Bear in mind that the env is exactly the same, the only moving part is Kamal v1 vs v2.

My best guess is that Traefik is ignoring the forwarded host due to security concerns and need to configured to trust is in order to pass it along (at least that's my understanding based on a quick look on their documentation), whereas Kamal Proxy is simply passing them along as is. So this was something that was affecting my previously, but I wasn't impacted due to the lack of trust from Traefik.

My solution for this was to improve my already existing CloudFlare middleware to ignore the X-Forwarded-For header:

🗒️ config/initializers/cloudflare_ip.rb:

# frozen_string_literal: true

class CloudflareIp
  def initialize(app)
    @app = app
  end

  def call(env)
    if env["HTTP_CF_CONNECTING_IP"]
      # persist headers before we change them
      env["HTTP_REMOTE_ADDR_BEFORE_CF"] = env["REMOTE_ADDR"]
      env["HTTP_X_FORWARDED_FOR_BEFORE_CF"] = env["HTTP_X_FORWARDED_FOR"]

      # load real values passed along by CloudFlare
      env["REMOTE_ADDR"] = env["HTTP_CF_CONNECTING_IP"]
      env["HTTP_X_FORWARDED_FOR"] = env["HTTP_CF_CONNECTING_IP"]
    end

    # CloudFlare Load Balancer sends an internal host on "X-Forwarded-Host", so we need to overwrite with the "Host" header
    if env["HTTP_X_FORWARDED_HOST"]
      env["HTTP_X_FORWARDED_HOST_BEFORE_OVERWRITE"] = env["HTTP_X_FORWARDED_HOST"]
      env["HTTP_X_FORWARDED_HOST"] = env["HTTP_HOST"]
    end

    @app.call(env)
  end
end

Rails.application.config.middleware.insert_before(0, CloudflareIp) unless Rails.env.local?

The new section is the if env["HTTP_X_FORWARDED_HOST"].