diff --git a/priv/posts/security/20241212210148_epmd-public-exposure.md b/priv/posts/security/20241212210148_epmd-public-exposure.md index c3374a0f..138bed67 100644 --- a/priv/posts/security/20241212210148_epmd-public-exposure.md +++ b/priv/posts/security/20241212210148_epmd-public-exposure.md @@ -10,105 +10,117 @@ EPMD, essential for Erlang and RabbitMQ clustering, is often exposed online—posing hidden security risks and requiring quick mitigation steps. --- -The Erlang Port Mapper Daemon (EPMD) is a built-in component that facilitates clustering across Erlang-based applications (including RabbitMQ). While EPMD is essential in trusted network environments, it was never intended for exposure on the public internet. Recent scans have revealed over 85,000 instances of publicly exposed EPMD, with roughly half associated with RabbitMQ servers. +The Erlang Port Mapper Daemon (EPMD) is a built-in component that helps Erlang-based applications (including RabbitMQ) discover each other’s distribution ports for clustering. Although EPMD itself isn’t directly dangerous, its exposure on the public internet often signals that Erlang Distribution ports are also exposed. This creates a serious security risk: if attackers find these distribution ports, they can potentially join your cluster, run arbitrary code, and compromise your systems. Recent scans have revealed over 85,000 instances of publicly accessible EPMD, with roughly half associated with RabbitMQ servers. -If left unsecured, an exposed EPMD endpoint can potentially grant attackers a foothold, enabling them to join your cluster, run arbitrary code, and compromise your systems. Fortunately, mitigation steps are straightforward: disable EPMD if you’re not clustering, or restrict it behind a firewall and proper network configuration. +If left unsecured, exposed Erlang Distribution ports let attackers gain a foothold in your system. Fortunately, mitigation steps are straightforward: disable EPMD if you’re not clustering, or restrict it behind a firewall and proper network configuration—and ensure Erlang Distribution is never exposed to untrusted networks. ## TL;DR **Check if you’re exposed:** - Run a port scan: ```bash - nmap -p 4369 -sV [Your Public IP] + nmap -A -sT -p 4369 [Your Public IP] ``` If the output shows: ``` - PORT STATE SERVICE - 4369/tcp open epmd + PORT STATE SERVICE VERSION + 4369/tcp open epmd Erlang Port Mapper Daemon + | epmd-info: + | epmd_port: 4369 + | nodes: + |_ rabbit: 25672 ``` - Your EPMD port is exposed. + Your EPMD port is exposed, indicating your Erlang Distribution ports may be discoverable as well. **For Erlang/Elixir users:** -- Remove or avoid using `-name` or `-sname` if you don’t need clustering. -- Bind distribution to a non-public interface with `vm.args`: + +- **Use a firewall or security group rules to restrict external access**: + Close all ports except those explicitly required for your application. Restricting access to only essential services reduces your overall attack surface and helps mitigate various security risks beyond just EPMD exposure. + +- **Bind distribution to a non-public interface**: + In `vm.args`: ```bash -kernel inet_dist_use_interface '{127, 0, 0, 1}' ``` -- Use a firewall or security group rules to restrict external access. + Ensuring that distribution ports are only accessible via localhost or a private network interface further reduces the attack surface. + +- **Remove or avoid using `-name` or `-sname` if you don’t need clustering**: + If you’re not clustering nodes, don’t assign them a name that registers with EPMD. Without a name, your node won’t appear in EPMD’s registry, making it harder for attackers to locate your distribution ports. **For RabbitMQ users:** -- RabbitMQ runs on Erlang and uses EPMD behind the scenes. Even if you never interact directly with Erlang, you may still be affected. +- RabbitMQ runs on Erlang and uses EPMD behind the scenes. Even if you never interact directly with Erlang, you may still be affected if Erlang Distribution ports are exposed. - Check the RabbitMQ networking documentation to learn how to identify exposed ports and close them: [https://www.rabbitmq.com/docs/networking#epmd](https://www.rabbitmq.com/docs/networking#epmd) -- The Erlang Ecosystem Foundation’s security hardening guide explains what EPMD is, which ports it uses, and how to close them down: +- The Erlang Ecosystem Foundation’s security hardening guide explains what EPMD is, which ports Erlang, and how to close them down: [https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/distribution](https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/distribution) ## What is EPMD and How Does it Work? -Erlang was designed from the ground up for concurrency, distribution, and fault tolerance. To support clustering (distributed Erlang nodes connecting and communicating seamlessly), Erlang relies on EPMD. EPMD runs as a small daemon that keeps track of Erlang nodes and their associated ports. When a node starts with `-name` or `-sname`, it registers with EPMD on the host machine’s port 4369. +Erlang was designed for concurrency, distribution, and fault tolerance. To support clustering (distributed Erlang nodes connecting and communicating seamlessly), Erlang relies on EPMD. EPMD runs as a small daemon that keeps track of nodes and their associated distribution ports. When a node starts with `-name` or `-sname`, it registers with EPMD on port 4369. -In a controlled environment—such as an internal network—this works well. EPMD makes it easy for nodes to discover each other, connect, and form distributed clusters. +In a controlled environment—such as an internal network—this works well. EPMD makes it easy for nodes to discover each other, connect, and form clusters. However, when EPMD is open to the public internet, it provides a roadmap for attackers to find and target the Erlang Distribution ports themselves—where the real danger lies. **Key points:** - EPMD listens on port 4369 by default. - It’s assumed that EPMD is only reachable from within a trusted network. - Many Erlang/Elixir build tools and release pipelines (like Rebar3, Mix, and Distillery releases) enable EPMD by default. -## Why is Exposing EPMD a Bad Idea? +## Why is Exposing EPMD (and Erlang Distribution) a Bad Idea? -### Risk of Unauthorized Cluster Access +### Risk of Unauthorized Cluster Access Through Erlang Distribution -EPMD itself does not enforce strong authentication. Erlang clustering uses a “cookie” as a shared secret, but this cookie is not intended as a robust security mechanism—it’s more of a sanity check to prevent accidental cross-node connections. If an attacker can guess or brute-force the cookie, they could join your cluster. +EPMD’s purpose is to help Erlang nodes discover each other’s distribution ports. When exposed, it effectively provides a roadmap to these ports. Erlang clustering uses a “cookie” as a shared secret, but this cookie is not a robust authentication mechanism—just a sanity check to prevent accidental connections. If an attacker can guess or brute-force it, they can join your cluster. -Once inside the cluster, the attacker could potentially run arbitrary Erlang Remote Procedure Calls (RPCs), giving them full control of the application and underlying system. While no widespread internet-based attacks on EPMD have been documented publicly, the theoretical risk is real. A known brute-force approach is described here: -[https://insinuator.net/2017/10/erlang-distribution-rce-and-a-cookie-bruteforcer/](https://insinuator.net/2017/10/erlang-distribution-rce-and-a-cookie-bruteforcer/) +Once inside, an attacker can run arbitrary Erlang Remote Procedure Calls (RPCs), giving them full control over the application and potentially the underlying system. While no widespread internet-based attacks on Erlang Distribution have been publicly documented, the theoretical risk is real. EPMD exposure makes it significantly easier for attackers to find and exploit these distribution ports. + +A known brute-force approach and related tools are described here: + +* [Erlang distribution RCE and a cookie bruteforcer](https://insinuator.net/2017/10/erlang-distribution-rce-and-a-cookie-bruteforcer/) +* [erl-matter](https://github.com/gteissier/erl-matter?tab=readme-ov-file) ### Scope of the Problem -Shodan (a search engine for internet-connected devices) reveals over 85,000 publicly accessible EPMD instances. Among them, around 40,000 are associated with RabbitMQ. -Top countries hosting exposed EPMD endpoints include China (18,444), the United States (15,771), and Germany (10,263). Major cloud hosting platforms (DigitalOcean, Hetzner, Aliyun, Tencent, OVH, Amazon, and others) also host significant numbers of exposed instances, typically due to default configurations and lack of firewall restrictions. +Shodan (a search engine for internet-connected devices) reveals over 85,000 publicly accessible EPMD instances. Among them, around 40,000 are associated with RabbitMQ. +Top countries hosting exposed EPMD endpoints include China (18,444), the United States (15,771), and Germany (10,263). Major cloud hosting platforms (DigitalOcean, Hetzner, Aliyun, Tencent, OVH, Amazon, and others) also host significant numbers of exposed instances. This typically occurs when users fail to implement proper firewall rules, or lack a full understanding of Erlang distribution’s security implications. [![World Map showing number of exposed EPMD instances](/images/posts/epmd-public-exposure/world-map.png) {: .responsive-image}](https://www.shodan.io/search/report?query=product%3A%22Erlang+Port+Mapper+Daemon%22) ### Not Just Erlang Developers at Risk -RabbitMQ, a popular message broker built on Erlang, runs EPMD under the hood. Many RabbitMQ users may not be aware that EPMD exists or that it’s exposed. This lack of awareness can lead to overlooked vulnerabilities, as users trust defaults without applying basic network hygiene. +RabbitMQ, a popular message broker built on Erlang, runs EPMD and Erlang Distribution under the hood. Many RabbitMQ users may not be aware that these ports are exposed. Without proper network hygiene, this can leave their systems vulnerable, even if they never use Erlang directly. -## How to Secure Your EPMD +## How to Secure Your Erlang Distribution (and EPMD) **1. Use a Firewall or Restricted Network Interfaces** -If you don’t need external clustering, ensure EPMD isn’t exposed: +If you don’t need external clustering, ensure EPMD and the Erlang Distribution ports aren’t exposed: -- Bind EPMD to localhost or a private network interface. In `vm.args`, specify: +- Bind Erlang Distrubution and EPMD to localhost or a private network interface. In `vm.args`, specify: ```bash - -kernel inet_dist_use_interface '{127, 0, 0, 1}' + -kernel inet_dist_use_interface '{127, 0, 0, 1}' -env ERL_EPMD_ADDRESS "127.0.0.1" ``` - Deploy firewalls, security groups, or network access control lists (ACLs) to ensure port 4369 is not reachable from untrusted networks. -**2. Remove Clustering Options if Not Needed** +**2. Mitigation is Simple** -If you’re not clustering multiple Erlang/RabbitMQ nodes, don’t start the node with `-name` or `-sname`. Without these options, your application won’t register with EPMD at all. If you only run a single RabbitMQ node and do not need distribution, consider disabling or limiting it according to the official networking and clustering documentation. - -**3. Mitigation is Simple** - -A small configuration change or a single firewall rule can often resolve the issue. Once you’ve made changes to close or secure EPMD, scan again to confirm: +Applying a strict firewall policy, configuring loopback interfaces, and removing unnecessary distribution settings can resolve the issue. Close all ports not explicitly required by your application—simply closing port 4369 (EPMD) is not enough, as Erlang distribution can use additional, dynamically assigned ports. After adjusting your firewall and configuration, verify that only your intended services are accessible: ```bash -nmap -p 4369 -sV [Public IP] +nmap -sT -p0- 127.0.0.1 [Public IP] ``` -If you’ve successfully closed the port, the output should look like this: -``` -PORT STATE SERVICE -4369/tcp closed epmd +If your application only needs HTTPS (443) open, for example, ensure that the result shows closed for all ports except the ones you have intentionally whitelisted, such as: + +```bash +PORT STATE SERVICE VERSION +443/tcp open https ``` -**4. Follow Best Practices from the Community** +**3. Follow Best Practices from the Community** -- **RabbitMQ**: The official networking guide shows how to identify EPMD ports and close them down. -- **Erlang Ecosystem Foundation**: The security hardening guide explains what EPMD is, which ports are used, and how to secure them. +- **RabbitMQ**: The official networking guide details how to identify and close EPMD and distribution ports. +- **Erlang Ecosystem Foundation**: The security hardening guide explains what EPMD and Erlang Distribution are, which ports are used, and how to secure them. ## Conclusion -EPMD is a powerful mechanism for distributed Erlang applications and RabbitMQ clusters, but it was never meant to be exposed to the internet. Fortunately, securing EPMD is straightforward, often just a matter of adding a firewall rule, adjusting a configuration setting, or using loopback interfaces. By taking these simple steps, you prevent unauthorized access, reduce risk, and keep your applications safe. +EPMD exposure is a warning sign. It indicates that Erlang Distribution ports—the true target for potential attackers—may also be accessible. By securing or disabling EPMD where unnecessary and ensuring Erlang Distribution ports are not exposed to untrusted networks, you can prevent unauthorized access, reduce risk, and keep your applications safe.