One of the most common areas Kubernetes operators struggle with in production involves creating and managing role-based access control (RBAC). This is so daunting that RBAC is often not implemented, or implemented halfway, or the configuration becomes impossible to maintain. In this post, we will contrast the traditional way of working with RBAC on Amazon EKS using Pulumi — Pulumi makes RBAC on Kubernetes so easy that you'll never create an insecure cluster again!
Install pulumi CLI and set up your AWS credentials.
Initialize a new Pulumi project from available templates. We use aws-typescript template here to install all dependencies and save the configuration.
$ brew install pulumi # download pulumi CLI
$ pulumi new aws-typescript --dir eks-rbac
$ cd eks-rbac && ls -l
-rw-r--r-- 1 nishidavidson staff 32 Apr 18 14:49 Pulumi.dev.yaml
-rw------- 1 nishidavidson staff 84 Apr 18 14:48 Pulumi.yaml
-rw------- 1 nishidavidson staff 273 Apr 18 14:48 index.ts
drwxr-xr-x 92 nishidavidson staff 2944 Apr 18 14:49 node_modules
-rw-r--r-- 1 nishidavidson staff 48352 Apr 18 14:49 package-lock.json
-rw------- 1 nishidavidson staff 228 Apr 18 14:48 package.json
-rw------- 1 nishidavidson staff 522 Apr 18 14:48 tsconfig.json
With Pulumi, you will modify and update the default index.ts file with AWS and EKS resource variable declarations. We show you how to add this code as we contrast Pulumi’s approach with the sequential traditional approach in the steps below. In the end, you will do a one-time run of pulumi up and watch all the steps in the Traditional Way come alive simultaneously.
You sequentially create three IAM roles (clusterAdminRole; AutomationRole; EnvProdRole) with aws command line tool as shown below:
$ aws iam create-role --role-name clusterAdminRole --assume-role-policy-document file://Role-Trust-Policy.json
{
"Role": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
},
"Effect": "Allow",
"Sid": ""
}
]
},
"RoleId": "AROASHIVKXX3SFFMUUEU6",
"CreateDate": "2019-04-17T17:43:03Z",
"RoleName": "clusterAdminRole",
"Path": "/",
"Arn": "arn:aws:iam::xxxxxxxxxxxx:role/clusterAdminRole"
}
}
$ cat Role-Trust-Policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "*"
}
]
}
You update the default index.ts file in your source code editor such as VSCode as follows:
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
/*
* 1) Single step deployment of three IAM Roles
*/
function createIAMRole(name: string): aws.iam.Role {
// Create an IAM Role...
return new aws.iam.Role(`${name}`, {
assumeRolePolicy: `{
"Version": "2012-10-17",
"Statement":[
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
},
"Action": "sts:AssumeRole"
}
]
}
`,
tags: {
"clusterAccess": `${name}-usr`,
},
});
}
// Administrator AWS IAM clusterAdminRole with full access to all AWS resources
const clusterAdminRole = createIAMRole("clusterAdminRole");
// Administer Automation role for use in pipelines, e.g. gitlab CI, Teamcity, etc.
const AutomationRole = createIAMRole("AutomationRole");
// Administer Prod role for use in Prod environment
const EnvProdRole = createIAMRole("EnvProdRole");
You go through the steps below to validate the cluster and k8s resource deployment based on your tool chain and your understanding of Kubernetes:
$ eksctl create cluster eks-nd-test
$ kubectl get no
NAME STATUS ROLES AGE VERSION
ip-192-168-41-125.us-east-2.compute.internal Ready <none> 11h v1.11.5
ip-192-168-5-250.us-east-2.compute.internal Ready <none> 11h v1.11.5
$ kubectl create -f automation-ns.yaml && kubectl create -f prod-ns.yaml
$ cat automation-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: automation
labels:
name: automation
You use our API docs and your source-code editor to autocomplete the default index.ts file.
/*
* 2) Single step deployment of EKS cluster with the most important variables
*/
const vpc = new awsx.Network("vpc", {usePrivateSubnets: false });
const cluster = new eks.Cluster("eks-cluster", {
vpcId : vpc.vpcId,
subnetIds : vpc.publicSubnetIds,
instanceType : "t2.micro",
nodeRootVolumeSize: 200,
desiredCapacity : 10,
maxSize : 10,
minSize : 3,
deployDashboard : false,
vpcCniOptions : {
warmIpTarget : 4,
},
roleMappings : [
// Provides full administrator cluster access to the k8s cluster
{
groups : ["system:masters"],
roleArn : clusterAdminRole.arn,
username : "admin-usr",
},
// Map IAM rolearn "AutomationRoleArn" to the k8s user with name "automation-usr", e.g. gitlab CI
{
groups : ["pulumi:automation-grp"],
roleArn : AutomationRole.arn,
username : "automation-usr",
},
// Map IAM role arn "EnvProdRoleArn" to the k8s user with name "prod-usr"
{
groups : ["pulumi:prod-grp"],
roleArn : EnvProdRole.arn,
username : "prod-usr",
},
],
});
export const clusterName = cluster.eksCluster.name;
function createNewNamespace(name: string): k8s.core.v1.Namespace {
//Create new namespace
return new k8s.core.v1.Namespace(name, { metadata: { name: name } }, { provider: cluster.provider });
}
//declare namespaces automation and prod
const automation = createNewNamespace("automation");
const prod = createNewNamespace("prod");
The Kubernetes RBAC API declares four top-level types that can be defined as YAMLs syntaxes: a) Role - represents a set of additive rules within a namespace; b) RoleBinding - grants namespace-wide access to k8s subjects and resources; c) ClusterRole - represents a set of additive rules within the cluster; d) ClusterRoleBinding - grants cluster-wide access to k8s subjects and resources.
You define three k8s users with different privileges in your cluster and test them sequentially: User type1 called admin-usr for users have cluster admin rights
$ cat user1.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ClusterAdminRole
# no namespace needed
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cluster-admin-binding
subjects:
- kind: User
name: "pulumi:admin-usr"
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: ClusterAdminRole
apiGroup: rbac.authorization.k8s.io`
User type2 called automation-usr for users that have permissions to all k8s resources in namespace automation. An e.g would be your CI/CD pipeline
$ cat user2.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: AutomationRole
namespace: automation
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: automation-binding
namespace: automation
subjects:
- kind: User
name: "pulumi:automation-usr"
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: AutomationRole
apiGroup: rbac.authorization.k8s.io
User type 3 called prod-usr for users that have read access to all k8s resources in the namespace prod
$ cat user3.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: EnvProdRole
namespace: prod
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: env-prod-binding
namespace: prod
subjects:
- kind: User
name: "pulumi:prod-usr"
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: EnvProdRole
apiGroup: rbac.authorization.k8s.io`
$ kubectl apply -f user1.yaml && kubectl apply -f user2.yaml && kubectl apply -f user3.yaml`
clusterrole.rbac.authorization.k8s.io/ClusterAdminRole created
clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created
role.rbac.authorization.k8s.io/AutomationRole created
rolebinding.rbac.authorization.k8s.io/automation created
role.rbac.authorization.k8s.io/EnvProdRole created
rolebinding.rbac.authorization.k8s.io/env-prod-binding created
Update your index.ts file with more code as follows:
/*
* 3) Single Step deployment of k8s RBAC configuration for user1, user2 and user3 per our example
*/
// User1 - Grant cluster admin access to all admins
new k8s.rbac.v1.ClusterRole("clusterAdminRole", {
metadata: {
name: "clusterAdminRole",
},
rules: [{
apiGroups: ["*"],
resources: ["*"],
verbs: ["*"],
}]
}, {provider: cluster.provider});
new k8s.rbac.v1.ClusterRoleBinding("cluster-admin-binding", {
metadata: {
name: "cluster-admin-binding",
},
subjects: [{
kind: "User",
name: "pulumi:admin-usr",
}],
roleRef: {
kind: "ClusterRole",
name: "clusterAdminRole",
apiGroup: "rbac.authorization.k8s.io",
},
}, {provider: cluster.provider});
// User2 called automation-usr for users with permissions to all k8s resources in namespace automation
new k8s.rbac.v1.Role("AutomationRole", {
metadata: {
name: "AutomationRole",
},
rules: [{
apiGroups: ["*"],
resources: ["*"],
verbs: ["*"],
}]
}, {provider: cluster.provider});
new k8s.rbac.v1.RoleBinding("automation-binding", {
metadata: {
name: "automation-binding",
},
subjects: [{
kind: "User",
name: "pulumi:automation-usr",
}],
roleRef: {
kind: "Role",
name: "AutomationRole",
apiGroup: "rbac.authorization.k8s.io",
},
}, {provider: cluster.provider});
// User3 called prod-usr for users with read access to all k8s resources in namespace env-prod
new k8s.rbac.v1.Role("EnvProdRole", {
metadata: {
name: "EnvProdRole",
},
rules: [{
apiGroups: ["*"],
resources: ["*"],
verbs: ["get", "watch", "list"],
}],
}, {provider: cluster.provider});
new k8s.rbac.v1.RoleBinding("env-prod-binding", {
metadata: {
name: "env-prod-binding",
},
subjects: [{
kind: "User",
name: "pulumi:prod-usr",
}],
roleRef: {
kind: "Role",
name: "EnvProdRole",
apiGroup: "rbac.authorization.k8s.io",
},
}, {provider: cluster.provider});
You get the three IAM role arns for clusterAdminRole, AutomationRole, EnvProdRole and update the configmap with k8s usernames pulumi:admin-usr, pulumi:automation-usr and pulumi:prod-usr.
mapRoles:
----
- groups:
- system:masters
rolearn: arn:aws:iam::XXXXXXXXXXXX:role/clusterAdminRole
username: pulumi:admin-usr
- groups:
rolearn: arn:aws:iam::XXXXXXXXXXXX:role/AutomationRole
username: pulumi:automation-usr
- groups:
rolearn: arn:aws:iam::XXXXXXXXXXXX:role/EnvProdRole
username: pulumi:prod-usr
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::XXXXXXXXXXXX:role/eksctl-eks-rbac-nd-test-nodegroup-NodeInstanceRole-NP542EG8JX8U
username: system:node:{{EC2PrivateDNSName}}`
This step is not required as you have already updated the EKS ConfigMap at cluster creation time in STEP 2 using “RoleMappings”. Simply run pulumi up with the full index.ts file. Watch all your components come alive simultaneously.
Setting up RBAC on one EKS cluster is a long convoluted sequential process that requires multiple validations along the way. Imagine the complexity involved when working with multiple tools for an environment that requires multiple groups with many users, namespaces, and clusters.
If you run pulumi up with this index.ts file.
$ pulumi stack output kubeconfig | jq > kubeconfig.yaml
$ export KUBECONFIG = kubeconfig.yaml
Assume the IAM role AutomationRole with access to all Kubernetes resources in namespace automation and test if the permissions work.
"users": [
{
"name": "aws",
"user": {
"exec": {
"apiVersion": "client.authentication.k8s.io/v1alpha1",
"args": [
"token",
"-i",
"eks-cluster-eksCluster-196b0de",
"-r",
"arn:aws:iam::xxxxxxxxxxxxxxxx:role/AutomationRole"
],
"command": "aws-iam-authenticator"
}
}
}
]
}
If you run kubectl get po --namespace=automation you will get an output No Resources Found as we did not create any resources in this namespace.
- YAML APOCALYPSE! The YAML configurations that this increased scale inevitably requires becomes cumbersome for most operators.
- CHANGE -- YIKES! Changing a roleRef in a RoleBinding, involves deleting the previous RoleBinding and creating a new one.
- WORKFLOW AUTOMATION: In workflows driven by automation, where infrastructure as code is often seen as the gold standard, this doesn’t fit in very well.
- RBAC AT SCALE: Auditing and managing the current users to see what access they have across one or multiple clusters is hard. You can’t just delete a RoleBinding from a repo and hope for some kind of automated task to manage the change for you.
- NO MORE YAMLs! Configuring YAMLs, operators or custom resources is now a thing in the past! You use Typescript or Javascript to program directly with our cloud SDK and connect all cloud services to your Kubernetes services with a simple reference to the object in your program.
- INCREASED DEVELOPMENT VELOCITY: You intuitively program Kubernetes objects with our SDK abstractions using minimal amount of code within hours instead of months. You “autocomplete” AWS, EKS, Kubernetes specifications within your IDE without understanding the entire API.
- EASY UPDATES: You change deployed k8s resources on one or multiple clusters by updating your typescript file
index.tsand runningpulumi up.The Pulumi console allows you to share your stack with your team in your GitHub, GitLab, or Atlassian-based organization. - WORKFLOW AUTOMATION and RBAC AT SCALE: You can delete or update multiple RoleBindings or Roles from your Pulumi stack source code. As you commit these changes to your repository, you can plan automated triggers that validate such changes as part of your CI/CD flow, whether you use Travis, CircleCI, AzureDevOps, etc. Pulumi even has a GitHub Application for surfacing results within pull requests.
In this post, we discussed how setting up Kubernetes RBAC with Pulumi is simple, comprehensive, non-sequential and part of your everyday programming experience instead of being a YAML and DSL tool chain drudgery. You can find the complete pulumi code for our example here. For more examples visit our GitHub examples page here.