Amazon EKS now allows you to assign IAM permissions to Kubernetes service accounts, which in-turns makes it possible to give pod level access to various AWS services. Pod level access is much more fine-grained than having an IAM role assigned at an EC2 Worker node.
This was one of the most requested feature from AWS customers, who earlier were using various open source tools like kube2iam, kiam and others for achieving this. Of course, in the other container orchestration service from AWS — Amazon ECS this level of assigning IAM roles to Tasks has been present for a long time.
There is an amazing AWS blog at https://aws.amazon.com/blogs/opensource/using-pod-security-policies-amazon-eks-clusters/ which shows how to set this up for getting pod level access to Amazon S3. There is also an extensive guide at https://eksctl.io/usage/iamserviceaccounts/ which has more information.
This blog is a quick hands-on to get myself familiar with the nitty-gritty of setting this up .. and I wanted to try something different, getting Pod level access to Amazon DynamoDB using the new IAM Roles for ServiceAccounts feature.
Step 0:
Make sure that you have the latest AWS CLI and eksctl installed on your workstation.
$ aws --version
aws-cli/1.16.234 Python/3.7.0 Darwin/17.7.0 botocore/1.12.224$ eksctl version
[ℹ] version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.5.2"}
Create the EKS cluster in your favorite region, and mine is Mumbai ;-)
$ eksctl create cluster ekscluster-1 --region ap-south-1Step 1:
I created a sample table in DynamoDB and then created an IAM Policy for access to this table. The IAM policy — aws-ddb-policy1 looks like:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "dynamodb:*",
"Resource": "arn:aws:dynamodb:ap-south-1:xxxx:table/MysfitsTable"
}
]
}The next couple of steps are to associate the Open ID Connect identity provider with the EKS cluster and to create an IAM service account with the IAM Policy attached to the EKS cluster.
$ eksctl utils associate-iam-oidc-provider --name ekscluster-1 --approve --region ap-south-1
$ eksctl create iamserviceaccount \
--name ddb-sa-2 \
--cluster ekscluster-1 \
--attach-policy-arn arn:aws:iam::xxxx:policy/aws-ddb-policy1 \
--approve \
--region ap-south-1
$ kubectl get sa
NAME SECRETS AGE
ddb-sa-2 1 99m
default 1 3d3h
s3-echoer 1 46h$ kubectl get sa ddb-sa-2 -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/eksctl-ekscluster-1-addon-iamserviceaccount-Role1-xxx
creationTimestamp: 2019-09-12T08:46:23Z
name: ddb-sa-2
namespace: default
resourceVersion: "372073"
selfLink: /api/v1/namespaces/default/serviceaccounts/ddb-sa-2
uid: xxx-d539-11e9-af15-020f5b903eee
secrets:
- name: ddb-sa-2-token-l952j
I have two simple Pod definitions, one with the Serviceaccount which gives access to the specific DynamoDB table and one without the Serviceaccount .. Ignore the container image that I have used, you can create an Pod with just a Linux image on it.
Pod definition with serviceaccount
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
serviceAccountName: ddb-sa-2
containers:
- name: mythical
image: cmani/mythical-test:latest
env:
- name: AWS_DEFAULT_REGION
value: "ap-south-1"
- name: ENABLE_IRP
value: "true"Pod definition without serviceaccount
apiVersion: v1
kind: Pod
metadata:
name: my-pod-noaccess
spec:
containers:
- name: mythical
image: cmani/mythical-test:latest
env:
- name: AWS_DEFAULT_REGION
value: "ap-south-1"
- name: ENABLE_IRP
value: "true"Create the Pods on the EKS cluster.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-pod 1/1 Running 0 81m
my-pod-noaccess 1/1 Running 0 81mStep 2:
The moment of reckoning, we will do a very basic test to check connectivity from within the pods to access the DynamoDB table.
The following commands maybe useful to you to install the few utilities like curl, unzip and the AWS CLI needed for testing inside the pod, this will not be required in the real world ..
$ kubectl exec -it my-pod -- bash$ apt-get update && apt-get install curl unzip -y
$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ ./awscli-bundle/install -b ~/bin/aws
The Pod with service account
Access is possible when trying to access DynamoDB tables as per the IAM policy.
$ kubectl exec -it my-pod -- bash# /root/bin/aws dynamodb describe-table --table-name MysfitsTable --region ap-south-1{
"Table": {
"TableArn": "arn:aws:dynamodb:ap-south-1:xxxx:table/MysfitsTable",
"AttributeDefinitions": [
{
"AttributeName": "GoodEvil",
"AttributeType": "S"
},
{
"AttributeName": "LawChaos",
"AttributeType": "S"
},
{
"AttributeName": "MysfitId",
"AttributeType": "S"
}
],
"GlobalSecondaryIndexes": [
{
"IndexSizeBytes": 9068,
"IndexName": "LawChaosIndex",
"Projection": {# /root/bin/aws dynamodb scan --table-name MysfitsTable --region ap-south-1
{
"Count": 12,
"Items": [
{
"Description": {
"S":
The Pod without service account
Access is denied when trying to access DynamoDB tables.
$ kubectl exec -it my-pod-noaccess -- bash# /root/bin/aws dynamodb describe-table --table-name MysfitsTable --region ap-south-1
An error occurred (AccessDeniedException) when calling the DescribeTable operation: User: arn:aws:sts::xx:assumed-role/eksctl-ekscluster-1-nodegroup-ng-NodeInstanceRole-xxx/i-xxxx is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:ap-south-1:xxx:table/MysfitsTable# /root/bin/aws dynamodb scan --table-name MysfitsTable --region ap-south-1
An error occurred (AccessDeniedException) when calling the Scan operation: User: arn:aws:sts::xxxx:assumed-role/eksctl-ekscluster-1-nodegroup-ng-NodeInstanceRole-xxx/i-xxxx is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:ap-south-1:xxxxx:table/MysfitsTable
Finally
This concludes this short blog, and this feature will be extremely useful in ensuring that Pods on Kubernetes running on Amazon EKS have access to only the required AWS services and more importantly use the very familiar IAM policies to control the access. !!
