The power of three-Running a Graviton2 based Amazon EKS cluster with Bottlerocket OS

“the power of three” by Shenghung Lin is licensed under CC BY-NC-ND 2.0

This blog (with less talk and more hands-on) will focus on the power of three coming together !!

1. Graviton2 -AWS Graviton processors are custom built by Amazon Web Services using 64-bit Arm Neoverse cores to deliver the best price performance for your cloud workloads running in Amazon EC2. AWS Graviton2 processors deliver a major leap in performance and capabilities over first-generation AWS Graviton processors.
2. Amazon EKS — Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that makes it easy for you to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. Customers such as Intel, Snap, Intuit, GoDaddy, and Autodesk trust EKS to run their most sensitive and mission critical applications because of its security, reliability, and scalability.
3. and finally, Bottlerocket — Bottlerocket is a free and open-source Linux-based operating system meant for hosting containers. Bottlerocket focuses on security and maintainability, providing a reliable, consistent, and safe platform for container-based workloads. Rather than a package manager that updates individual pieces of software, Bottlerocket downloads a full filesystem image and reboots into it. It can automatically roll back if boot failures occur, and workload failures can trigger manual rollbacks. See this blog on Bottlerocket general availability for more details. AWS-provided Bottlerocket images are available now for both container orchestration platforms from AWS — Amazon EKS (Generally available) and Amazon ECS (Preview).

I have blogged recently about running Java apps on Graviton2 based processors, but on Amazon Linux at https://medium.com/@cmani/running-a-amazon-corretto-based-java-app-on-an-amazon-eks-cluster-powered-by-aws-graviton2-based-abdd1ad1c703

This blog will explore the nitty-gritties of creating an Kubernetes cluster on Amazon EKS, running EKS data plane with EC2 leveraging Graviton2 processors and the NEW Bottlerocket operating system. As always, this blog was triggered by my quest to explore and learn .. It will start from the basics, and then touch upon some common aspects of running a Kubernetes cluster.

My sincere thanks to my colleague and Graviton SME, Scott Malkie, in helping me overcome initial blues with bottlerocket ;-)

Important Links:

1. Bottlerocket project page — https://github.com/bottlerocket-os/bottlerocket
2. Using a Bottlerocket AMI with Amazon EKS — https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART-EKS.md
3. Bottlerocket FAQ — https://aws.amazon.com/bottlerocket/faqs/
4. Amazon EKS bottlerocket documentation— https://docs.aws.amazon.com/eks/latest/userguide/launch-node-bottlerocket.html
5. Amazon EKS on AWS Graviton2 blog — https://aws.amazon.com/blogs/containers/eks-on-graviton-generally-available/

Creating the EKS cluster

Make sure you have the latest eksctl:

$ eksctl version
0.27.0

Get the correct AMI for ARM/Graviton2 and create a keypair:

$ aws ssm get-parameter --region us-west-2 --name "/aws/service/bottlerocket/aws-k8s-1.17/arm64/latest/image_id" --query Parameter.Value --output text
ami-0b844f017a4995324
$ aws ec2 create-key-pair --key-name bottlerocketkey --query 'KeyMaterial' --output text > bottlerocketkey.pem
$ chmod 400 bottlerocketkey.pem

eksctl — config file for creating cluster

We have two configuration options below — One with an SSH access to the admin container and the second option is without the ssh option.

bottle-rocker-ssh.yaml — For test servers, where you need ssh. The admin container has an SSH server that lets you log in as ec2-user using your EC2-registered SSH key. It runs outside of Bottlerocket’s container orchestrator in a separate instance of containerd. Concentrate on the highlighted elements ..

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: bottlerocket-graviton2
  region: us-west-2
  version: '1.17'
nodeGroups:
  - name: ng-bottlerocket1
    ami: ami-0b844f017a4995324
    instanceType: m6g.large
    desiredCapacity: 3
    amiFamily: Bottlerocket
    iam:
       attachPolicyARNs:
          - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
          - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    ssh:
        allow: true
        publicKeyName: bottlerocketkey
    bottlerocket:
      enableAdminContainer: true
      settings:
        motd: "Hello from eksctl!"

bottle-rocker-withoutssh.yaml (without ssh)

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: bottlerocket-graviton2-nossh
  region: us-west-2
  version: '1.17'
nodeGroups:
  - name: ng-bottlerocket1
    ami: ami-0b844f017a4995324
    instanceType: m6g.large
    desiredCapacity: 3
    amiFamily: Bottlerocket
    iam:
       attachPolicyARNs:
          - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
          - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    bottlerocket:
      enableAdminContainer: true
      settings:
        motd: "Hello from eksctl!"

Create the cluster:

$  eksctl create cluster --config-file ./bottle-rocket-ssh.yaml

Check the worker nodes are provisioned correctly:

Amazon EKS worker nodes

Now, to test a deployment with a sample app for arm based processors like Graviton2, using the plone app as detailed in this blog - https://aws.amazon.com/blogs/containers/eks-on-graviton-generally-available/

plone.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: plone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: plone
  template:
    metadata:
      labels:
        app: plone
    spec:
      containers:
      - name: main
        image: arm64v8/plone:5.2.1
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: plone
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: plone

and check the app is working ..

$ kubectl apply -f plone.yaml
$ kubectl get deploy,po,svc
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/plone   1/1     1            1           17m
NAME                         READY   STATUS    RESTARTS   AGE
pod/plone-758cf48666-dckgq   1/1     Running   0          17m
NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.100.0.1     <none>        443/TCP   169m
service/plone        ClusterIP   10.100.65.23   <none>        80/TCP    17m

Running a Amazon Corretto OpenJDK app

How can I complete a deployment without running my favorite hello world Java app on Bottlerocket ?? ;-)

I have blogged about running Amazon Corretto based Spring boot java app a few times here https://medium.com/@cmani/running-a-amazon-corretto-based-java-app-on-an-amazon-eks-cluster-powered-by-aws-graviton2-based-abdd1ad1c703 and at https://medium.com/@cmani/java-apps-on-amazon-corretto-aws-ec2-a1-instances-on-amazon-elastic-container-services-is-an-1a0c0d127fc2 (on Amazon ECS). Hence, I will not go into the details of this simple spring boot app and the github repo ..

The following is the code of my simple app, which is meant to spit out the details of the environment that its running:

package hello;
  
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@SpringBootApplication
@RestController
public class Application {
@RequestMapping("/")
public String home() {
String str = "Hello from Amazon Corretto running on inside Amazon EKS on AWS Graviton2 processors and the new Bottlerocket Operating System <b
r> The Java Runtime version is " + System.getProperty("java.runtime.version") + " from "+ System.getProperty("java.vm.name") + " and my Java home is " + System
.getProperty("java.home") + "\n <br>";
str = str.concat("Operating System details: ").concat(System.getProperty("os.arch")).concat(" ").concat(System.getProperty("os.name")).concat(
" ").concat(System.getProperty("os.version")).concat("\n <br>");
str = str.concat("Note: This is for demonstration purposes ONLY !! \n");
return str;
   
}
public static void main(String[] args) {
    SpringApplication.run(Application.class, args);
}
}

Build the spring boot app and push it in Amazon ECR, and then deploy it on my test cluster on Amazon EKS with this deployment file:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    app: my-app
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: java
        image: xxx.dkr.ecr.us-west-2.amazonaws.com/java-graviton2:latest
        ports:
        - containerPort: 8080

Accessing the loadbalancer URL, will display the following:

Java app on an Amazon EKS cluster with bottlerocket OS and Graviton2 based processor nodes

The data is similar to what was displayed in my other blog , with the exception of the Linux version, which now displays Linux 5.4.50 for the bottlerocket OS version.

Exploring Bottlerocket worker nodes

To improve security, there’s no SSH server in a Bottlerocket image, and not even a shell !! There are a couple of methods you can use to explore Bottlerocket.

Control container

Bottlerocket has a “control” container, enabled by default, that runs outside of the orchestrator in a separate instance of containerd. This container runs the AWS SSM agent that lets you run commands, or start shell sessions, on Bottlerocket instances in EC2.

You can access the control container either using the AWS Systems Manager via AWS CLI or using the AWS console.

Using the CLI:

$ aws ssm start-session --target "i-07xxxx"

Using the AWS Console for accessing the Systems Manager:

TIP: Before using the AWS Session manager, disable the Run as support option in Session Manager/Preferences screen:

Session Manager Preferences

Start a session to one of the bottlerocket EC2 nodes using the Systems Manager/Session Manager.

Starting session with SessionId: i-0xxx-xxx
Welcome to Bottlerocket's control container!
This container gives you access to the Bottlerocket API, which in turn lets you
inspect and configure the system.  You'll probably want to use the `apiclient`
tool for that; for example, to inspect the system:
apiclient -u /settings
You can run `apiclient --help` for usage details, and check the main
Bottlerocket documentation for descriptions of all settings and examples of
changing them.
If you need to debug the system further, you can enable the admin container.
This enables SSH access to the system using the key you specified when you
launched the instance.  This environment has more debugging tools installed,
and allows you to get root access to the host.
To enable the admin container, run:
enable-admin-container
[ssm-user@ip-192-168-79-83 /]$  apiclient -u /os
{"pretty_name":"Bottlerocket OS 1.0.1","variant_id":"aws-k8s-1.17","version_id":"1.0.1","build_id":"2a181156","arch":"aarch64"}
ssm-user@ip-192-168-79-83 /]$ apiclient -m GET -u /settings
{"motd":"Hello from eksctl!","kubernetes":{"cluster-name":"bottlerocket-graviton2-nossh","cluster-certificate":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENxxxx ....
[ssm-user@ip-192-168-79-83 /]$ apiclient -X PATCH -u /settings -d '{"motd": "my own value!"}'
[ssm-user@ip-192-168-79-83 /]$ apiclient -m POST -u /tx/commit_and_apply
[ssm-user@ip-192-168-79-83 /]$ apiclient -m GET -u /settings | grep motd
{"motd":"my own value!","kubernetes":{"cluster-name":"bottlerocket-graviton2-nossh","cluster-certificate":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJ

Please refer https://github.com/bottlerocket-os/bottlerocket/tree/develop/sources/api/apiclient for mode details.

Admin container

Bottlerocket has an administrative container, disabled by default, that runs outside of the orchestrator in a separate instance of containerd. This container has an SSH server that lets you log in as ec2-user using your EC2-registered SSH key.

$ ssh -i ./bottlerocketkey.pem ec2-user@ec2-xx-xx-xx-17.us-west-2.compute.amazonaws.com     
        
The authenticity of host 'ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com (xx.xx.xx.xx)' can't be established.
ECDSA key fingerprint is SHA256:y3A1G/xxxxxx.
ECDSA key fingerprint is MD5:cd:1e:xx:6a:5b:xx:f4:xx:4a:xx:xx:59:xx:xx:66:f9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com,xx.xx.xx.xx' (ECDSA) to the list of known hosts.
Welcome to Bottlerocket's admin container!
This container provides access to the Bottlerocket host filesystems (see
/.bottlerocket/rootfs) and contains common tools for inspection and
troubleshooting.  It is based on Amazon Linux 2, and most things are in the
same places you would find them on an AL2 host.
To permit more intrusive troubleshooting, including actions that mutate the
running state of the Bottlerocket host, we provide a tool called "sheltie"
(`sudo sheltie`).  When run, this tool drops you into a root shell in the
Bottlerocket host's root filesystem.
$ sudo sheltie
bash-5.0# updog check-update
No update available
<< If updates are available, then run the following to update >>
# updog update
<< to collect logs from the bottlerocket worker node >>
bash-5.0# logdog
Running: exec containerd-config containerd --config /etc/containerd/config.toml config dump
Running: exec containerd-config-host containerd --config /etc/host-containerd/config.toml config dump
Running: exec df df -h
Running: exec df-inodes df -hi
Running: exec dmesg dmesg --color=never --nopager
Running: exec iptables-filter iptables -nvL -t filter
Running: exec iptables-nat iptables -nvL -t nat
Running: exec journalctl-boots journalctl --list-boots --no-pager
Running: exec journalctl.errors journalctl -p err -a --no-pager
Running: exec journalctl.log journalctl -a --no-pager
Running: exec proc-mounts cat /proc/mounts
Running: exec settings.json apiclient --method GET --uri /
Running: exec signpost signpost status
Running: exec wicked wicked show all
Running: file os-release /etc/os-release
Running: exec kube-status systemctl status kube* -l --no-pager
logs are at: /tmp/bottlerocket-logs.tar.gz
bash-5.0# exit
exit
ec2-user@ip-192-168-84-23 ~]$ ls -al /.bottlerocket/rootfs/tmp/bottlerocket-logs.tar.gz 
-rw-r--r--. 1 root root 216916 Sep 23 05:17 /.bottlerocket/rootfs/tmp/bottlerocket-logs.tar.gz

You can retrieve the log files to your local machine using ssh, using this tip from the bottlerocket docs:

$ ssh -i ./bottlerocketkey.pem \
> ec2-user@ec2-xx-xxx-xxxx-xx.us-west-2.compute.amazonaws.com \
> "cat /.bottlerocket/rootfs/tmp/bottlerocket-logs.tar.gz" > bottlerocket-logs.tar.gz
$ ls -l bottlerocket-logs.tar.gz
-rw-rw-r-- 1 ec2-user ec2-user 216916 Sep 24 08:55 bottlerocket-logs.tar.gz
$ gunzip bottlerocket-logs.tar.gz 
$ tar tvf bottlerocket-logs.tar 
drwxr-xr-x 0/0               0 2020-09-23 05:17 bottlerocket-logs/
-rw-r--r-- 0/0               0 2020-09-23 05:17 bottlerocket-logs/logdog.errors
-rw-r--r-- 0/0            3673 2020-09-23 05:17 bottlerocket-logs/containerd-config
-rw-r--r-- 0/0            1719 2020-09-23 05:17 bottlerocket-logs/containerd-config-host
-rw-r--r-- 0/0            2217 2020-09-23 05:17 bottlerocket-logs/df
-rw-r--r-- 0/0            2263 2020-09-23 05:17 bottlerocket-logs/df-inodes
-rw-r--r-- 0/0           29548 2020-09-23 05:17 bottlerocket-logs/dmesg
-rw-r--r-- 0/0            2676 2020-09-23 05:17 bottlerocket-logs/iptables-filter
-rw-r--r-- 0/0            7021 2020-09-23 05:17 bottlerocket-logs/iptables-nat
-rw-r--r-- 0/0              94 2020-09-23 05:17 bottlerocket-logs/journalctl-boots
-rw-r--r-- 0/0            1224 2020-09-23 05:17 bottlerocket-logs/journalctl.errors
-rw-r--r-- 0/0         3142269 2020-09-23 05:17 bottlerocket-logs/journalctl.log
-rw-r--r-- 0/0            9478 2020-09-23 05:17 bottlerocket-logs/proc-mounts
-rw-r--r-- 0/0            4077 2020-09-23 05:17 bottlerocket-logs/settings.json
-rw-r--r-- 0/0             271 2020-09-23 05:17 bottlerocket-logs/signpost
-rw-r--r-- 0/0             503 2020-09-23 05:17 bottlerocket-logs/wicked
-rw-r--r-- 0/0             129 2020-09-23 05:17 bottlerocket-logs/os-release
-rw-r--r-- 0/0           10316 2020-09-23 05:17 bottlerocket-logs/kube-status

To do:

I will be updating this blog as I explore other facets of bottlerocket with Amazon EKS …

Finally . The coming together of these three important elements — Amazon EKS, Graviton2 and Bottlerocket, in my personal opinion, is the next stage in the evolution of running containers in a cost-effective, scalable and secure manner.

And of course, all three are available to test in multiple AWS Regions including Asia Pacific — Mumbai region !!

Hope you found the blog useful ..