Quick single-node K3s on AWS

September 22, 2024

I wanted to deploy the quicker, cheaper and lightest Kubernetes instalation on AWS I could think of for developing, teaching, testing CI/CD pipelines, and other things where a expensive, production-ready EKS cluster would be overkill.

Keep in mind this setup is absolutely inappropriate for production.

To keep things cheap I wanted to use a ARM-based AMI. Luckily, the installation is the same.

The k3s documentation says a t4g.nano should be enough, but it usually freezes during instalation. So we'll be using t4g.micro instead. It has 2 cores and 1 GB of RAM which is the recommended config which is also the cheapest instance with 1 GB of RAM costing around 6.25 USD per month. I assume we can't go cheaper.

I tried using Amazon Linux at first, but without success, so I switched to Ubuntu.

I'm using K3s, though MicroK8s could have been a solid alternative. It just annoys me a bit the fact that MicroK8s ONLY runs on Ubuntu.

I'm assuming you already have a VPC and a Subnet – which can easily be created with the AWS VPC Terraform module – and a previously created EC2 key pair.

We’ll need the usual 80, 443, 22 ports and 6443 for the Kubernetes API.

locals {
  vpc_id    = "vpc-"
  subnet_id = "subnet-"
  key_name  = "my-key"
}

data "aws_ami" "ubuntu_arm" {
  most_recent = true
  owners      = ["099720109477"]  # Canonical

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-*"]  # Ubuntu 20.04 ARM AMI
  }

  filter {
    name   = "architecture"
    values = ["arm64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_security_group" "k3s" {
  vpc_id = local.vpc_id

   ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 6443
    to_port     = 6443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_instance" "this" {
  ami                         = data.aws_ami.ubuntu_arm.id

  instance_type               = "t4g.micro"

  subnet_id                   = local.subnet_id
  vpc_security_group_ids = [aws_security_group.k3s.id]

  associate_public_ip_address = true

  tags = {
    Name = "k3s"
  }

  key_name = local.key_name

  user_data = "${file("init.sh")}"
}

output "public_dns" {
  value = aws_instance.this.public_dns
}

During bootstrap, we can fetch the Instance Metadata Service (169.254.169.254) to dynamically grab the instance's public IP and include it as a valid IP, along with 127.0.0.1.

#!/bin/bash

curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s - --advertise-address=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4)

Now we can ssh and cat the configuration file, replacing 127.0.0.1 with the public IP and updating our .kube\config.

ssh the_public_dns -l ubuntu -i "path-to-my\pem.pem"

cat /etc/rancher/k3s/k3s.yaml | sed "s|https://127.0.0.1:6443|https://$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4):6443|"

By default it comes with Traefik for ingress installed, so that's what we will be using for now.

For testing, we'll be deploy a http-echo, a simple "Hello, world!" container from Hashicorp.

Set the host to your instance’s Public IPv4 DNS for now; you can mess with DNS settings later.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: hashicorp/http-echo
          args:
            - -listen=:80
            - -text=hello from k3s
          resources:
            limits:
              cpu: 200m
              memory: 64Mi
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
    - host: ec2-X-X-X-X.compute-1.amazonaws.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: echo
                port:
                  number: 80

Once you’re done, don’t forget to clean up the mess!

terraform destroy