Homelab on Adur

Building an IaC repository for the homelab

Mon, 20 Nov 2023 00:00:00 +0000

What triggered this

A couple of months ago I spent time hardening the K3s cluster. I went through an entire weekend changing configurations, tuning kernel parameters, swapping Flannel for Cilium, writing network policies. By the end the cluster was in much better shape.

But I had done all of it by hand.

If I need to recreate that node from scratch tomorrow, how long does it take? Probably two or three days digging through my own notes scattered across text files, terminal history, and chat messages. And I would still miss things. That is not sustainable.

So I decided to build a proper infrastructure repository. Not a demo, not a proof of concept — the repository where the definition of everything I run at home lives, sanitized enough to publish.

What the homelab looks like

Before talking about the repository structure, it helps to explain what needs to be managed. The setup is:

A main server running Proxmox where several VMs live
Two additional physical nodes forming the K3s cluster
A router running OpenWrt
A NAS running TrueNAS

It is not a huge environment, but it has enough variety that managing everything by hand is a real problem. Especially because Proxmox, the VMs, the cluster, and the NAS have configurations that interact with each other: IPs, internal DNS, certificates, users.

Repository structure

After a few experiments, this is the layout that works for me:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


homelab-infra/
├── terraform/
│ ├── modules/
│ │ ├── proxmox-vm/
│ │ ├── dns-record/
│ │ └── network-vlan/
│ └── environments/
│ ├── main.tf
│ ├── variables.tf
│ └── terraform.tfvars.example
├── ansible/
│ ├── inventory/
│ │ ├── hosts.yml
│ │ └── group_vars/
│ ├── playbooks/
│ │ ├── bootstrap.yml
│ │ ├── k3s-server.yml
│ │ ├── k3s-agent.yml
│ │ └── hardening.yml
│ └── roles/
│ ├── common/
│ ├── cis-level1/
│ └── k3s/
├── kubernetes/
│ ├── base/
│ ├── apps/
│ │ ├── monitoring/
│ │ ├── storage/
│ │ └── networking/
│ └── policies/
│ ├── gatekeeper/
│ └── seccomp/
├── .gitlab-ci.yml
├── .sops.yaml
└── README.md

Three clearly separated layers: provisioning (Terraform), node configuration (Ansible), and Kubernetes workloads. Security policies live inside kubernetes/policies/ because they are Kubernetes resources, but I treat them as a conceptually distinct layer.

Terraform: provisioning with Proxmox

The Proxmox provider for Terraform is the Telmate one (telmate/proxmox). It is not official, but it is the most widely used and works reasonably well.

The proxmox-vm module wraps VM creation with the parameters I use regularly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# terraform/modules/proxmox-vm/main.tf
resource "proxmox_vm_qemu" "vm" {
 name = var.name
 target_node = var.target_node
 clone = var.template
 full_clone = true

 cores = var.cores
 memory = var.memory
 sockets = 1

 disk {
 size = var.disk_size
 type = "virtio"
 storage = var.storage_pool
 discard = "on"
 }

 network {
 model = "virtio"
 bridge = var.network_bridge
 tag = var.vlan_tag
 }

 ipconfig0 = "ip=${var.ip_address}/24,gw=${var.gateway}"
 nameserver = var.nameserver
 searchdomain = var.searchdomain

 ciuser = var.ssh_user
 sshkeys = var.ssh_public_key

 lifecycle {
 ignore_changes = [
 network,
 ]
 }
}

Sensitive variables — the Proxmox API token, SSH keys — are not in the repository in plaintext. I use SOPS to encrypt the terraform.tfvars file with age:

1
2
3
4
5
6


# .sops.yaml
creation_rules:
 - path_regex: .*\.tfvars$
 age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 - path_regex: ansible/inventory/group_vars/.*\.yml$
 age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The encrypted file (terraform.tfvars) is committed to Git. Decrypting it requires the age private key, which lives on the CI server and my local machine — never in the repository.

Ansible: node configuration

Once Terraform provisions the VMs, Ansible configures them. The bootstrap.yml playbook does the minimum needed to get a freshly created node into working shape:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ansible/playbooks/bootstrap.yml
---
- name: Bootstrap new nodes
 hosts: all
 become: true
 roles:
 - common
 - cis-level1

- name: Configure K3s server nodes
 hosts: k3s_servers
 become: true
 roles:
 - k3s

The common role installs base packages, configures NTP, hardens SSH, sets sysctl parameters, and creates system users. The cis-level1 role applies CIS Benchmark Level 1 recommendations for Debian.

I did not write the CIS role from scratch. I started from the community role dev-sec/ansible-collection-hardening and adapted it. There are quite a few tasks the default role applies that do not fit a homelab — things designed for production servers with strict audit requirements. I went through each task, understood what it did, and decided if it applied to my case.

Some things I disabled:

1
2
3
4
5


# ansible/roles/cis-level1/defaults/main.yml
os_auth_pam_pwquality_enable: false # No local users with passwords
os_security_users_allow: ["vagrant"] # Dev environment only
os_filesystem_whitelist:
 - vfat  # Required for UEFI boot

And some things I added specifically for K3s:

1
2
3
4
5
6
7
8


# Kernel parameters required by K3s with protect-kernel-defaults
kernel_parameters:
 - { name: "kernel.panic", value: "10" }
 - { name: "kernel.panic_on_oops", value: "1" }
 - { name: "vm.overcommit_memory", value: "1" }
 - { name: "vm.panic_on_oom", value: "0" }
 - { name: "fs.inotify.max_user_watches", value: "524288" }
 - { name: "fs.inotify.max_user_instances", value: "512" }

Kubernetes: manifests with Kustomize

For Kubernetes manifests I use Kustomize over Helm where possible. Helm is more powerful for complex things, but for my own applications Kustomize is sufficient and produces readable YAML.

The basic Kustomize structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubernetes/apps/monitoring/
├── base/
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ ├── prometheus-deployment.yaml
│ └── grafana-deployment.yaml
└── overlays/
 └── homelab/
 ├── kustomization.yaml
 └── patches/
 └── resource-limits.yaml

The homelab overlay adds environment-specific settings without modifying the base manifests:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# kubernetes/apps/monitoring/overlays/homelab/patches/resource-limits.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: prometheus
 namespace: monitoring
spec:
 template:
 spec:
 containers:
 - name: prometheus
 resources:
 requests:
 memory: "256Mi"
 cpu: "100m"
 limits:
 memory: "512Mi"
 cpu: "500m"

OPA/Gatekeeper: policies as code

Gatekeeper is a Kubernetes admission controller that uses OPA (Open Policy Agent) to evaluate policies written in Rego. Instead of letting any pod deploy with any configuration, policies reject manifests that do not meet security requirements.

Active policies:

No containers running as root

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# kubernetes/policies/gatekeeper/no-root-containers.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPAllowedUsers
metadata:
 name: psp-pods-allowed-user-ranges
spec:
 match:
 kinds:
 - apiGroups: [""]
 kinds: ["Pod"]
 excludedNamespaces:
 - kube-system
 - falco
 parameters:
 runAsUser:
 rule: MustRunAsNonRoot
 runAsGroup:
 rule: MustRunAs
 ranges:
 - min: 1000
 max: 65535

Required resource limits

Without resource limits, a pod can consume all the node’s memory and bring down the cluster. This policy prevents that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# kubernetes/policies/gatekeeper/require-resource-limits.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
 name: k8srequiredresources
spec:
 crd:
 spec:
 names:
 kind: K8sRequiredResources
 targets:
 - target: admission.k8s.gatekeeper.sh
 rego: |
 package k8srequiredresources

 violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not container.resources.limits.memory
 msg := sprintf("Container '%v' has no memory limit defined", [container.name])
 }

 violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not container.resources.limits.cpu
 msg := sprintf("Container '%v' has no CPU limit defined", [container.name])
 }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
 name: require-resource-limits
spec:
 match:
 kinds:
 - apiGroups: [""]
 kinds: ["Pod"]
 excludedNamespaces:
 - kube-system

Trusted image registries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


# kubernetes/policies/gatekeeper/allowed-registries.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
 name: k8sallowedrepos
spec:
 crd:
 spec:
 names:
 kind: K8sAllowedRepos
 validation:
 openAPIV3Schema:
 type: object
 properties:
 repos:
 type: array
 items:
 type: string
 targets:
 - target: admission.k8s.gatekeeper.sh
 rego: |
 package k8sallowedrepos

 violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not any_repo_matches(container.image)
 msg := sprintf("Image '%v' does not come from an allowed registry", [container.image])
 }

 any_repo_matches(image) {
 repo := input.parameters.repos[_]
 startswith(image, repo)
 }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
 name: allowed-registries
spec:
 match:
 kinds:
 - apiGroups: [""]
 kinds: ["Pod"]
 excludedNamespaces:
 - kube-system
 parameters:
 repos:
 - "registry.homelab.internal/"
 - "ghcr.io/my-user/"
 - "quay.io/prometheus/"
 - "grafana/"

Seccomp profiles

Seccomp profiles restrict which system calls a container can make. Kubernetes has a default profile (RuntimeDefault) that is already reasonable, but for applications I know well I define tighter profiles.

The profiles live in the repository and are deployed as ConfigMaps or directly to the nodes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


// kubernetes/policies/seccomp/web-app-profile.json
{
 "defaultAction": "SCMP_ACT_ERRNO",
 "architectures": ["SCMP_ARCH_X86_64"],
 "syscalls": [
 {
 "names": [
 "accept4", "bind", "brk", "clone", "close", "connect",
 "epoll_create1", "epoll_ctl", "epoll_wait", "execve",
 "exit_group", "fcntl", "fstat", "futex", "getdents64",
 "getpid", "getsockname", "getsockopt", "listen", "lstat",
 "mmap", "mprotect", "munmap", "nanosleep", "newfstatat",
 "openat", "poll", "prctl", "read", "recvfrom", "rt_sigaction",
 "rt_sigprocmask", "rt_sigreturn", "sendto", "set_robust_list",
 "setsockopt", "sigaltstack", "socket", "stat", "write"
 ],
 "action": "SCMP_ACT_ALLOW"
 }
 ]
}

Referenced from the pod spec:

1
2
3
4
5


spec:
 securityContext:
 seccompProfile:
 type: Localhost
 localhostProfile: "web-app-profile.json"

Building a seccomp profile from scratch is tedious. My approach is to start with RuntimeDefault, use strace to see what syscalls the application actually makes, and then build a tighter profile for workloads I want to restrict further.

CI/CD pipeline

The repository has a GitLab CI pipeline that automates applying changes. The flow is:

On merge requests: terraform plan and ansible-lint to catch problems before merging
On merge to main: terraform apply and, if Ansible files changed, the corresponding playbook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


# .gitlab-ci.yml (excerpt)
stages:
 - validate
 - plan
 - apply

variables:
 TF_ROOT: "${CI_PROJECT_DIR}/terraform/environments"
 ANSIBLE_CONFIG: "${CI_PROJECT_DIR}/ansible/ansible.cfg"

terraform-validate:
 stage: validate
 image: hashicorp/terraform:1.6
 script:
 - cd "$TF_ROOT"
 - terraform init -backend=false
 - terraform validate
 rules:
 - changes:
 - terraform/**/*

terraform-plan:
 stage: plan
 image: hashicorp/terraform:1.6
 script:
 - cd "$TF_ROOT"
 - terraform init
 - terraform plan -out=tfplan
 artifacts:
 paths:
 - "${TF_ROOT}/tfplan"
 expire_in: 1 week
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 changes:
 - terraform/**/*

terraform-apply:
 stage: apply
 image: hashicorp/terraform:1.6
 script:
 - cd "$TF_ROOT"
 - terraform init
 - terraform apply -auto-approve
 rules:
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
 changes:
 - terraform/**/*
 when: manual

ansible-lint:
 stage: validate
 image: python:3.11-slim
 script:
 - pip install ansible ansible-lint
 - ansible-lint ansible/
 rules:
 - changes:
 - ansible/**/*

The terraform apply is manual — I do not want infrastructure changing automatically without my approval. The plan runs automatically on the MR for visibility.

What I sanitized

Publishing the repository required reviewing what should not be there:

Internal IPs: replaced with example ranges (192.168.1.x)
Domain names: the internal homelab domain (homelab.internal in the repo, something different in production)
Usernames: real usernames are not in the repository
SSH public keys: replaced with placeholders
Password hashes: removed from the Ansible inventory
Application secrets: encrypted with SOPS or removed, with an .example file alongside

The rule I followed: if someone with access to my local network could use that information to attack something, it does not go into the repository in plaintext. Everything else can be there.

What is still pending

There are still things I manage by hand that should be codified:

OpenWrt: the router configuration is the hardest to bring into IaC. There is a Terraform module for OpenWrt that is not very well maintained. For now I manage it with an Ansible script that backs up the configuration and another that restores it. Not idempotent, but it works.

TrueNAS: it has a fairly complete REST API. There is a Terraform provider in development. It is on my radar for the next iteration.

Backups: I have backups, but the process is not in the repository. It lives in another loose script that will eventually end up here.

The repository is never “finished.” What matters is that the current state of the homelab is represented in it, and that any change goes through Git.

How I hardened my K3s homelab

Thu, 14 Sep 2023 00:00:00 +0000

Starting point

I have been running K3s on a couple of physical machines at home for a while. I originally set it up to learn Kubernetes without deploying something heavy, and over time I started using it for small projects: a few web apps, monitoring services, things like that. The setup worked fine but I had never sat down to think carefully about security.

A few months ago I decided to do that exercise. No rush, no generic checklist from the internet. Just look at what I had, understand what was wrong, and fix it.

What I found was not catastrophic, but there were quite a few things I did not like. I am writing this down because K3s has its own quirks compared to a standard Kubernetes cluster, and most hardening guides cover kubeadm or managed clusters, not home setups running on a single binary.

What makes K3s different

Before getting into what I did, it is worth understanding what makes K3s distinct from a security standpoint.

K3s packages everything in a single binary: API server, scheduler, controller manager, kubelet, kube-proxy, and containerd. Instead of etcd it uses SQLite by default. The default CNI is Flannel. The default ingress controller is Traefik. All of this makes the installation very simple, but it also means there are active components you might not need, and some design decisions favor ease of use over security.

The main configuration file lives at /etc/rancher/k3s/config.yaml. That is where most of what I am going to touch ends up.

The node before the cluster

Before touching anything Kubernetes-related, the operating system. The cluster runs on Debian, so I started there.

SSH

I had password authentication enabled. That is the first thing to turn off:

1
2
3
4


# /etc/ssh/sshd_config
PasswordAuthentication no
PermitRootLogin no
AllowUsers myuser

I also changed the default port. It does not stop someone determined from finding you, but it eliminates a lot of noise in the logs.

Firewall with nftables

The machine had its network interfaces completely open. With a home K3s cluster, the API server (port 6443) should not be reachable from outside your local network. My basic rules with nftables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# List active ruleset
nft list ruleset

# Restrictive default policy
nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; }

# Allow loopback and established connections
nft add rule inet filter input iifname lo accept
nft add rule inet filter input ct state established,related accept

# SSH only from local network
nft add rule inet filter input ip saddr 192.168.1.0/24 tcp dport 22 accept

# K3s API server only from local network
nft add rule inet filter input ip saddr 192.168.1.0/24 tcp dport 6443 accept

# Kubernetes node ports
nft add rule inet filter input tcp dport 30000-32767 accept

# Traefik (if used as ingress)
nft add rule inet filter input tcp dport { 80, 443 } accept

# ICMP
nft add rule inet filter input icmp type echo-request accept

What surprised me when reviewing this: the API server was listening on all interfaces and was reachable directly from the WAN because the router had a port forward for another service that was dragging along some traffic. Small scare, nothing exploited, but not something I wanted to leave.

Kernel parameters

K3s with the --protect-kernel-defaults flag verifies that certain kernel parameters are set correctly. If they are not, startup fails with a clear message. Better to configure them beforehand:

1
2
3
4
5
6
7


# /etc/sysctl.d/90-k3s-hardening.conf
kernel.panic = 10
kernel.panic_on_oops = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512

1

sysctl --system

K3s configuration

With the node in order, onto the cluster.

Base configuration file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: "0600"
protect-kernel-defaults: true
secrets-encryption: true
kube-apiserver-arg:
 - "anonymous-auth=false"
 - "audit-log-path=/var/log/k3s/audit.log"
 - "audit-log-maxage=30"
 - "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
kube-controller-manager-arg:
 - "terminated-pod-gc-threshold=10"
kubelet-arg:
 - "streaming-connection-idle-timeout=5m"
 - "protect-kernel-defaults=true"
 - "make-iptables-util-chains=true"

The three most important lines here:

secrets-encryption: true enables encryption at rest for Kubernetes secrets. In K3s this is a first-class flag, no need to configure EncryptionConfiguration manually like in kubeadm.
anonymous-auth=false removes anonymous access to the API server.
write-kubeconfig-mode: "0600" enforces restrictive permissions on the kubeconfig that K3s generates at /etc/rancher/k3s/k3s.yaml.

Audit policy

Without audit logs you have no idea what is happening in your cluster. This is a reasonable minimum:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# /etc/rancher/k3s/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
 - level: Metadata
 resources:
 - group: ""
 resources: ["secrets"]
 - level: RequestResponse
 resources:
 - group: ""
 resources: ["pods/exec", "pods/attach"]
 - level: Request
 verbs: ["create", "update", "patch", "delete"]
 - level: None
 users: ["system:kube-proxy"]
 verbs: ["watch"]
 resources:
 - group: ""
 resources: ["endpoints", "services", "services/status"]
 - level: Metadata
 omitStages:
 - RequestReceived

The logs go to /var/log/k3s/audit.log. In my case I collect them with Promtail and send them to Loki, so I can run queries from Grafana when I need to review something.

Kubeconfig

The kubeconfig K3s generates at /etc/rancher/k3s/k3s.yaml has admin credentials. A mistake I spotted in my own setup: I had that file copied to ~/.kube/config with 644 permissions. Any process running under my user could read it.

1
2


# Correct permissions
chmod 600 ~/.kube/config

If you have additional users who need cluster access, create ServiceAccounts or use client certificates with limited RBAC. Do not distribute the admin kubeconfig.

CNI: from Flannel to Cilium

Flannel is the K3s default. It works, but it does not support NetworkPolicies out of the box. That means all pods can communicate with each other without restriction.

I switched to Cilium. The process with K3s requires disabling Flannel first:

1
2
3


# /etc/rancher/k3s/config.yaml (add)
flannel-backend: none
disable-network-policy: true

Then install Cilium with Helm:

1
2
3
4
5
6
7
8
9


helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium \
 --namespace kube-system \
 --set operator.replicas=1 \
 --set ipam.mode=kubernetes \
 --set kubeProxyReplacement=strict \
 --set k8sServiceHost=192.168.1.10 \
 --set k8sServicePort=6443

With kubeProxyReplacement=strict, Cilium also replaces kube-proxy, which gives better performance with eBPF and better network traffic visibility.

After the switch I started applying NetworkPolicies. The first and most important one: default deny in all application namespaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-all
 namespace: apps
spec:
 podSelector: {}
 policyTypes:
 - Ingress
 - Egress

From there, explicitly allowing only what is necessary.

Secrets: what to do when it is just K3s

In a corporate environment you would use Vault or External Secrets Operator backed by AWS Secrets Manager or similar. In a homelab that is overkill. My solution was simpler.

K3s’s encryption at rest already protects secrets in the database. For manifests in Git I use age to encrypt sensitive values before committing them:

1
2


# Encrypt a value
echo "my-real-password" | age -r $(cat ~/.config/age/recipient.txt) | base64

The encrypted value goes into the repository. Deploying it requires a manual decryption step. Less convenient than an automatic operator, but for a homelab it is good enough and does not add operational complexity.

Falco for runtime detection

Everything above is prevention. Falco is detection: it monitors system calls and generates alerts when something suspicious happens inside a container.

I installed it with Helm:

1
2
3
4
5


helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
 --namespace falco \
 --create-namespace \
 --set driver.kind=ebpf

The default rules already cover quite a bit: shell execution in containers, writes to system directories, privilege changes, unexpected network connections. I added some custom rules for my specific use case.

Alerts go to Loki as well. That way I have the API server audit logs and Falco alerts in the same place.

Pod Security Standards

With K3s >= 1.25 you can use Pod Security Admission directly. I applied the baseline level on application namespaces and restricted where possible:

1
2
3
4
5
6
7


apiVersion: v1
kind: Namespace
metadata:
 name: apps
 labels:
 pod-security.kubernetes.io/enforce: baseline
 pod-security.kubernetes.io/warn: restricted

The restricted level is strict: containers cannot run as root, the root filesystem must be read-only, and all Linux capabilities must be dropped. Some of the workloads I had deployed did not comply. I fixed them one by one before raising the enforce level.

What I left for later

Not everything is perfect. There are things I know I should do that I have not gotten to yet.

Rootless K3s: you can run K3s entirely without root, which reduces the impact of a potential privilege escalation. I have looked into it and it has some limitations with certain network features, but for my use case it should work.

Image verification with cosign: signing and verifying images before deploying them. I have a private registry for my own images and the third-party ones I use, but I am not verifying signatures yet.

CIS Benchmark: K3s has official documentation for the CIS K3s benchmark. I have reviewed it partially but have not run kube-bench systematically to see what is still pending.

What I learned

The default K3s installation is surprisingly reasonable for what it is: a distribution designed to be easy to deploy. But “reasonable” is not the same as “hardened.”

What surprised me most was how easy most of the changes turned out to be. K3s has first-class flags for many things that require manual configuration in kubeadm. The secrets-encryption, protect-kernel-defaults, the audit policy directly in the config file. It is not that hard if you sit down and read the documentation.

The biggest risk in a homelab is not that someone on the internet attacks you directly. It is that you have a bunch of services running on the same network as your personal life, and if one gets compromised, the blast radius can be larger than it seems. Worth taking seriously even if it is a toy environment.