Hardening on Adur

Kubernetes security hardening guide

Sun, 03 Nov 2024 00:00:00 +0000

Kubernetes Attack Surface

Out-of-the-box Kubernetes isn’t secure. The cluster exposes multiple attack vectors that need systematic attention:

API Server - Central control point; unauthorized access = full cluster compromise
etcd - Stores all cluster state including secrets in base64 (not encrypted by default)
Kubelet - Node agent; exposed kubelet API leaks pod info and allows command execution
Container runtime - Breakout vulnerabilities give host access
Network - Pods can communicate freely with any other pod by default
Supply chain - Malicious or vulnerable images introduce backdoors

This guide covers the key hardening measures organized by attack vector.

API Server Hardening

The API server is the most critical component. Secure it first:

Disable anonymous authentication: set --anonymous-auth=false
Enable audit logging: track who did what (covered in detail below)
Restrict access: use firewall rules or security groups to limit who can reach the API server
Enable admission controllers: PodSecurity, NodeRestriction, ResourceQuota, and LimitRanger should all be active
Use OIDC for authentication: integrate with your identity provider instead of relying on client certificates

1
2


# Check current API server flags (on kubeadm clusters)
kubectl -n kube-system get pod kube-apiserver-<node> -o jsonpath='{.spec.containers[0].command}' | tr ',' '\n'

RBAC Best Practices

RBAC is your primary authorization mechanism. Enforce least privilege rigorously.

Key Rules

Never use cluster-admin for workloads – it grants unlimited access
Use Roles (namespaced) over ClusterRoles whenever possible
Avoid wildcards in resource or verb specifications
Bind to ServiceAccounts, not users for automated workloads
Review bindings regularly – permissions accumulate over time

Example: Restricted Deployment Role

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 namespace: production
 name: deployment-manager
rules:
 # Can manage deployments
 - apiGroups: ["apps"]
 resources: ["deployments"]
 verbs: ["get", "list", "watch", "create", "update", "patch"]
 # Can view pods and logs
 - apiGroups: [""]
 resources: ["pods", "pods/log"]
 verbs: ["get", "list", "watch"]
 # Can view services
 - apiGroups: [""]
 resources: ["services"]
 verbs: ["get", "list", "watch"]
 # Explicitly NO access to secrets, configmaps/write, or exec
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 namespace: production
 name: deployment-manager-binding
subjects:
 - kind: ServiceAccount
 name: ci-deployer
 namespace: production
roleRef:
 kind: Role
 name: deployment-manager
 apiGroup: rbac.authorization.k8s.io

Audit existing RBAC with:

1
2
3
4
5
6


# List all cluster-admin bindings
kubectl get clusterrolebindings -o json | \
 jq '.items[] | select(.roleRef.name == "cluster-admin") | {name: .metadata.name, subjects: .subjects}'

# Check what a specific service account can do
kubectl auth can-i --list --as=system:serviceaccount:production:ci-deployer -n production

Pod Security Standards

Kubernetes Pod Security Standards (PSS) define three security restriction levels. Since 1.25, Pod Security Admission (PSA) is the built-in enforcement mechanism (replaces PodSecurityPolicy).

The Three Levels

Level	Description	Use Case
Privileged	No restrictions	System-level workloads (CNI, storage drivers)
Baseline	Blocks known privilege escalations	General-purpose workloads
Restricted	Maximum hardening	Sensitive workloads, multi-tenant clusters

Enforcing Pod Security Standards

Apply standards at the namespace level using labels:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apiVersion: v1
kind: Namespace
metadata:
 name: production
 labels:
 # Enforce restricted: reject pods that violate
 pod-security.kubernetes.io/enforce: restricted
 pod-security.kubernetes.io/enforce-version: latest
 # Warn on baseline violations (shows warning but allows)
 pod-security.kubernetes.io/warn: restricted
 pod-security.kubernetes.io/warn-version: latest
 # Audit: log violations
 pod-security.kubernetes.io/audit: restricted
 pod-security.kubernetes.io/audit-version: latest

Compliant Pod Example

A pod that passes the restricted level:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apiVersion: v1
kind: Pod
metadata:
 name: secure-app
 namespace: production
spec:
 securityContext:
 runAsNonRoot: true
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: app
 image: myregistry.com/app:v1.2.3@sha256:abc123...
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 runAsUser: 1000
 runAsGroup: 1000
 capabilities:
 drop:
 - ALL
 resources:
 limits:
 memory: "256Mi"
 cpu: "500m"
 requests:
 memory: "128Mi"
 cpu: "250m"

Key security settings to always include:

runAsNonRoot: true – never run containers as root
readOnlyRootFilesystem: true – prevent filesystem modifications
allowPrivilegeEscalation: false – block setuid/setgid
capabilities.drop: ["ALL"] – remove all Linux capabilities
seccompProfile.type: RuntimeDefault – apply default seccomp profile
Always specify resource limits to prevent DoS

NetworkPolicies

By default, all pods can communicate with all other pods in a cluster. NetworkPolicies restrict this to only the traffic that is necessary.

Default Deny All

Start every namespace with a default deny policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-all
 namespace: production
spec:
 podSelector: {}
 policyTypes:
 - Ingress
 - Egress

Allow Specific Traffic

Then explicitly allow only what is needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-api-to-db
 namespace: production
spec:
 podSelector:
 matchLabels:
 app: database
 policyTypes:
 - Ingress
 ingress:
 - from:
 - podSelector:
 matchLabels:
 app: api-server
 ports:
 - protocol: TCP
 port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: api-server-egress
 namespace: production
spec:
 podSelector:
 matchLabels:
 app: api-server
 policyTypes:
 - Egress
 egress:
 # Allow DNS resolution
 - to: []
 ports:
 - protocol: UDP
 port: 53
 - protocol: TCP
 port: 53
 # Allow connection to database
 - to:
 - podSelector:
 matchLabels:
 app: database
 ports:
 - protocol: TCP
 port: 5432
 # Allow external HTTPS
 - to:
 - ipBlock:
 cidr: 0.0.0.0/0
 except:
 - 10.0.0.0/8
 - 172.16.0.0/12
 - 192.168.0.0/16
 ports:
 - protocol: TCP
 port: 443

Note: NetworkPolicies require a CNI plugin that supports them (Calico, Cilium, Weave Net). The default kubenet does not enforce NetworkPolicies.

Secrets Management

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with read access to Secrets in a namespace can decode them trivially. Proper secrets management requires additional tooling.

Option 1: Sealed Secrets

Sealed Secrets (by Bitnami) encrypts secrets client-side so they can be safely stored in Git:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Install kubeseal CLI
# Encrypt a secret
kubectl create secret generic db-creds \
 --from-literal=password=supersecret \
 --dry-run=client -o yaml | \
 kubeseal --format yaml > sealed-db-creds.yaml

# The sealed secret can be committed to Git
# Only the cluster's controller can decrypt it
kubectl apply -f sealed-db-creds.yaml

Option 2: External Secrets Operator

External Secrets Operator syncs secrets from external providers (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) into Kubernetes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: db-credentials
 namespace: production
spec:
 refreshInterval: 1h
 secretStoreRef:
 name: aws-secrets-manager
 kind: ClusterSecretStore
 target:
 name: db-credentials
 data:
 - secretKey: password
 remoteRef:
 key: production/database
 property: password

Enable Encryption at Rest

Ensure etcd encrypts secrets at rest:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
 - resources:
 - secrets
 providers:
 - aescbc:
 keys:
 - name: key1
 secret: <base64-encoded-32-byte-key>
 - identity: {}

Audit Logging

Kubernetes audit logs record every request to the API server, providing a detailed trail of who did what and when.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
 # Log all requests to secrets at the Metadata level
 - level: Metadata
 resources:
 - group: ""
 resources: ["secrets"]
 # Log pod exec/attach at RequestResponse level
 - level: RequestResponse
 resources:
 - group: ""
 resources: ["pods/exec", "pods/attach"]
 # Log all write operations at Request level
 - level: Request
 verbs: ["create", "update", "patch", "delete"]
 # Log everything else at Metadata level
 - level: Metadata
 omitStages:
 - RequestReceived

Enable in the API server with:

1
2
3
4
5


--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100

Ship audit logs to your SIEM or log aggregation system (Loki, Elasticsearch) for analysis and alerting.

Image Policies with Kyverno

Kyverno is a policy engine for Kubernetes that validates, mutates, and generates resources based on policies. It is simpler to adopt than OPA Gatekeeper because policies are written as Kubernetes resources rather than Rego.

Require Image Digest and Trusted Registry

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: require-image-digest
spec:
 validationFailureAction: Enforce
 rules:
 - name: require-digest
 match:
 any:
 - resources:
 kinds:
 - Pod
 validate:
 message: "Images must use a digest (@sha256:...) instead of a tag"
 pattern:
 spec:
 containers:
 - image: "*@sha256:*"
 - name: require-trusted-registry
 match:
 any:
 - resources:
 kinds:
 - Pod
 validate:
 message: "Images must come from the trusted registry"
 pattern:
 spec:
 containers:
 - image: "myregistry.com/*"

This policy ensures that only images from your trusted registry with pinned digests are deployed, preventing both supply chain attacks and tag mutability issues.

CIS Kubernetes Benchmark

The CIS Kubernetes Benchmark provides a comprehensive set of security recommendations. Run automated checks with kube-bench:

1
2
3
4
5


# Run CIS benchmark checks
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml

# View results
kubectl logs job/kube-bench

Address findings by priority: critical items first (API server auth, etcd encryption), then high (RBAC, network policies), then medium and low.

Hardening Checklist

Use this checklist as a starting point for securing your cluster:

API server: anonymous auth disabled, audit logging enabled
etcd: encrypted at rest, access restricted to API server only
RBAC: no unnecessary cluster-admin bindings, least privilege enforced
Pod Security: restricted PSS enforced on production namespaces
NetworkPolicies: default deny in all namespaces, explicit allow rules
Secrets: external secrets manager or sealed secrets, encryption at rest
Images: signed images, trusted registry enforcement, digest pinning
Nodes: automatic security updates, CIS-hardened OS
Audit: API server audit logs shipped to SIEM
Monitoring: alerts on RBAC changes, privileged pod creation, exec into pods
Supply chain: vulnerability scanning in CI/CD, admission-time image scanning
Network: TLS between all components, service mesh for mTLS between pods

Security hardening is not a one-time activity. Schedule quarterly reviews to reassess your posture, run CIS benchmarks, review RBAC bindings, and update policies as your cluster evolves.

How I hardened my K3s homelab

Thu, 14 Sep 2023 00:00:00 +0000

Starting point

I have been running K3s on a couple of physical machines at home for a while. I originally set it up to learn Kubernetes without deploying something heavy, and over time I started using it for small projects: a few web apps, monitoring services, things like that. The setup worked fine but I had never sat down to think carefully about security.

A few months ago I decided to do that exercise. No rush, no generic checklist from the internet. Just look at what I had, understand what was wrong, and fix it.

What I found was not catastrophic, but there were quite a few things I did not like. I am writing this down because K3s has its own quirks compared to a standard Kubernetes cluster, and most hardening guides cover kubeadm or managed clusters, not home setups running on a single binary.

What makes K3s different

Before getting into what I did, it is worth understanding what makes K3s distinct from a security standpoint.

K3s packages everything in a single binary: API server, scheduler, controller manager, kubelet, kube-proxy, and containerd. Instead of etcd it uses SQLite by default. The default CNI is Flannel. The default ingress controller is Traefik. All of this makes the installation very simple, but it also means there are active components you might not need, and some design decisions favor ease of use over security.

The main configuration file lives at /etc/rancher/k3s/config.yaml. That is where most of what I am going to touch ends up.

The node before the cluster

Before touching anything Kubernetes-related, the operating system. The cluster runs on Debian, so I started there.

SSH

I had password authentication enabled. That is the first thing to turn off:

1
2
3
4


# /etc/ssh/sshd_config
PasswordAuthentication no
PermitRootLogin no
AllowUsers myuser

I also changed the default port. It does not stop someone determined from finding you, but it eliminates a lot of noise in the logs.

Firewall with nftables

The machine had its network interfaces completely open. With a home K3s cluster, the API server (port 6443) should not be reachable from outside your local network. My basic rules with nftables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# List active ruleset
nft list ruleset

# Restrictive default policy
nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; }

# Allow loopback and established connections
nft add rule inet filter input iifname lo accept
nft add rule inet filter input ct state established,related accept

# SSH only from local network
nft add rule inet filter input ip saddr 192.168.1.0/24 tcp dport 22 accept

# K3s API server only from local network
nft add rule inet filter input ip saddr 192.168.1.0/24 tcp dport 6443 accept

# Kubernetes node ports
nft add rule inet filter input tcp dport 30000-32767 accept

# Traefik (if used as ingress)
nft add rule inet filter input tcp dport { 80, 443 } accept

# ICMP
nft add rule inet filter input icmp type echo-request accept

What surprised me when reviewing this: the API server was listening on all interfaces and was reachable directly from the WAN because the router had a port forward for another service that was dragging along some traffic. Small scare, nothing exploited, but not something I wanted to leave.

Kernel parameters

K3s with the --protect-kernel-defaults flag verifies that certain kernel parameters are set correctly. If they are not, startup fails with a clear message. Better to configure them beforehand:

1
2
3
4
5
6
7


# /etc/sysctl.d/90-k3s-hardening.conf
kernel.panic = 10
kernel.panic_on_oops = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512

1

sysctl --system

K3s configuration

With the node in order, onto the cluster.

Base configuration file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: "0600"
protect-kernel-defaults: true
secrets-encryption: true
kube-apiserver-arg:
 - "anonymous-auth=false"
 - "audit-log-path=/var/log/k3s/audit.log"
 - "audit-log-maxage=30"
 - "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
kube-controller-manager-arg:
 - "terminated-pod-gc-threshold=10"
kubelet-arg:
 - "streaming-connection-idle-timeout=5m"
 - "protect-kernel-defaults=true"
 - "make-iptables-util-chains=true"

The three most important lines here:

secrets-encryption: true enables encryption at rest for Kubernetes secrets. In K3s this is a first-class flag, no need to configure EncryptionConfiguration manually like in kubeadm.
anonymous-auth=false removes anonymous access to the API server.
write-kubeconfig-mode: "0600" enforces restrictive permissions on the kubeconfig that K3s generates at /etc/rancher/k3s/k3s.yaml.

Audit policy

Without audit logs you have no idea what is happening in your cluster. This is a reasonable minimum:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# /etc/rancher/k3s/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
 - level: Metadata
 resources:
 - group: ""
 resources: ["secrets"]
 - level: RequestResponse
 resources:
 - group: ""
 resources: ["pods/exec", "pods/attach"]
 - level: Request
 verbs: ["create", "update", "patch", "delete"]
 - level: None
 users: ["system:kube-proxy"]
 verbs: ["watch"]
 resources:
 - group: ""
 resources: ["endpoints", "services", "services/status"]
 - level: Metadata
 omitStages:
 - RequestReceived

The logs go to /var/log/k3s/audit.log. In my case I collect them with Promtail and send them to Loki, so I can run queries from Grafana when I need to review something.

Kubeconfig

The kubeconfig K3s generates at /etc/rancher/k3s/k3s.yaml has admin credentials. A mistake I spotted in my own setup: I had that file copied to ~/.kube/config with 644 permissions. Any process running under my user could read it.

1
2


# Correct permissions
chmod 600 ~/.kube/config

If you have additional users who need cluster access, create ServiceAccounts or use client certificates with limited RBAC. Do not distribute the admin kubeconfig.

CNI: from Flannel to Cilium

Flannel is the K3s default. It works, but it does not support NetworkPolicies out of the box. That means all pods can communicate with each other without restriction.

I switched to Cilium. The process with K3s requires disabling Flannel first:

1
2
3


# /etc/rancher/k3s/config.yaml (add)
flannel-backend: none
disable-network-policy: true

Then install Cilium with Helm:

1
2
3
4
5
6
7
8
9


helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium \
 --namespace kube-system \
 --set operator.replicas=1 \
 --set ipam.mode=kubernetes \
 --set kubeProxyReplacement=strict \
 --set k8sServiceHost=192.168.1.10 \
 --set k8sServicePort=6443

With kubeProxyReplacement=strict, Cilium also replaces kube-proxy, which gives better performance with eBPF and better network traffic visibility.

After the switch I started applying NetworkPolicies. The first and most important one: default deny in all application namespaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-all
 namespace: apps
spec:
 podSelector: {}
 policyTypes:
 - Ingress
 - Egress

From there, explicitly allowing only what is necessary.

Secrets: what to do when it is just K3s

In a corporate environment you would use Vault or External Secrets Operator backed by AWS Secrets Manager or similar. In a homelab that is overkill. My solution was simpler.

K3s’s encryption at rest already protects secrets in the database. For manifests in Git I use age to encrypt sensitive values before committing them:

1
2


# Encrypt a value
echo "my-real-password" | age -r $(cat ~/.config/age/recipient.txt) | base64

The encrypted value goes into the repository. Deploying it requires a manual decryption step. Less convenient than an automatic operator, but for a homelab it is good enough and does not add operational complexity.

Falco for runtime detection

Everything above is prevention. Falco is detection: it monitors system calls and generates alerts when something suspicious happens inside a container.

I installed it with Helm:

1
2
3
4
5


helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
 --namespace falco \
 --create-namespace \
 --set driver.kind=ebpf

The default rules already cover quite a bit: shell execution in containers, writes to system directories, privilege changes, unexpected network connections. I added some custom rules for my specific use case.

Alerts go to Loki as well. That way I have the API server audit logs and Falco alerts in the same place.

Pod Security Standards

With K3s >= 1.25 you can use Pod Security Admission directly. I applied the baseline level on application namespaces and restricted where possible:

1
2
3
4
5
6
7


apiVersion: v1
kind: Namespace
metadata:
 name: apps
 labels:
 pod-security.kubernetes.io/enforce: baseline
 pod-security.kubernetes.io/warn: restricted

The restricted level is strict: containers cannot run as root, the root filesystem must be read-only, and all Linux capabilities must be dropped. Some of the workloads I had deployed did not comply. I fixed them one by one before raising the enforce level.

What I left for later

Not everything is perfect. There are things I know I should do that I have not gotten to yet.

Rootless K3s: you can run K3s entirely without root, which reduces the impact of a potential privilege escalation. I have looked into it and it has some limitations with certain network features, but for my use case it should work.

Image verification with cosign: signing and verifying images before deploying them. I have a private registry for my own images and the third-party ones I use, but I am not verifying signatures yet.

CIS Benchmark: K3s has official documentation for the CIS K3s benchmark. I have reviewed it partially but have not run kube-bench systematically to see what is still pending.

What I learned

The default K3s installation is surprisingly reasonable for what it is: a distribution designed to be easy to deploy. But “reasonable” is not the same as “hardened.”

What surprised me most was how easy most of the changes turned out to be. K3s has first-class flags for many things that require manual configuration in kubeadm. The secrets-encryption, protect-kernel-defaults, the audit policy directly in the config file. It is not that hard if you sit down and read the documentation.

The biggest risk in a homelab is not that someone on the internet attacks you directly. It is that you have a bunch of services running on the same network as your personal life, and if one gets compromised, the blast radius can be larger than it seems. Worth taking seriously even if it is a toy environment.