Ansible on Adur

Building an IaC repository for the homelab

Mon, 20 Nov 2023 00:00:00 +0000

What triggered this

A couple of months ago I spent time hardening the K3s cluster. I went through an entire weekend changing configurations, tuning kernel parameters, swapping Flannel for Cilium, writing network policies. By the end the cluster was in much better shape.

But I had done all of it by hand.

If I need to recreate that node from scratch tomorrow, how long does it take? Probably two or three days digging through my own notes scattered across text files, terminal history, and chat messages. And I would still miss things. That is not sustainable.

So I decided to build a proper infrastructure repository. Not a demo, not a proof of concept — the repository where the definition of everything I run at home lives, sanitized enough to publish.

What the homelab looks like

Before talking about the repository structure, it helps to explain what needs to be managed. The setup is:

A main server running Proxmox where several VMs live
Two additional physical nodes forming the K3s cluster
A router running OpenWrt
A NAS running TrueNAS

It is not a huge environment, but it has enough variety that managing everything by hand is a real problem. Especially because Proxmox, the VMs, the cluster, and the NAS have configurations that interact with each other: IPs, internal DNS, certificates, users.

Repository structure

After a few experiments, this is the layout that works for me:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


homelab-infra/
├── terraform/
│ ├── modules/
│ │ ├── proxmox-vm/
│ │ ├── dns-record/
│ │ └── network-vlan/
│ └── environments/
│ ├── main.tf
│ ├── variables.tf
│ └── terraform.tfvars.example
├── ansible/
│ ├── inventory/
│ │ ├── hosts.yml
│ │ └── group_vars/
│ ├── playbooks/
│ │ ├── bootstrap.yml
│ │ ├── k3s-server.yml
│ │ ├── k3s-agent.yml
│ │ └── hardening.yml
│ └── roles/
│ ├── common/
│ ├── cis-level1/
│ └── k3s/
├── kubernetes/
│ ├── base/
│ ├── apps/
│ │ ├── monitoring/
│ │ ├── storage/
│ │ └── networking/
│ └── policies/
│ ├── gatekeeper/
│ └── seccomp/
├── .gitlab-ci.yml
├── .sops.yaml
└── README.md

Three clearly separated layers: provisioning (Terraform), node configuration (Ansible), and Kubernetes workloads. Security policies live inside kubernetes/policies/ because they are Kubernetes resources, but I treat them as a conceptually distinct layer.

Terraform: provisioning with Proxmox

The Proxmox provider for Terraform is the Telmate one (telmate/proxmox). It is not official, but it is the most widely used and works reasonably well.

The proxmox-vm module wraps VM creation with the parameters I use regularly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# terraform/modules/proxmox-vm/main.tf
resource "proxmox_vm_qemu" "vm" {
 name = var.name
 target_node = var.target_node
 clone = var.template
 full_clone = true

 cores = var.cores
 memory = var.memory
 sockets = 1

 disk {
 size = var.disk_size
 type = "virtio"
 storage = var.storage_pool
 discard = "on"
 }

 network {
 model = "virtio"
 bridge = var.network_bridge
 tag = var.vlan_tag
 }

 ipconfig0 = "ip=${var.ip_address}/24,gw=${var.gateway}"
 nameserver = var.nameserver
 searchdomain = var.searchdomain

 ciuser = var.ssh_user
 sshkeys = var.ssh_public_key

 lifecycle {
 ignore_changes = [
 network,
 ]
 }
}

Sensitive variables — the Proxmox API token, SSH keys — are not in the repository in plaintext. I use SOPS to encrypt the terraform.tfvars file with age:

1
2
3
4
5
6


# .sops.yaml
creation_rules:
 - path_regex: .*\.tfvars$
 age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 - path_regex: ansible/inventory/group_vars/.*\.yml$
 age: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The encrypted file (terraform.tfvars) is committed to Git. Decrypting it requires the age private key, which lives on the CI server and my local machine — never in the repository.

Ansible: node configuration

Once Terraform provisions the VMs, Ansible configures them. The bootstrap.yml playbook does the minimum needed to get a freshly created node into working shape:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ansible/playbooks/bootstrap.yml
---
- name: Bootstrap new nodes
 hosts: all
 become: true
 roles:
 - common
 - cis-level1

- name: Configure K3s server nodes
 hosts: k3s_servers
 become: true
 roles:
 - k3s

The common role installs base packages, configures NTP, hardens SSH, sets sysctl parameters, and creates system users. The cis-level1 role applies CIS Benchmark Level 1 recommendations for Debian.

I did not write the CIS role from scratch. I started from the community role dev-sec/ansible-collection-hardening and adapted it. There are quite a few tasks the default role applies that do not fit a homelab — things designed for production servers with strict audit requirements. I went through each task, understood what it did, and decided if it applied to my case.

Some things I disabled:

1
2
3
4
5


# ansible/roles/cis-level1/defaults/main.yml
os_auth_pam_pwquality_enable: false # No local users with passwords
os_security_users_allow: ["vagrant"] # Dev environment only
os_filesystem_whitelist:
 - vfat  # Required for UEFI boot

And some things I added specifically for K3s:

1
2
3
4
5
6
7
8


# Kernel parameters required by K3s with protect-kernel-defaults
kernel_parameters:
 - { name: "kernel.panic", value: "10" }
 - { name: "kernel.panic_on_oops", value: "1" }
 - { name: "vm.overcommit_memory", value: "1" }
 - { name: "vm.panic_on_oom", value: "0" }
 - { name: "fs.inotify.max_user_watches", value: "524288" }
 - { name: "fs.inotify.max_user_instances", value: "512" }

Kubernetes: manifests with Kustomize

For Kubernetes manifests I use Kustomize over Helm where possible. Helm is more powerful for complex things, but for my own applications Kustomize is sufficient and produces readable YAML.

The basic Kustomize structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubernetes/apps/monitoring/
├── base/
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ ├── prometheus-deployment.yaml
│ └── grafana-deployment.yaml
└── overlays/
 └── homelab/
 ├── kustomization.yaml
 └── patches/
 └── resource-limits.yaml

The homelab overlay adds environment-specific settings without modifying the base manifests:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# kubernetes/apps/monitoring/overlays/homelab/patches/resource-limits.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: prometheus
 namespace: monitoring
spec:
 template:
 spec:
 containers:
 - name: prometheus
 resources:
 requests:
 memory: "256Mi"
 cpu: "100m"
 limits:
 memory: "512Mi"
 cpu: "500m"

OPA/Gatekeeper: policies as code

Gatekeeper is a Kubernetes admission controller that uses OPA (Open Policy Agent) to evaluate policies written in Rego. Instead of letting any pod deploy with any configuration, policies reject manifests that do not meet security requirements.

Active policies:

No containers running as root

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# kubernetes/policies/gatekeeper/no-root-containers.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPAllowedUsers
metadata:
 name: psp-pods-allowed-user-ranges
spec:
 match:
 kinds:
 - apiGroups: [""]
 kinds: ["Pod"]
 excludedNamespaces:
 - kube-system
 - falco
 parameters:
 runAsUser:
 rule: MustRunAsNonRoot
 runAsGroup:
 rule: MustRunAs
 ranges:
 - min: 1000
 max: 65535

Required resource limits

Without resource limits, a pod can consume all the node’s memory and bring down the cluster. This policy prevents that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# kubernetes/policies/gatekeeper/require-resource-limits.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
 name: k8srequiredresources
spec:
 crd:
 spec:
 names:
 kind: K8sRequiredResources
 targets:
 - target: admission.k8s.gatekeeper.sh
 rego: |
 package k8srequiredresources

 violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not container.resources.limits.memory
 msg := sprintf("Container '%v' has no memory limit defined", [container.name])
 }

 violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not container.resources.limits.cpu
 msg := sprintf("Container '%v' has no CPU limit defined", [container.name])
 }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
 name: require-resource-limits
spec:
 match:
 kinds:
 - apiGroups: [""]
 kinds: ["Pod"]
 excludedNamespaces:
 - kube-system

Trusted image registries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


# kubernetes/policies/gatekeeper/allowed-registries.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
 name: k8sallowedrepos
spec:
 crd:
 spec:
 names:
 kind: K8sAllowedRepos
 validation:
 openAPIV3Schema:
 type: object
 properties:
 repos:
 type: array
 items:
 type: string
 targets:
 - target: admission.k8s.gatekeeper.sh
 rego: |
 package k8sallowedrepos

 violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not any_repo_matches(container.image)
 msg := sprintf("Image '%v' does not come from an allowed registry", [container.image])
 }

 any_repo_matches(image) {
 repo := input.parameters.repos[_]
 startswith(image, repo)
 }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
 name: allowed-registries
spec:
 match:
 kinds:
 - apiGroups: [""]
 kinds: ["Pod"]
 excludedNamespaces:
 - kube-system
 parameters:
 repos:
 - "registry.homelab.internal/"
 - "ghcr.io/my-user/"
 - "quay.io/prometheus/"
 - "grafana/"

Seccomp profiles

Seccomp profiles restrict which system calls a container can make. Kubernetes has a default profile (RuntimeDefault) that is already reasonable, but for applications I know well I define tighter profiles.

The profiles live in the repository and are deployed as ConfigMaps or directly to the nodes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


// kubernetes/policies/seccomp/web-app-profile.json
{
 "defaultAction": "SCMP_ACT_ERRNO",
 "architectures": ["SCMP_ARCH_X86_64"],
 "syscalls": [
 {
 "names": [
 "accept4", "bind", "brk", "clone", "close", "connect",
 "epoll_create1", "epoll_ctl", "epoll_wait", "execve",
 "exit_group", "fcntl", "fstat", "futex", "getdents64",
 "getpid", "getsockname", "getsockopt", "listen", "lstat",
 "mmap", "mprotect", "munmap", "nanosleep", "newfstatat",
 "openat", "poll", "prctl", "read", "recvfrom", "rt_sigaction",
 "rt_sigprocmask", "rt_sigreturn", "sendto", "set_robust_list",
 "setsockopt", "sigaltstack", "socket", "stat", "write"
 ],
 "action": "SCMP_ACT_ALLOW"
 }
 ]
}

Referenced from the pod spec:

1
2
3
4
5


spec:
 securityContext:
 seccompProfile:
 type: Localhost
 localhostProfile: "web-app-profile.json"

Building a seccomp profile from scratch is tedious. My approach is to start with RuntimeDefault, use strace to see what syscalls the application actually makes, and then build a tighter profile for workloads I want to restrict further.

CI/CD pipeline

The repository has a GitLab CI pipeline that automates applying changes. The flow is:

On merge requests: terraform plan and ansible-lint to catch problems before merging
On merge to main: terraform apply and, if Ansible files changed, the corresponding playbook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


# .gitlab-ci.yml (excerpt)
stages:
 - validate
 - plan
 - apply

variables:
 TF_ROOT: "${CI_PROJECT_DIR}/terraform/environments"
 ANSIBLE_CONFIG: "${CI_PROJECT_DIR}/ansible/ansible.cfg"

terraform-validate:
 stage: validate
 image: hashicorp/terraform:1.6
 script:
 - cd "$TF_ROOT"
 - terraform init -backend=false
 - terraform validate
 rules:
 - changes:
 - terraform/**/*

terraform-plan:
 stage: plan
 image: hashicorp/terraform:1.6
 script:
 - cd "$TF_ROOT"
 - terraform init
 - terraform plan -out=tfplan
 artifacts:
 paths:
 - "${TF_ROOT}/tfplan"
 expire_in: 1 week
 rules:
 - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 changes:
 - terraform/**/*

terraform-apply:
 stage: apply
 image: hashicorp/terraform:1.6
 script:
 - cd "$TF_ROOT"
 - terraform init
 - terraform apply -auto-approve
 rules:
 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
 changes:
 - terraform/**/*
 when: manual

ansible-lint:
 stage: validate
 image: python:3.11-slim
 script:
 - pip install ansible ansible-lint
 - ansible-lint ansible/
 rules:
 - changes:
 - ansible/**/*

The terraform apply is manual — I do not want infrastructure changing automatically without my approval. The plan runs automatically on the MR for visibility.

What I sanitized

Publishing the repository required reviewing what should not be there:

Internal IPs: replaced with example ranges (192.168.1.x)
Domain names: the internal homelab domain (homelab.internal in the repo, something different in production)
Usernames: real usernames are not in the repository
SSH public keys: replaced with placeholders
Password hashes: removed from the Ansible inventory
Application secrets: encrypted with SOPS or removed, with an .example file alongside

The rule I followed: if someone with access to my local network could use that information to attack something, it does not go into the repository in plaintext. Everything else can be there.

What is still pending

There are still things I manage by hand that should be codified:

OpenWrt: the router configuration is the hardest to bring into IaC. There is a Terraform module for OpenWrt that is not very well maintained. For now I manage it with an Ansible script that backs up the configuration and another that restores it. Not idempotent, but it works.

TrueNAS: it has a fairly complete REST API. There is a Terraform provider in development. It is on my radar for the next iteration.

Backups: I have backups, but the process is not in the repository. It lives in another loose script that will eventually end up here.

The repository is never “finished.” What matters is that the current state of the homelab is represented in it, and that any change goes through Git.

Installing and Configuring Debian, Ubuntu, RaspberryPiOS, LibreElec or Arch on a Raspberry Pi

Thu, 21 Oct 2021 00:00:00 +0000

This post outlines the steps to install and configure a Raspberry Pi for deploying a series of services.

1. OS Installation

1.1. Ubuntu, RaspberryPiOS or LibreElec

Using the Raspberry Pi Imager tool, you can flash the image of Ubuntu, RaspberryPiOS, LibreElec, or whichever you prefer, onto any micro SD card or USB drive ¹. For example, we will choose the Raspbian OS image with desktop and flash it onto a 64GB USB drive.

Another way to flash an image onto an SD card or a USB drive is by using the following commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# See the partitions
lsblk

# Umount the USB partition
umount /dev/sdc1

# Format in vFAT
mkfs.vfat -F 32 /dev/sdc -I

# Flash the ISO into USB
dd status=progress if=NAME.iso of=/dev/sdc

1.2. Arch

To install Arch, the steps from the AArch64 version of the Arch installation ² were followed. We reproduced those steps for AArch64.

Start fdisk to partition the SD card:

1

 fdisk /dev/sdX

At the fdisk prompt, delete old partitions and create a new one:

Type o. This will clear out any partitions on the drive.
Type p to list partitions. There should be no partitions left.
Type n, then p for primary, 1 for the first partition on the drive, press ENTER to accept the default first sector, then type +200M for the last sector.
Type t, then c to set the first partition to type W95 FAT32 (LBA).
Type n, then p for primary, 2 for the second partition on the drive, and then press ENTER twice to accept the default first and last sector.
Write the partition table and exit by typing w.

Create and mount the FAT filesystem:

1
2
3


mkfs.vfat /dev/sdX1
mkdir boot
mount /dev/sdX1 boot

Create and mount the ext4 filesystem:

1
2
3


mkfs.ext4 /dev/sdX2
mkdir root
mount /dev/sdX2 root

Download and extract the root filesystem (as root, not via sudo):

1
2
3


wget http://os.archlinuxarm.org/os/ArchLinuxARM-rpi-aarch64-latest.tar.gz
bsdtar -xpf ArchLinuxARM-rpi-aarch64-latest.tar.gz -C root
sync

Move boot files to the first partition:

1

mv root/boot/* boot

Before unmounting the partitions, update /etc/fstab for the different SD block device compared to the Raspberry Pi 3:

1

sed -i 's/mmcblk0/mmcblk1/g' root/etc/fstab

Unmount the two partitions:

1

umount boot root

Insert the SD card into the Raspberry Pi, connect ethernet, and apply 5V power.
Use the serial console or SSH to the IP address given to the board by your router.

Login as the default user alarm with the password alarm.
The default root password is root.

Initialize the pacman keyring and populate the Arch Linux ARM package signing keys:

1
2


pacman-key --init
pacman-key --populate archlinuxarm

2. Basic Configuration

Once the operating system is installed, there are several options to configure the Raspberry Pi. The simplest approach is to run a network scan with sudo nmap -f 192.168.1.0/24 to identify the IP address assigned to the Raspberry Pi by the router. Then connect via ssh pi@192.168.10.250 (for Raspbian) or ssh alarm@192.168.10.250 (for Arch). You can also set it up headlessly, without an extra monitor, keyboard, and mouse, as explained in a previous post, or by using these three external peripherals. In this case, we will use an external monitor, keyboard, and mouse to simplify the walkthrough. Once the Raspberry Pi is powered on with the USB drive or SD card connected, a dialog will appear to configure the language, Wi-Fi, and a password (for example, use KeepassXC to generate a random password).

2.1 System Update

Once we have a console with a non-root user, we will open a new console as the root user:

1

su -

The default password is usually root or similar.

Debian-based:

1
2


# Update and upgrade packages system
apt-get update -y && sudo apt-get upgrade -y

Arch-based:

1
2


# Update and upgrade packages system
pacman -Syu

2.2 Time Configuration

We follow this guide on first steps after installing Arch ³ or any minimalist distribution such as HypriotOS. We set the timezone to the appropriate one:

1

timedatectl set-timezone Europe/London

We synchronize the clock with the internet:

1

timedatectl set-ntp true

2.3 Locale Configuration

Uncomment the desired locale in the locale.gen file (for example: en_US.UTF-8):

1

nano /etc/locale.gen

Run:

1

locale-gen

And run:

1

localectl set-locale LANG=en_US.UTF-8

2.4 Hostname Change

Change the hostname with:

1

hostnamectl set-hostname <name>

Add an alias for the hostname in the /etc/hosts file of the computer you are using for the configuration. Use nano /etc/hosts:

1
2


127.0.0.1		localhost.localdomain	<name>	localhost
::1				localhost.localdomain	<name>	localhost

(Optional) Enable colored output in pacman

If using Arch, run:

1

sed -i 's/#Color/Color/' /etc/pacman.conf

(Optional) Add 8GB of SWAP memory

If using Arch, run:

1

fallocate -l 8192M /swapfile

(Optional) New user with sudo permissions

If using Arch, we will now use the visudo utility to edit group permissions for running administrative commands with sudo.

1
2


pacman -S sudo
EDITOR=nano visudo

Uncomment the following line:

1
2


## Uncomment to allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) ALL

Create a new sudo group with:

1

sudo groupadd sudo

Create a new user with:

1

useradd -m -G sudo username

Set a password for the new user:

1

passwd username

Once the new user is created, delete the alarm user.

1

userdel alarm

On Debian or derivatives, modify the permissions of the user created during installation and add them to the sudo group with the following command:

1
2


su -
usermod -aG sudo username

Reboot the system:

1

reboot

2.5 SSH

The next step is to copy the public key to the ~/.ssh/authorized_keys file on the Raspberry Pi. Use the following command:

1

ssh-copy-id -i <identity.pub> pi@<raspberry ip or node-1>

Now it will prompt for the SSH key password, and we will connect to the Raspberry Pi via:

1

ssh pi@node-1

(Optional) Wi-Fi Configuration

Reproduced from the guide: first steps after installing Arch ³

karog, on ArchLinux ARM forums provided a simple way to connect to Wi-Fi. As root, do the following steps:

nano /etc/systemd/network/wlan0.network to configure the wlan0 interface:
Add the following contents to the file:

1
2
3
4
5


[Match]
Name=wlan0

[Network]
DHCP=yes

wpa_passphrase "<SSID>" "<PASSWORD>" > /etc/wpa_supplicant/wpa_supplicant-wlan0.conf. Replace and with your respective Wi-Fi network name and password.
systemctl enable wpa_supplicant@wlan0 to enable the Wi-Fi when booting
systemctl start wpa_supplicant@wlan0 to connect to Wi-Fi.

You’re good to go!

If you ever want to remove Wi-Fi connection (e.g. when you want to make it connect only through ethernet):

systemctl stop wpa_supplicant@wlan0
systemctl disable wpa_supplicant@wlan0
rm /etc/wpa_supplicant/wpa_supplicant-wlan0.conf
rm /etc/systemd/network/wlan0.network

3. Software Installation

3.1. Basic Packages

Debian:

1

sudo apt install -y software-properties-common git wget

Arch:

1

sudo pacman -S -y git wget

3.2. Docker and Docker Compose

It is recommended to install docker rootless.

3.2.1. Installation on Debian

Install Docker using the installation script:

1
2


curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

For docker-compose, install it through Python 3:

1
2
3
4


sudo apt-get install libffi-dev libssl-dev
sudo apt install python3-dev
sudo apt-get install -y python3 python3-pip
sudo pip3 install docker-compose

Add the user to the docker group:

1

sudo usermod -aG docker ${USER}

3.2.2. Installation on Arch

Install Docker and docker-compose from the official repositories:

1
2


sudo pacman -Sy docker
sudo pacman -Sy docker-compose

Enable the service with:

1

sudo systemctl start docker.service

Enable it on every reboot:

1

sudo systemctl start docker.service

Add the user to the docker group:

1

sudo usermod -aG docker ${USER}

3.2.3 Docker Rootless

Arch:

1
2


sudo pacman -S shadow
sudo pacman -S fuse-overlayfs

Add kernel.unprivileged_userns_clone=1 in /etc/sysctl.conf:

1
2


sudo nano /etc/sysctl.conf
sudo sysctl --system

Common:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


sudo touch /etc/subuid && sudo touch /etc/subgid
su -
echo "pi:100000:65536" >> /etc/subgid
echo "pi:100000:65536" >> /etc/subuid
exit

sudo systemctl disable --now docker.service docker.socket
curl -fsSL https://get.docker.com/rootless | sh

systemctl --user start docker
systemctl --user enable docker
sudo loginctl enable-linger $(whoami)

export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock

Test it with docker run -d -p 8080:80 nginx.

If you want it to start at boot, run the following commands:

1
2


 sudo systemctl enable docker.service
 sudo systemctl enable containerd.service

3.3. Ansible

Debian

1

sudo apt install -y ansible

Arch:

1

sudo pacman -Sy ansible

3.3.1 SSH Hardening with Ansible

Using the Ansible collections devsec.hardening, we apply security mechanisms and verify them with the following commands.

Installation:

1

ansible-galaxy install dev-sec.ssh-hardening

Create a playbook for each Ansible role called ansible-ssh-hardening.yaml.
Run these playbooks with the following commands:

1

ansible-playbook ansible-ssh-hardening.yaml --ask-become-pass

You can also create an Ansible playbook with any modules you wish to include.

Add your SSH key to an ssh-agent using zsh (or bash):

1
2


ssh-agent zsh
ssh-add ~/.ssh/id_ed25519

Run the Ansible playbook with the required sudo password for the commands:

1

ansible-playbook playbook.yaml --ask-become-pass

3.4. Installing zsh and Oh My Zsh

Debian⁴:

1
2


sudo apt install zsh
sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Arch:

1
2


sudo pacman -Sy zsh zsh-completions
sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

3.4.1. Installing Powerlevel10k

Download and place the 4 Meslo Nerd .ttf fonts in /usr/local/share/fonts. They must have permissions 644 (-rw-r–r–).⁵

MesloLGS NF Regular.ttf
MesloLGS NF Bold.ttf
MesloLGS NF Italic.ttf
MesloLGS NF Bold Italic.ttf Create the /usr/local/share/fonts directory:

1
2


sudo mkdir /usr/local/share/fonts
cd /usr/local/share/fonts

Download the fonts:

1

sudo wget https://github.com/romkatv/powerlevel10k-media/raw/master/MesloLGS%20NF%20Regular.ttf https://github.com/romkatv/powerlevel10k-media/raw/master/MesloLGS%20NF%20Bold.ttf https://github.com/romkatv/powerlevel10k-media/raw/master/MesloLGS%20NF%20Italic.ttf https://github.com/romkatv/powerlevel10k-media/raw/master/MesloLGS%20NF%20Bold%20Italic.ttf

Clone the powerlevel10k project:

1
2
3


git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ~/powerlevel10k
echo 'source ~/powerlevel10k/powerlevel10k.zsh-theme' >>~/.zshrc
exec zsh

Replace the following value in ~/.zshrc:

1

ZSH_THEME="powerlevel10k/powerlevel10k"

3.4.2. Installing zsh Plugins

To add a series of useful plugins, edit the ~/.zshrc file with nano ~/.zshrc and modify the following:

1
2


#plugins=(git)
plugins=(git git-extras history ansible zsh-autosuggestions zsh-syntax-highlighting docker-helpers docker docker-compose kubectl kubectx colorize nmap pip ssh-agent sudo pipenv fzf fzf-docker)

We need to install the zsh-autosuggestions, zsh-syntax-highlighting, docker-helpers, fzf, and fzf-docker plugins.

To install zsh-autosuggestions:

1

git clone https://github.com/zsh-users/zsh-autosuggestions ~/.oh-my-zsh/plugins/zsh-autosuggestions

To install zsh-syntax-highlighting:

1

git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ~/.oh-my-zsh/plugins/zsh-syntax-highlighting

To install fzf use the official repositories:

1

sudo pacman -Sy fzf

To install fzf-docker:

1

git clone https://github.com/pierpo/fzf-docker ~/.oh-my-zsh/plugins/fzf-docker

To install docker-helpers:

1

git clone https://github.com/unixorn/docker-helpers.zshplugin ~/.oh-my-zsh/plugins/docker-helpers

Customize to your liking and reload the ~/.zshrc file:

1

source ~/.zshrc

Snapd

Installation

Install the snapd and core packages:

1

sudo apt install -y snapd core

Add the Snap Executables Path to bash and Zsh PATH

Add the snap executables path to PATH:

1
2
3
4


echo "export PATH=$PATH:/snap/bin" >> ~/.bashrc
source ~/.bashrc
echo "export PATH=$PATH:/snap/bin" >> ~/.zshrc
source ~/.zshrc

Verify the path was added correctly:

1

echo $PATH

Create a symbolic link from the directory that stores snap launchers (/var/lib/snapd/desktop/applications) to the system applications directory (usr/share/applications/):

1

sudo ln -s /var/lib/snapd/desktop/applications /usr/share/applications/snapd

Flatpak

Installation

From the official Flatpak documentation, follow these steps:

Install Flatpak

1

sudo apt install flatpak -y

Add the Flatpak repository

1

flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Reboot the system to apply the changes.

Create a symbolic link from the directory that stores flatpak launchers (/var/lib/flatpak/exports/share/applications/) to the system applications directory (usr/share/applications/):

1

sudo ln -s /var/lib/flatpak/exports/share/applications/ /usr/share/applications/flatpak

KeePassXC

Note: installed via Snap because the official repositories have an outdated version.

Install via Snap:

1

sudo snap install keepassxc

Download the browser extension.
Configure the browser extension using an official KeePassXC script. Save the script and run it:

1
2


wget https://raw.githubusercontent.com/keepassxreboot/keepassxc/master/utils/keepassxc-snap-helper.sh
zsh keepassxc-snap-helper.sh

If you get the error Could not find keepassxc.proxy! Ensure the keepassxc snap is installed properly., this is because the snap executables path is missing from PATH. Add it with:

1
2
3
4


echo "export PATH=$PATH:/snap/bin" >> ~/.zshrc
source ~/.zshrc
echo "export PATH=$PATH:/snap/bin" >> ~/.bashrc
source ~/.bashrc

Run the script again:

1

bash keepassxc-snap-helper.sh

VSCodium ⁴

Add the repository GPG key:

1

wget -qO - https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg | gpg --dearmor | sudo dd of=/etc/apt/trusted.gpg.d/vscodium.gpg

Add the repository:

1
2
3
4


echo 'deb
 [ signed-by=/usr/share/keyrings/vscodium-archive-keyring.gpg ]
https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/debs vscodium main'
 | sudo tee /etc/apt/sources.list.d/vscodium.list

Update repositories and install VSCodium:

1

sudo apt update && sudo apt install codium

OS and SSH Hardening

Thu, 21 Oct 2021 00:00:00 +0000

This post outlines the steps to harden the OS and SSH on a Raspberry Pi, so you can later deploy a set of secured services.

1. SSH Configuration

The next step requires copying your public key to the ~/.ssh/authorized_keys file on the Raspberry Pi. Use the following command:

1

ssh-copy-id -i <identity.pub> pi@<raspberry ip or node-1>

It will prompt for your SSH key password. Then connect to the Raspberry Pi with:

1

ssh pi@node-1

2. Installing Ansible

Debian

1

sudo apt install -y ansible

Arch:

1

sudo pacman -Sy ansible

3. OS and SSH Configuration Using Ansible

3.1. Using a collection

Installation:

1
2


ansible-galaxy install dev-sec.os-hardening
ansible-galaxy install dev-sec.ssh-hardening

Create a playbook for each Ansible role named ansible-os-hardening.yaml and ansible-ssh-hardening.yaml.
Run these playbooks with the following commands:

1
2


ansible-playbook ansible-os-hardening.yaml --ask-become-pass
ansible-playbook ansible-ssh-hardening.yaml --ask-become-pass

3.2. Using a basic playbook

Add your SSH key to an ssh-agent using zsh (or bash):

1
2


ssh-agent zsh
ssh-add ~/.ssh/id_ed25519

Run the Ansible playbook with the sudo password required for the commands:

1

ansible-playbook playbook.yaml --ask-become-pass

Ansible on Adur

Building an IaC repository for the homelab

What triggered this

What the homelab looks like

Repository structure

Terraform: provisioning with Proxmox

Ansible: node configuration

Kubernetes: manifests with Kustomize

OPA/Gatekeeper: policies as code

No containers running as root

Required resource limits

Trusted image registries

Seccomp profiles

CI/CD pipeline

What I sanitized

What is still pending

Installing and Configuring Debian, Ubuntu, RaspberryPiOS, LibreElec or Arch on a Raspberry Pi

1. OS Installation

1.1. Ubuntu, RaspberryPiOS or LibreElec

1.2. Arch

2. Basic Configuration

2.1 System Update

2.2 Time Configuration

2.3 Locale Configuration

2.4 Hostname Change

(Optional) Enable colored output in pacman

(Optional) Add 8GB of SWAP memory

(Optional) New user with sudo permissions

2.5 SSH

(Optional) Wi-Fi Configuration

3. Software Installation

3.1. Basic Packages

3.2. Docker and Docker Compose

3.2.1. Installation on Debian

3.2.2. Installation on Arch

3.2.3 Docker Rootless

3.3. Ansible

3.3.1 SSH Hardening with Ansible

3.4. Installing zsh and Oh My Zsh

3.4.1. Installing Powerlevel10k

3.4.2. Installing zsh Plugins

Snapd

Installation

Add the Snap Executables Path to bash and Zsh PATH

Add Launchers to the Applications Menu

Flatpak

Installation

Add Launchers to the Menu

KeePassXC

VSCodium 4

OS and SSH Hardening

1. SSH Configuration

2. Installing Ansible

3. OS and SSH Configuration Using Ansible

3.1. Using a collection

3.2. Using a basic playbook

VSCodium ⁴