Homeserver Setup

© 2023 Taqi Tahmid

This is the top level directory for the homeserver setup. It contains the following directories:

1. ansible: Contains the ansible playbooks and roles for setting up the homeserver. Proxmox is used as the hypervisor, so the playbooks are written with that in mind. The Kubernetes cluster is set up using K3s. This has not been automated yet and is a manual process.

2. docker: Contains the docker-compose files for setting up the different services. Right now Kubernetes is preferred over docker-compose, so this directory is not actively maintained.

3. kubernetes: Contains the kubernetes manifests and helm charts for setting up the different services.

Services

The following services are set up on the homeserver:

1. AdGuard Home
2. Private Docker Registry
3. Jellyfin
4. My Portfolio Website
5. Postgres Database
6. Pocketbase Backend

In future the following services are planned to be set up:

1. Nextcloud
2. Gitea
3. Monitoring Stack

Homeserver Setup

I have two mini PCs with Intel N1000 CPUs and 16GB of RAM each and 500GB SSDs. Both of them are running Proxmox and are connected to a 1Gbps network. The Proxmox is set up in a cluster configuration.

There are four VMs dedicated for the Kubernetes cluster. The VMs are running Ubuntu 22.04 and have 4GB of RAM and 2 CPUs each. The VMs are connected to a bridge network so that they can communicate with each other. Two VMs are confiugred as dual control plane and worker nodes and two VMs are configured as worker nodes. The Kubernetes cluster is set up using K3s.

Proxmox Installation

The Proxmox installation is done on the mini PCs. The installation is done by booting from a USB drive with the Proxmox ISO. The installation is done on the SSD and the network is configured during the installation. The Proxmox is configured in a cluster configuration.

Ref: proxmox-installation

Promox VM increase disk size

Access the Proxmox Web Interface:

1. Log in to the Proxmox web interface. Select the VM:
2. In the left sidebar, click on the VM to resize. Resize the Disk:
3. Go to the Hardware tab.
4. Select the disk to resize (e.g., scsi0, ide0, etc.).
5. Click on the Resize button in the toolbar.
6. Enter 50G (or 50000 for 50GB) in the size field.

After that login to the VM and create a new partition or add to existing one

sudo fdisk /dev/sda
Press p to print the partition table.
Press d to delete the existing partition (note: ensure data is safe).
Press n to create a new partition and follow the prompts. Make sure to use the
same starting sector as the previous partition to avoid data loss. Press w to
write changes and exit.
sudo mkfs.ext4 /dev/sdaX

Proxmox Passthrough Physical Disk to VM

Ref: proxmox-pve It is possible to pass through a physical disk attached to the hardware to the VM. This implementation passes through a NVME storage to the dockerhost VM.

# List the disk by-id with lsblk
lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" \
  ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");\
  print "";}'|grep -v -E 'part|lvm'

# Hot plug or add the physical device as a new SCSI disk
qm set 103  -scsi2 /dev/disk/by-id/usb-WD_BLACK_SN770_1TB_012938055C4B-0:0

# Check with the following command
grep 5C4B  /etc/pve/qemu-server/103.conf

# After that reboot the VM and verify with lsblk command
lsblk

Setup Master and worker Nodes for K3s

The cluster configuration consists of 4 VMs configured as 2 master and 2 worker k3s nodes. The master nodes also function as worker nodes.

# On the first master run the following command
curl -sfL https://get.k3s.io | sh -s - server --cluster-init --disable servicelb

# This will generate a token under /var/lib/rancher/k3s/server/node-token
# Which will be required to for adding nodes to the cluster

# On  the second master run the following command
export TOKEN=<token>
export MASTER1_IP=<ip>
curl -sfL https://get.k3s.io | \
    sh -s - server --server https://${MASTER1_IP}:6443 \
    --token ${TOKEN} --disable servicelb

# Similarly on the worker nodes run the following command
export TOKEN=<token>
export MASTER1_IP=<ip>
curl -sfL https://get.k3s.io | \
  K3S_URL=https://${MASTER1_IP}:6443 K3S_TOKEN=${TOKEN} sh -

Configure Metallb load balancer for k3s

The metallb loadbalancer is used for services instead of k3s default servicelb as it offers advanced features and supports IP address pools for load balancer configuration.

# On any of the master nodes run the following command
kubectl apply -f \
  https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml

# Ensure that metallb is installed with the following command
kubectl get pods -n metallb-system

# On the host machine apply the metallb cofigmap under metallb directory
kubectl apply -f /home/taqi/homeserver/k3s-infra/metallb/metallbConfig.yaml

# Test that the loadbalancer is working with nginx deployment
kubectl create namespace nginx
kubectl create deployment nginx --image=nginx -n nginx
kubectl expose deployment nginx --port=80 --type=LoadBalancer -n nginx

# If nginx service gets an external IP and it is accessible from browser then
the configuration is complete
kubectl delete namespace nginx

Cloud Image for VMs

A cloud image is a pre-configured disk image that is optimized for use in cloud environments. These images typically include a minimal set of software and configurations that are necessary to run in a virtualized environment, such as a cloud or a hypervisor like Proxmox. Cloud images are designed to be lightweight and can be quickly deployed to create new virtual machines. They often come with cloud-init support, which allows for easy customization and initialization of instances at boot time.

The cloud iamges are used for setting up the VMs in Proxmox. No traditional template is used for setting up the VMs. The cloud images are available for download from the following link: Ubuntu 22.04