Proxmox 8

These are my notes for building a new Proxmox 8 server.

Proxmox 8 Server Build (PVE2)

I’ve got new hardware and want to bring this server up at the same as as the original so need new IP addresses (which will become static)

Use balena etcher to create a USB bootable disk

Boot disks RAID1 - Mirror

RAID1 (Mirror) on m.2 drives

https://www.youtube.com/watch?v=lhRPhBDysVI

Going to separate out boot drive from data. So that if data drives fail I can boot.

# can see any errors
# zfs pool is called: rpool
zpool status

Networking

Hostname FQDN - pve2.dave.home
IP - 192.168.1.88 (was assigned by router DHCP, then made static). It may be easier to assign manually an IP next time outside the DHCP range.
Gateway - 192.168.1.254 (the router)
DNS - 192.168.1.254 (the router)

https://192.168.1.88:8006 - PM admin interface

Updates

Disable Enterprise repos

Add no subscription repos

PVE 8.2.4 running and updated on new hardware.

Data Disks - RAIDZ-1 - ZFS Datapool

I’ve got 4 data disks which I want to be running for resilience.

RAIDZ-1 (tolerates 1 drive loss) which needs at least 3 disks

I don’t need anything special (just want redundancy but not overly) so lets use RAIDZ-1

pve2, disks, zfs, create ZFS

datapool - (tank is common name)
compression on (saves on writes, and cpu is cheap)
ashift - 12 (default)

zpool status
zfs list

Datadirectory on ZFS Datapool

DataDirectory

Now I can upload an ISO

Ubuntu 24.04.1 LTS

Ubuntu 22.04.5 - my current standard build.

Create a VM

https://www.youtube.com/watch?v=sZcOlW-DwrU craft computing pm 8 - very good explanations as to what the different settings are.

BIOS

disks - give 80GB - on ZFS created with thin provisioning, so it only uses whats needed

max performance of the VM. Min of 4 threads/cores. I have 28 cores in 1 socket.

memory max - give 4GB at least only uses what needed (ballooning)

and

Start up the VM. It got an IP from the DHCP server of 192.168.1.251, and as I installed SSH I can just use my terminal to connect

# so pm can see the IP address
sudo apt install qemu-guest-agent

Firewall and DHCP Networking

I want to run multiple websites inside PM. I used to run pfSense for firewalls and internal DHCP.

Is it possible to use just the proxmox firewall and router DHCP?

Update - have decided to keep running pfSense because my DHCP server (router) is behaving strangely, and pfsense has worked for many years.

video - for running internal pm firewall this is good.

pfSense 2.7.2

I’m going to reinstall from scratch pfSense to understand it more

https://www.dlford.io/pfsense-nat-how-to-home-lab-part-3/ - great tutorial

Download pfSense - 2.7.2. Need to checkout (free) and unzip download into iso then upload to proxmox2. It was a BETA version but that was all I could get.

networking

proxmox2, system, network, create, linux bridge

vmbr0 is there already. connected to en01 (wired ethernet). 192.168.1.88/24
vmbr1 - comment: internal server network

reboot server

Create pfsense vm

create vm

Follow tutorial. TODO - see what seetting worked and tweak

Disk use datadirectory

WAN: vtnet0 LAN: vtnet1

Install CE

Disk - ZFS and GPT. Stripe. No redudancy.

Reboot

WAN: vtnet0 LAN: vtnet1 (don’t do this)

192.168.1.179 is allocated from my DHCP server to WAN side ie vtnet0

As pfSense uses 192.168.1.x by default on the LAN we can’t use this to start with.

Option 3? to reassign network ie disable LAN interface

Option 8 to go into shell

# turn off firewall (so can access admin from WAN)
# http://192.168.1.179
pfctl -d 

# u: admin
# p: pfsense

hostname: pfsense2
domain: dave.home
dns: 8.8.8.8, 1.1.1.1 (I’m was using 192.168.1.254 on pm host - overridden now)

Static IP - forcing just to use the DHCP allocated 192.168.1.179

Can’t Conntect

1) Assign interface - disable LAN
8) Shell - pfctld -d
try another browser (Chrome does bad caching - Firefox is useful especially as we’re on http by default)

Add firewall rule to WAN to allow admin interface

Firewall > Aliases > Ports > Add

pfSense_Admin_Ports

80, 443, 22, 8080

Add a rule.

Couldn’t get the rule to reorder to go above teh

Interface, WAN

Uncheck block bogon networks

8080 for Admin Interface

So that 80 and 443 can be freed up for regular web traffic.

System, Advanced, Admin Access, TCP Port 8080, HTTPS

Add LAN Interface

Interfaces, Interface Assignments

vtnet1 (network port)

Interfaces, LAN, Enable

Static IPv4

172.16.44.1/24

System, Advanced, Networking, Kea DHCP (As ISC DHCP is Deprecated)

Services, DHCP Server, LAN

Range: 172.16.44.10 - 172.16.44.100 (I had to add the range then enable)

Enable

Make sure the proxmox side has the LAN range, so it can get access to the internet.

Template Server VM

Create a normal VM on the vmbr1 internal networking

I’ve got an IP address on 172.16.44.10

but can’t see the internet - was a problem with proxmox LAN range not enabled.

n: blue
u: dave
p: letmein

sudo apt update
sudo apt upgrade
sudo apt install -y qemu-guest-agent

sudo visudo

# add this under sudo ALL=(ALL:ALL_ ALL)
dave ALL=(ALL) NOPASSWD:ALL

Tick guest-trim

Shutdown VM, right click, Convert to Template

Create a new VM based off Template

Right click on template and create new VM

setup port forward so can SSH to it

Get MAC of new VM from UI

eg: BC:24:11:64:57:A8

Firewall, NAT, Port Forward

ssh pfsense2 -p 103


# run boilerplate
# https://www.dlford.io/secure-ssh-access-how-to-home-lab-part-5/#boilerplate
sudo bash -c "bash <(wget -qO- https://raw.githubusercontent.com/dlford/ubuntu-vm-boilerplate/master/run.sh)"

DHCP Reserve (so IP doesn’t change)

Services, DHCP Server, LAN, Edit Static Mapping (at bottom)

MAC and IP eg 172.16.44.103

Change the Port Forward

Firewall, NAT, Port Forward

Port forward for 80 and 443 from WAN (websites)

sudo apt install nginx

Lets open up 172.16.44.103 ports 80 and 443

Firewall, NAT, Port Forward (this adds associated filter rule)

Okay this did work by testing my hitting my public IP from http.

Mounts and Shares

# unmount from old PM
umount /dev/sdf
# or
umount /mnt/wd-2tb-usb
umount /mnt/windows-share

# mount back on old PM
mount /dev/sdd /mnt/black-256-stick

mount /dev/sdf /mnt/wd-2tb-usb


# new PM
# list usb devices - Western Digital Elements 2621 is my disk
lsusb

# look for sdb, sdc etc.. with correct size of disk - mine was sdc
# sde is 250GB stick
# sdf is 1.8TB drive
# this shows the mount if it is there
lsblk

# shows drives and mounted on
df -h

mkdir /mnt/black-256-stick
mkdir /mnt/wd-2tb-usb

mount /dev/sde /mnt/black-256-stick
mount /dev/sdf /mnt/wd-2tb-usb

8th Oct - having some strange issues with slowing down on backups (taking an hour) on my black-256-stick. The 2tb drive is okay.

Adding the mounted disk into PM

Survive a reboot

I’m not sure if windows share does survive a reboot. I had to do mount -h to get it to wake up again.

# find device eg /dev/sdf - this is my 2tb wd drive
lsblk

# find uuid of device rather than /dev/sdd (as this may change)
# eg ab6e004f-6c99-4d33-b145-4208201c404e
blkid

vi /etc/fstab

# UUID=<UUID> <mount_point> <filesystem_type> <options> <dump> <pass>
# dump 0.. used for backup utility
# pass 2... root filesystems use 1 

# wd-2tb-usb
UUID=ab6e004f-6c99-4d33-b145-4208201c404e /mnt/wd-2tb-usb ext4 defaults,nofail 0 2

# to make sure system picks up new version of fstab
systemctl daemon-reload

# test the changes without rebooting
mount -a

# see if mount point is now active
df -h

Spin down USB External disk when not in use

Lets save energy (and less noise) by spinning down the drive when I don’t need it.

# check which drive in the usb to spin down.. /dev/sde 
df -h

# put drive into standby mode
hdparm -y /dev/sde

# I go this error but it did go into standby mode
#  issuing standby command
# SG_IO: bad/missing sense data, sb[]:  f0 00 01 00 50 40 00 0a 00 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

It does seem to spin up again when I ask it to show backups

Lets see if on spin down state, it wakes up and does a backup at 0100 - yes it does.

# crontab -e
# add this line to crontab
0 3,15 * * * hdparm -y /dev/sde

New VM DHCP and NAT

It gave it a DCHP IP, so lets make it static.

1) Static IP - Services, DHCP Server, LAN, Edit Static Mapping (at bottom)

96:30:EC:A6:0C:93

172.16.44.104

2) Open so can SSH - Firewall, NAT, Port Forward

3) Open up port 80 to this new reverse proxy.

Change name of restored machine

# https://www.dlford.io/secure-ssh-access-how-to-home-lab-part-5/#boilerplate
sudo bash -c "bash <(wget -qO- https://raw.githubusercontent.com/dlford/ubuntu-vm-boilerplate/master/run.sh)"

# on dev - shortcut 
sudo vim ~/.bash_aliases
alias 4='ssh pfsense2 -p 114'

right click on folder, properties, sharing, Advanced Sharing, Share this folder

create local user on win: bob password …. not an email user.. just local.

Windows features

Windows firewall - Allow an app through firewall. File Sharing both.

mkdir /mnt/windows-share

# I am inside proxmox in a 172.16 address range.. need a way to get out
mount -t cifs -o username=bob,password='xxxxxxx' //192.168.1.152/proxmox /mnt/windows-share

# automate mounting
vi /etc/fstab
//192.168.1.152/proxmox /mnt/windows-share cifs username=bob,password='secret',iocharset=utf8 0 0


# nofail - allow boot if mount not available
# automaout - systemd will try to mount the share on access rather than boot
//192.168.1.152/proxmox /mnt/windows-share cifs username=bob,password='secret',iocharset=utf8,nofail,x-systemd.automount 0 0


# **using this**
# notice no quotes on password
//192.168.1.152/proxmox /mnt/windows-share cifs username=bob,password=secret,iocharset=utf8,vers=3.0,nofail 0 0

# tests fstab
mount -a 

# got some weird reboot issues...
# and could only see share when I manually when to it inside bash.. then proxmox could see it.

ZFS

nvme0n1 - 465G
nvme1n1 - 465G
sda 1tb
sdb 1tb
sdc 1tb
sdd 1td
sde - 233G
sdf 1.8TB

zpool status

datapool - data
rpool - mirror

# my rpool is full up (mirror)
df -k

du -h --max-depth=1

umount /dev/sde

# show mounts
mount

lsblk with black-256-stick and wd-2tb-usb mounted

VM Websites / APIs

http://mateer.hopto.org/ is the name I got from https://www.noip.com/ which my router talks to to provide dynamic DNS.

I’ve got an Nginx reverse proxy in front of all websites which forwards traffic to a specific internal IP via http. It also handles certificates via certbot

hoverflylagoons.co.uk - Apache / PHP . MySQL - 2GB
theskatefarmcic.co.uk - Apache / PHP / MySQL - 2GB
API that runs a hatespeech AI called from https://osr4rightstools.org/
website that runs firemap called from

Maybe move over

https://osr4rightstools.org/ - .NET 1GB RAM. todo - certbot for certs
https://auto-archiver.com/ - .NET 1GB RAM todo certbot for certs
https://arlawesi.org.uk/ - .NET 1GB RAM todo certbot for certs

Test sites

https://hmsoftware.org/ - test site #1. Have used for Rails
https://hmsoftware.uk/

Sites hosted elsewhere

https://hmsoftware.co.uk - static site hosted on netlify.com main company website which is being developed soon into a portfolio site
homebrewbeer.netlify.app - Hugo based static site on Netlify
https://davemateer.com/ - Jekyll blog on pages.github.com/

VM Workers

General archiver eg auto-archiver.com/ - Python which runs fine in 4GB RAM
Tweet service that polls an Azure database every minute to see if any Tweets to send out. Python. 1GB RAM
YouTube screenshotter archiver - I’m using playwright.dev via a Proxy brightdata.com/ and using headful Firefox

Networking

Plusnet Two Router / DHCP is on 192.168.1.254

Proxmox is on a static IP allocation of 192.168.1.139 - port 8006 gives us the Proxmox web UI

pfSense VM has a DHCP (!fix this) alloction of 192.168.1.144 - port 8080 gives us the pfSense web UI

All port 80 and 443 traffic is routed to pfSense from the router.

Power Costs

Big server may draw 300W at idle. eg dual xeon

My UK provider good energy charges me 24.70 pence per kWh on 3rd Sept 2024. And 59.18 pence per day

So the xeon server would cost 0.3 kW x 24 hours = 7.2 kWh/day = £1.78 / day £53 / month

My 13th Gen Intel draws about 40W at idle running wifi, disks, etc..

RGB Machine

40W at rest it draws = 0.04 kW * 24 hours = 0.96 kWh/day = £0.237 per day = £7 per month

Storage

Drive Writes Per Day (DWPD) - how many times the entire capacity can be written per day over warrenty period. eg 1TB SSD DWPD rating of 1 - can write 1TB every day for 3 to 5 years.

SATA DOM (Disk on Module) - often used for boot drives. SATA SSD - 2.5” nvme Mechanical - for boot not recommended.. as SSD is cheaper, lower power and more reliable.

For boot drives approx 10 TB/year is a good number.. so shouldn’t wear out the SSD at all.

So after my drive failure(s) of consumer ssd’s, I want to make my server more resilient.

m.2 2 * 1TB just for extra relaibility (check DWPD). He uses 2 x 256 AirDisk SSD
SSD

Mirror the m.2 drives for the OS

## Logs

Look for problems eg

Sep 02 06:54:11 pve smartd[718]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 69
Sep 02 06:54:12 pve smartd[718]: Device: /dev/sdb [SAT], 8 Currently unreadable (pending) sectors
Sep 02 06:54:12 pve smartd[718]: Device: /dev/sdb [SAT], 8 Offline uncorrectable sectors

My sdb is a mechanical disk I’m not using.

## Disks

Have got 26% wearout on this Samsung SSD 840 Pro. This is my backup server which I’m rebuilding soon.

This is nothing to worry about and drive shoudl still have plenty of life left.

Servers Setup

BIOS settings for my silver server (which had SSD and potential hardware failture). Notice auto power on.

Router Setup

http://192.168.1.254/ Router Admin

Router thinks that .139 is not connected (but it is as the PM Admin interface answers). 1 port having 2 IP addresses is confusing the GUI for the router.

There are 2 .139 addresses as the MAC address changed when I got the backup PM server.

(Static) means the local IP address is assigned by the device. It was initially assigned DHCP when Proxmox was installing, now it is fixed. Interestinly I can’t change the router to ‘always use this IP address’ as it isn’t assigned by DHCP anymore. I hope DHCP wont assign .88 anymore.

I did deleted some entries which were not used anymore, and the router did some strange things like being extremely slow (and management interface loading slowly) and needed a boot.

All 80 and 443 traffic is forwarded to pfsense on 192.168.1.144 (whc)

https://192.168.1.139:8006/ PM Admin interface
https://192.168.1.144:8080/ Pfsense Admin interface

This router can assign 64 IP addresses on DHCP

Lessons Learned

SSD’s fail.
Hardware fails.
Standard routers are fickle (DHCP leases sometimes get confused / UI gets confused)
Electricity cuts do happen (PM did recover fine)
I want a backup PM server ready to go (and a testing server as well)
You always need more disk space than you think on a VM
Home connection with dynamic DNS has been great. Was 48Mbps down and 12MMbps up. Now on fibre at 500/110.
Backups are super important
Template VM’s useful for spinning up test VM’s quickly
pfSense is hard
System has been very reliable
Been invaluable for archiving as many services now (eg YouTube) will block cloud based IP’s

Proxmox 8 Server Build (PVE2)

Boot disks RAID1 - Mirror

Networking

Updates

Data Disks - RAIDZ-1 - ZFS Datapool

Datadirectory on ZFS Datapool

Create a VM

Firewall and DHCP Networking

pfSense 2.7.2

networking

Create pfsense vm

Can’t Conntect

Add firewall rule to WAN to allow admin interface

Interface, WAN

8080 for Admin Interface

Add LAN Interface

Template Server VM

Create a new VM based off Template

setup port forward so can SSH to it

DHCP Reserve (so IP doesn’t change)

Port forward for 80 and 443 from WAN (websites)

Mounts and Shares

Survive a reboot

Spin down USB External disk when not in use

New VM DHCP and NAT

Open share on Windows 10

ZFS

VM Websites / APIs

VM Workers

Networking

Power Costs

RGB Machine

Storage

Servers Setup

Router Setup

Lessons Learned