These are some basic routines I go over during a Proxmox VE install. Putting them down here so I won’t forget about them. Information might change with time.

Install

File System

During install you get to choose which file system you want to use. I’m going with a ZFS mirrored (RAID1) setup for Proxmox VE itself. Note that by saying Proxmox VE itself, I mean that VMs / Containers will run on other disks. Will touch on this later.

Why ZFS? Basically, it’s great. If you want to know more about it, check out -

Why mirrored

Losing one ssd on Proxmox is acceptable. I do not expect to lose both SSDs at the same time.
RAIDZ needs at least 3 drives, while mirroring only needs 2.
For workloads prioritizing IOPs, more VDEVs are preferred. Though this might not matter for Proxmox VE itself, it does matter for VMs and their pool, as will be discussed later.

ZFS settings during install

Read more here as well as references below.

ashift = 12

The ashift value of the pool defines the minimum block size of the pool. As I have drives with 4K physical sectors (see here), ashift = 12 corresponds to 4K sectors (2^12 = 4096). Even if you have a drive that uses 512 byte physical sectors, which is rare, using ashift = 12 is still fine. Note that once the ashift value is set for a pool, there is no going back unless you destroy the pool, which would mean reinstalling Proxmox VE. This is also why using ashift = 12 is recommended even for physical 512 byte sector drives, as if you add a 4K sector drive you will encounter performance issues.

However, if you have those rare 8K sector SSDs - set ashift = 13. See also on this topic: ArchWiki

compress = lz4

LZ4 does not tax on IO performance but still compresses. Simple as that.

checksum & copies

Leave as-is.

hdsize

Normally it should be left at default, as this will let ZFS use all the free space it can use. However, if you want to leave some space for partitioning later, you can set it here. For example, because of complications regarding swap on ZFS (you shouldn’t), leave some space to add a swap partition later.

Email

Set it to a email address you own, as alerts will be sent here. This can be modified later.

Hostname

This can not be modified later.

After Install

Creating another zpool for VMs and Containers

Check if your pool is using entire disks - run zdb and find whole_disk.

It is recommended to point ZFS at an entire disk (ie. /dev/sdx rather than /dev/sdx1), which will automatically create a GPT (GUID Partition Table) and add an 8 MB reserved partition at the end of the disk for legacy bootloaders. Source: ArchWiki There is also a reason regarding IO, however I was unable to find documentation sources for this. Source: Reddit

Reason to use whole disk - Reddit

Also, I don’t like the look of having boot & EFI & ZFS partitions on a single disk, so I would rather have dedicated disks for storage.

Partition vs Whole Disk

I use mirrored setups for this - not RAIDZ. Reasons have been listed above.

ZFS trim and scrub

This is actually already enabled by default.

root@ayanami:~# cat /etc/cron.d/zfsutils-linux 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# TRIM the first Sunday of every month.
24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi

# Scrub the second Sunday of every month.
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi

Postfix settings so alert emails won’t go to spam

I have my own email server with mailcow. If you use a different server / provider, your settings may vary.

Set email from address

Go to Datacenter -> Options -> “Email from address”. Enter your email from address here.

Get prerequisites

apt-get install libsasl2-modules

main.cf

nano /etc/postfix/main.cf

Comment out the relayhost = line.

Add the following at the end.

relayhost = mail.relay.host:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_tls_security_level = encrypt

sasl_passwd

echo "mail.relay.host [email protected]:YourAppPassword" > /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
postfix reload

Send testmail

mail -s Testmail [email protected]
Testmail
.

Check mail queue

root@ayanami:~# mailq
Mail queue is empty

Troubleshooting

If you have errors like -

root@proxmox:~# mailq
-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
43B2E100F17    1399 Mon Nov  1 18:36:56  [email protected]
(SASL authentication failed; cannot authenticate to server smtp.sendgrid.net[13.114.210.107]: no mechanism available)

Then you might have messed up the following -

apt-get install libsasl2-modules
postmap /etc/postfix/sasl_passwd
Wrong account / password

Use postqueue -f to retry the queue.

System and ZFS alerts

If you have your correct email set at installation, you have nothing else to worry about. All alerts will be sent to root@proxmox, and then forwarded to the set email address.

If you need to fix it, you can find it at Datacenter -> Permissions -> Users -> root -> E-Mail.

To test them out, just pull your drives out and wait for a email alert to arrive.

Enable PCI-E passthrough

Make your modifications using the commmand line.

Follow these guides:

Disable Conntrack for asymmetrical routes

Create / modify /etc/pve/nodes/<nodename>/host.fw and add the following:

[OPTIONS]
nf_conntrack_allow_invalid: 1

Then restart the Proxmox VE firewall.

pve-firewall stop && pve-firewall start

Source: blog.swineson.me

References & Sources

最后修改于 2023-09-18