Introduction and Motivation
This post will go over my process for building a home server, and hopefully will help anyone looking to do something similar. Be aware this is my first build ever, and my knowledge is limited.
The first thing I needed to figure out was what I was going to use it for. I had been playing with the idea for a long time and had done some early explorations with a Raspberry PI, but I wanted this to be the real thing.
Right now I use a XMG Core 15 that I got when I started college and still serves me well. I use a dual-boot configuration with Windows and Ubuntu that has given me lots of headaches over the years but now seems to be stable. I can’t get rid of Windows since most of my audio and video software is not available for Linux (Ableton, Touchdesigner…).
I intend to use the laptop exclusively on Windows and remote into my server for any computer science related work. One of the main reasons to build the system is my thesis (Ophthalmological image registration using Implicit Neural Representations), which requires me to do a bunch of training that my laptop isn’t suited for (my GPU only has 6Gb of VRAM) and I haven’t been given access to the college compute center yet.
The objective is to build a system that will allow me to experiment, mainly around machine learning, but also with some other services. I settled pretty early on using Proxmox instead of a traditional operating system. It allows you to manage VMs and containers easily and connect via a web interface, so it seemed perfect for my needs. I also have friends who use it in their own setups and could help me when I get stuck. Another of the objectives of this build was to familiarize myself more with servers and virtualization environments, so it should work great to learn about NAS or media servers.
Another of the things that motivated this project was being able to run my own LLMs locally, particularly now that open source has almost caught up to the state of the art. I have seen many accounts of people running Llama3 70b Instruct on a dual 3090 setup and it´s the cheapest VRAM/euro that I could find. I know that in terms of money, it’s probably better to rent a cloud server for all my needs, but I prefer to self-host everything that I can and the experience gained from this is non-trivial.
I ended up getting just one GPU for the time being but planning my rig as if I had 2, so I can add another one later on. I chose the 3090s since they still support NVLink, but recently found that tinygrad released a patch to add P2P support to 4090s, so in retrospective I would have chosen to build a dual 4090 system (even thought it would have taken more time).
With that said, I would like to mention some builds and resources I found that helped inform my choices:
- Sam’s Rig Blog (Part 1 & 2) - 2x3090, open-air.
- IVA: Mini Deep Learning Rig - 2x3090, open-air.
- Den’s Deep Learning Rig - 2x3090, case.
- Detailed Explanation on YouTube - 2x3090, case.
- Deep Learning Hardware Guide (2018), Consumer Hardware and GPUs (2023) - Best resource by far, slightly out of date.
- My Deep Learning Rig - 8x3090 !, open-air.
- Dual RTX 4090 Workstation - 2x4090.
Build
Here is my PCPartPicker Part List:
Except for the case, the power supply, and the storage drives, everything else was bought second-hand or refurbished. Right now is not a particularly good time to buy computer parts (as new generations of CPUs and GPUs are supposed to be coming soon), but thanks to the crypto boom and bust, there are a lot of people selling their mining rigs for parts. Mining rigs are similar to rigs built for machine learning, since they mostly care about GPU performance, and crypto tasks are typically lighter than gaming or rendering.
That’s why some of the example builds are open-air, apart from the aesthetics, it allows for much better air-cooling and temperatures, leading to better performance. The catch is that it’s much more vulnerable, either to dust or anything else happening around it. Another option was to use a water-cooled solution (3090s can get really hot depending on the model), but since this was my first build I didn’t want to push it and I opted for a big PC case. I could have used a rack mount, but apparently they handle air ventilation a bit worse and are more expensive, loud, and power hungry. I chose the Lian Li O11 Evo after researching all the options that could fit 2x3090s.
The rest of the components were chosen based mostly on what I found on the second-hand market. I picked the AMD Ryzen 9 5950X even though it uses the older AM4 socket since it has 16 cores and I won’t be needing a lot of CPU power anyway. I had limited choices for a motherboard since it needed to have 4-slot spacing between two PCIe x16 slots (for the two 3090s to fit) and operate at least in x8/x8 mode (from what I’ve seen you don’t need more than that to avoid bottlenecks)1. I ended up settling for the Asus X570-E, which works but turns out to be a bit too small to allow for good airflow between two graphic cards. I plan to add the second card using a PCIe riser and a vertical mount in the case to avoid that problem.
Because this setup will be pretty thermally intensive when I get the second card, I am planning on capping their power consumption, that’s why I didn’t go with a bigger power supply. Once I do that, I’m planning on running a test to see whether it’s worth it for me to get an NVLink connector, as depending on the task it might not improve my performance any significant amount. Surprisingly the computer turned on my first attempt (thought I had to wait until my GPU arrived since my CPU doesn’t have integrated graphics), but quickly started to bump into some problems.
It has some RGB in the motherboard and RAM that I’m still struggling to turn off. I cleaned up the cables up after and I’m still waiting for some other fans I plan to add.
Setup
I was able to set up proxnox after solving some conflicts in the BIOS configuration and using the ‘nomodeset’ argument (for the kernel to not initialize the video drivers), and quickly got some machines up and running. I started running into problems when configuring the PCIe passthrough to use the GPU directly from the virtual machine and after quite some time managed to get it kind of working.
After that, I set up a VPN to connect to my server from anywhere without having to expose it to the internet. Here I also run into quite a few problems since my router is behind a switch and the main home router, effectively under a double NAT. Also, my ISP doesn’t allow for static IP addresses in our current plan, so I planned to use a DDNS to work around that. After setting up some port forwards on both routers and some troubleshooting, I also got that working with my OpenVPN server running on a proxmox container. However, this setup was quite convoluted and started running into problems once I started running a NAS (TrueNas container) and started messing with the firewall.
After some research and deliberation, I ended up surrendering to the circumstances and decided to abandon proxmox (for now) and install Ubuntu Server. The main reason behind the change was that, while a helpful experience, I don’t want to spend that much time setting up and arranging my home network and devices, I mostly want them to work and be able to invest my time in the things I truly care about, like computer vision, music or writing this blog. I don’t regret doing this and I suspect that if I hadn’t set my priorities straight at that point I could have gone off the deep end and wasted all my summer on this project. Also, I can always come back to it whenever I’m interested again so it’s not that big of a loss.
Ubuntu Server was much easier to setup from the beginning, I briefly pondered using Debian as my distro, as I had some experience thanks to a subject I took on my 3rd year, but then started remembering all the random pains that I suffered during that course (manually setting up firewalls, apache servers and log collection among many other things).
Skipped all my VPN problems by using Tailscale(I also tried ZeroTier but found it a bit more cumbersome) and quickly created a Nextcloud instance to satisfy to use as my NAS. Typically I use vscode server or ssh to connect to the machine but I also have physical access in case it’s needed. I plan on using a cloudflare tunel whenever I need to open a service to the internet.
I also tried both Netdata and Graphana+Prometheus for all my monitoring, but ended up sticking to the second one, as I found it much more easy to personalize and was much easier to get gpu metrics. Other than that, I’m slowly migrating everything that I need there so I can format my laptop. I used to do that regularly (at least once a year) since I felt like it helped me reorganize my life (usually at the end of summer) but stopped doing it a couple of years ago because of uni and fear of losing some weird configurations that I wasn’t sure I would be able to replicate.
Some other things that I started playing with were AUTO1111 and ComfyUI, I have experience with the first and really wanted to try the second, and so far I’m really enjoying the experience of not having to stress about the time left on my cloud instance. Also started using Ollama to build a RAG chatbot with local documents, it was easy to get something working thank to the amount of examples there are but I got some work left to customize it to my needs, I might end up writing a blog post on that if it ends well.
I plan on updating this post when I get the second GPU, until then, any comments or suggestions are always welcomed at [email protected]
Footnotes
-
Usually this is not displayed on product pages, so a good trick to find motherboards like this is to sort by Nvidia-SLI support. ↩