This time I am going to introduced my version of a home lab. This is not as “pro” as many others have but have a good combination of lab and a home computer in a same package. The post does not contain instructions of how to mimic what I have, only a story of my own home lab.
Little bit of history first. I have had some sort of home lab for as long as I remember. When I was building my first version I just used my gaming computer which I had at the time and used virtualization to boot up some virtual machines. That works fine still in my opinion, but since then I have liked the idea of having the home lab separated from the daily driver that I am using.
The first version of the hypervisor that I bought for this purpose was a old HP workstation which had Xeon processor and quite a lot of memory. This was maybe 4-5 years ago. I installed Proxmox first on the thing, after which I moved to ESXi. This was working very nicely although the oldness of the CPU was showing up even if it had quite some cores. I had a complete AD domain with 4-5 domain joined devices available there which was working very nicely for testing purposes. However, a year ago I wanted to buy a new desktop, mostly for gaming but of course it is also a daily driver when I am working on a computer at home. I was thinking if I could combine the daily driver and the lab to a single computer.
I started to research my options. I knew that it is possible to pass through a GPU and an SSD to one virtual machine using almost any hypervisor available. However, I am starting to be a little lazy when it comes to infrastructure problem solving – I tend to prefer solutions that work well out of the box. From this perspective, I found a hypervisor known as Unraid. Unraid isn’t really only a hypervisor, actually I think that this feature was added much later to the product. The main function of Unraid is to act as a storage server, a NAS. However, it is also capable of acting as a hypervisor and also docker manager – which are the workloads that I needed.
BTW, here are the specs for the machine:
- MSI MEG x570 Unify
- AMD Ryzen 9 5950X
- 64GB DDR4 3200MHz CL16 (Thinking of doubling this now that the price is a little cheaper)
- 2* Corsair 1TB SSD M.2 (Soon adding a third M.2 SSD)
- 2* 500GB SSD sata (Model eludes me, these are older)
- Arctic Liquid Freezer II (AIO mostly cause I don’t like to have the heavy AIR cooler bolted to the motherboard, again just an issue my head and not a “real” issue)
- Fractal Design Meshify 2 Black Solid (To hide the stupid LEDs on the AIO)
- Nvidia GTX 1070 Ti (this is a little older, still works fine for my use)
So why Unraid instead of Proxmox or ESXi? Well, the GPU and SSD passthrough feature to a Virtual Machine are made super easy in Unraid. It is also more of a consumer product so it is quite easy to use (not saying that the others wouldn’t be, they are in simple scenarios). So I chose to go with Unraid, even though I had to pay a license fee for it (I think it was 60 €). This was around a year ago. First, I installed my daily driver and it was very very easy to pass through all the things that I needed, including some of the USB ports for peripherals. It just worked. I had first installed Windows natively just to check the performance and that there are no issues with the new hardware and that benchmarks gave the expected results. As a VM I pinned half of the cores to the daily driver while the other half was left for virtualization purposes. The performance was as to be expected; half of the shown performance when I had Windows installed natively. I was expecting a bit of a performance loss from the virtualization layer but didn’t see any.
After I had my daily driver setup I started to migrate the lab over from the ESXi. This was very easy to do, there are ready-made tools which supported this migration nicely. Everything worked from this front well although I wasn’t really happy with the home lab as it wasn’t very automated. At this time, I hadn’t run any live malware on the system (or I had, but only offline so no network connectivity) so the network was very flat. I decided to change that while migrating and I created two Virtual LANs. One was for more “safe” environment where there were no malware being run and the other was for running the malware. I have a Ubiquiti router (EdgeRouter) which has quite nice features available. This router supports VLANs and also allows to configure firewall policies to block traffic between of them. So I used the router to create the VLANs, and I also blocked traffic from each VLAN to any other VLAN. I do have a management network from which I can connect to all the VLANs though.
So now I had 2 VLANs (for this usage, I actually have 5 in total). I put the old lab to VLAN 2, which is the more safe VLAN with network connectivity to internet. Then on the VLAN 3 I started to install new things. There I had Windows VMs that I was using for testing purposes. I also wanted to use Defender for Endpoint for analyzing the live malware at some stage. So, I needed to have internet connectivity which is problematic as I don’t really want to route the potentially malicious traffic through my own ISP as they might not like that very much. I did not really fancy the idea of having a separate internet connection for this (although I would really recommend having a separate internet connection for this). How I decided to tackle this problem is by using a VPN. So, I basically started to route the traffic through a VPN for the machines that I was using for testing – which worked nicely. This way, the ISP should not be bothered and also my original IP is not revealed for the potential bad guys. If the VPN is not working, the traffic is sinkholed. BTW, Unraid works amazing with VLANs, you can choose the VLAN when giving an adapter to a VM and the VM doesn’t have to be aware.
This setup was working quite nicely for running POC code, Malware and for example Atomic Red Team tests – the legitimate tests being run on the more “safe” side of the lab. There of course were issues like the malware analysis was done a standalone device without being in a domain. Also, I was running out all the trials for the MDE license and I haven’t found a way to buy a license for a cheap price as the standalone offering for MDE is not available from MS. I think the cheapest license which includes MDE is somewhere on the lines of ~40€ a month which is quite a lot for this use. So I wanted to have a new environment which would hopefully have a ready-made monitoring at least with endpoint data, domain and a domain joined windows device for testing purposes. I had known DetectionLab for some time now but hadn’t really used it myself. Though, I had only heard good things.
DetectionLab is a repository which contains a bunch of scripts and tools which automate the process of deploying a test / detection environment. The tools include Packer, Vagrant, PowerShell, Ansible and Terraform. Some of these I know, some I don’t but fortunately this repository supports my laziness to deal with infrastructure – it is well documented and automated. This was built for Virtualbox and VMware workstation/fusion first, but the nice people have since created instructions for some hypervisors/cloud too: HyperV, ESXi, AWS, Azure. There is also some help provided for LibVirt and Proxmox but not official support. My first instinct was of course to go with LibVirt but the instructions aren’t amazing for it. So this was a little disappointing and I was thinking how to solve this thing. Two options; go with other hypervisor or tinker with LibVirt.
I decided to go with other hypervisor but I didn’t want to get rid of Unraid so here comes the nested virtualization (probably a bad idea, but lazy people do weird things). I enabled the nesting with Unraid and then installed ESXi on top of Unraid and allocated resources to it. After configuring everything as instructed in DetectionLab instructions I launched another VM (Ubuntu server) which I used to launch the actual deployment towards the ESXi. After this, I have to say, everything went super smoothly! I had the ESXi hosted in the VLAN 3. At this stage I also configured the network so that ALL the network traffic is routed through a VPN from this VLAN. Before this, only part of the traffic was (selected hosts). This of course meant that I can’t access the network from management network anymore as packets are routed to the wrong address. This I am currently solving, but can work around the issue already.
Now I had a great environment from DetectionLab where I have sysmon, Powershell transaction logging, other EVTX logging, Splunk + Velociraptor amongst other things on which I can also run malware if I want to. Also, ESXi supports snapshots which makes recovering much easier when the testing has been done – this was a little painful with Unraid although doable. This has so far worked VERY nicely and also I like that I can at any given moment redeploy the whole DetectionLab with ease. I also do know that this is NOT the most secure way of doing things, it would be much more secure to have a completely different box for testing with it’s own internet connection. However, I don’t fancy any extra computers in my home eating up electricity and I do like to have multipurpose for the fairly expensive desktop that I bought.
I highly recommend this sort of setup for anyone interested in testing. If not willing to play around the network related things, at least take a look at DetectionLab if you are interested in building a home lab for detection purposes. It is amazing and makes at least my life a lot easier. It saves a lot of time as you don’t have to configure everything yourself, simply deploy the things using the ready-made repo. You can also add further components yourself.
My experiences from this setup have been very positive. The Unraid itself is stable and the only issues that I’ve encountered are more to do with me living in the countrier side where there are power cuts occasionally (I don’t have an UPS, at least yet). The Unraid itself have been stable and I’ve been able to do all the things that I wish for the home lab with relative ease. The setup allows me to run a single quite powerful desktop and have several workloads on top of it, getting more out of the expensive hardware. The daily driver has worked very well for my use case. The performance is great and if you do not know that you are running a virtual machine you can’t really tell from the user experience. Of course, there have been some challenges in my road which are added below.
Issues encountered so far:
- Some anticheats do check for virtual machines and kick you out if you are gaming on a VM. I am sure that this could be worked around by spoofing the fact that it is a VM but I have not bothered – I just do not play the games that don’t work when running from a VM.
- Detection Lab was discussed already but it showed well that if not using the most mainstream hypervisors some things can be more painful
A little different subject this time, hopefully enjoyable to some. I got the idea for the post from my Podcast partner in crime, so thanks for you Juuso for pitching the idea to me!