the white box depicted is my home server, built from used parts except the 3x4TB WD Red HDDs inside. Very proud to see what this little mATX box can do, with only 16gigs of ram!

running truenas scale and a lot of k3s pods, it works like a charm 🫡

  • terribleplan@lemmy.nrd.liEnglish
    1·
    1 year ago

    Business in the front:

    • Mikrotik CRS2004-1G-12S+2XS, acting as a router. The 10g core switch plugs into it as well as the connection to upstairs
    • 2u cable management thing
    • Mikrotik CRS326-24S+2Q+, most 10g capable things hook into this, it uses its QSFP+ ports to uplink to the router and downlink to the (rear) 1g switch.
    • 4u with a shelf, there are 4x mini-pcs here, most of them have a super janky 10g connection via an M.2 to PCIe riser.
    • “echo”, Dell R710. I am working on migrating off of/decomissioning this host.
    • “alpha”, Dell R720. Recently brought back from the dead. Recently put a new (to me) external SAS card into it, and it acts as the “head” unit for the disk shelf I recently bought.
    • “foxtrot”, Dell R720xd. I love modern-ish servers with >= 12-disks per 2u. I would consider running a rack full of these if I could… forgive the lack of a label, my label maker broke at some point before acquiring this machine.
    • “delta”, “Quantum” something or other, which is really just a whitelabeled Supermicro 3u server.
    • Unnamed disk shelf, “NFS04-JBOD1” to its previous owner. Some Supermicro JBOD that does 45 drives in 4u, hooked up to alpha.

    Party in the back:

    • You can see the cheap monitor I use for console access.
    • TP-Link EAP650, sitting on top of the rack. Downstairs WAP.
    • Mikrotik CRS328-24P-4S+, rear-facing 1g PoE/access switch. The downstairs WAP hooks into that as well as the one mini-PC I didn’t put a 10g card on. It also provides power (but not connectivity) to the upstairs switch. It used to get a lot more use before I went to 10g basically everywhere. Bonds 4x SFP+ to upllink via the 10g switch in front.
    • You can see my cable management, which I would describe as “adequate”.
    • You can see my (lack of) power distribution and power backup strategy, which I would describe as “I seriously need to buy some PDUs and UPSs”

    I opted for a smaller rack as my basement is pretty short.

    As far as workloads:

    • alpha and foxtrot (and eventually delta) are the storage hosts running Ubuntu and using gluster. All spinning disks. ~160TiB raw
    • delta currently runs TrueNAS, working on moving all of the storage into gluster and adding this in to that. ~78TiB raw, with some bays used for SSDs (l2arc/zil) and 3 used in a mirror for “important” data.
    • echo, currently running 1 (Ubuntu) VM in Proxmox. This is where the “important” (frp, Traefik, DNS, etc) workloads run right now.
    • mini-pcs, running ubuntu, all sorts of random stuff (dockerized), including this Lemmy instance. Mounting the gluster storage if necessary. They also have a gluster volume amongst themselves for highly redundant SSD-backed storage.

    The gaps in the naming scheme:

    • I don’t remember what happened to bravo, it was another R710, pretty sure it died, or I may have given it away, or it may be sitting in a disused corner of my basement.
    • We don’t talk about charlie, charlie died long ago. It was a C2100. Terrible hardware. Delta was bought because charlie died.

    Networking:

    • The servers are all connected over bonded 2x10g SFP+ DACs to the 10g switch.
    • The 1g switch is connected to the 10g switch with QSFP+ breakout to bonded 4x SFP+ DAC
    • The 10g switch is connected to the router with QSFP+ breakout to bonded 4x SFP+ DAC
    • The router connects to my ISP router (which I sadly can’t bypass…) using a 10GBASE-T SFP+.
    • The router connects to an upstairs 10g switch (Mikrotik CRS305-1G-4S+) via a SFP28 AOC (for future upgrade possibilities)
    • I used to do a lot of fancy stuff with VLANs and L3 routing and stuff… now it’s just a flat L2 network. Sue me.