If you selfhost any applications and have wondered how you can best access those applications from outside your network, you’ve undoubtedly come across Cloudflare. Cloudflare offers two services in particular that might be attractive to homelabbers and selfhosters, reverse proxying and “tunnels”.

Both of these offer some degree of benefit; proxying potentially offers a degree of Denial of Service (DoS) ¹ protection and use of their Web Application Firewall (WAF), and the tunnel additionally adds the benefit of circumventing the host’s firewall and port-forwarding rules. To sweeten the deal, both of these services are offered “for free”, with the proxying actually being the default option on their free plans.

Lets examine these benefits briefly and see why they’re as popular as they are.

Advantages

DoS Protection

By virtue of being such a massive network, Cloudflare’s proxies is able to absorb a massive number of requests and block those requests from reaching their customer. You can think of it as a dam, when a massive flood of requests come in, Cloudflare is in front of the customer holding back the waters.

WAF

Cloudflare’s WAF allows customers to set firewall rules on the proxy itself, which can block or limit outside requests from ever reaching the customer, similar to the DoS protection.

Bypassing Firewalls (tunnel)

Many firewalls are configured to allow outgoing connections and block incoming connection. This is great for security, since it prevents others from accessing your stuff from the public internet, but this can be a problem if you want people to access your website or services. Cloudflare’s tunnel service takes advantage of the “allow outgoing” rule to establish a connection from the host server to Cloudflare, and then allowing incoming requests to tunnel through this connection and access the host server.

Bypassing Port-Fowarding rules (tunnel)

If you’re hosting a service from behind an IPv4 address, there’s a good chance that you’re using a technology called Network Address Translation (NAT) ². This allows you to have multiple systems running behind a single public IP address. This concept can be taken further by Carrier Grade (CG) NAT, where the ISP applies NAT rules to their customers. The problem with NAT is that it becomes difficult to make an incoming request to a machine behind NAT. To an outside system, all of the machines behind the NAT layer appears to have a single address. To partially overcome this, a rule needs to be configured that says “when a request comes in for port X, forward it to private address Y, port Z.” This allows, in a limited way, an outside connection to make use of the NAT address and a particular port to send traffic to a specific machine behind the NAT layer. In the case of CGNAT however, these rules would need to be created by the ISP, a service which most do not offer. Similar to taking advantage of the outgoing firewall rules, a tunnel also uses outgoing ports to avoid the need to create specific port forwarding rules and then directly tunnels a list of ports straight to the host machine.

On the surface, this all sounds like a great product; it allows smaller customers to selfhost even if in a sub-optimal environment.

Considerations

Cloudflare’s proxy and tunnel models both have several aspects, however, which might make some want to reconsider their use.

DoS Protection

DoS attacks can vary in size and scope and their impact will depend not only on the volume of the attack, but also on the capabilities of the targeted system. For example, for a small home server, a few hundred connections per second might be enough cause a denial of service. On the extreme opposite, Cloudflare’s servers routinely handle an average of 100,000,000 requests per second without issue. With those numbers in mind, a DoS on a small home server might be so small that it goes unnoticed by Cloudflare’s protections. I do have to say “might” here, though, because I could not find a definitive answer to how Cloudflare determines what a DoS attack looks like. I would expect, however, that the scale of a home server would be so small compared to the majority of Cloudflare’s customer base that an effective DoS attack would not look significantly different than “normal” traffic.
Lastly, and quite importantly, this protection only exists for attacks targeted at a domain name. Domains exist to make the internet easier for humans, for a computer script or bot, an IP address is far simpler. If an attacker targets the hosts IP address directly, that attack will completely bypass whatever protection Cloudflare was providing. If you’re behind CGNAT, this means the attack will target the ISP, but if you have a publicly addressable IP address, that attack is directly targeting your home router.

Security

Cloudflare offers these services for free, and as the saying goes: “If you are not paying for it, you’re not the customer; you’re the product.” Cloudflare advertises that their proxy and tunnel services can optionally provide “end-to-end” (e2e) encryption. In the traditional and commonly used definition, this means that traffic is encrypted by the client (your or your users) and decrypted by the host (your server). Under the normal definition, no intermediary device can decrypt and read the traffic.

Cloudflare, however, uses the term a little differently, as you can see in this graphic.

Instead of providing traditional e2e, Cloudflare acts as a “Man in the Middle” (MITM), receiving the traffic, decrypting it, analyzing it, and then re-encrypting it before sending it along. Cloudflare does this in order to provide their services; they collect and store the unencrypted data in order to apply WAF rules, analyze patterns, and so on.
Now, Cloudflare is a giant company with billions in contracts; contracts they could potentially lose if they were found to be misusing customer data. They wouldn’t benefit by leaking your nude photos from your NextCloud instance or exposing your password for Home Assistant, but you should understand that by giving them the keys to your data, you are placing your trust in them. This MITM position also means that, theoretically, Cloudflare could alter your content without the you (or your users) knowing about it. Normally this would cause a modern browser to display a very large “SOMEONE MIGHT BE TRYING TO STEAL YOUR DATA” warning, but because you are specifically allowing Cloudflare to intercept your data, the browser has no way of confirming whether the content actually came from your server or Cloudflare themselves.
Cloudflare does have a privacy policy, which explains exactly how Cloudflare intends to use your data, and a transparency report, which is intended to show exactly how many times each year Cloudflare has provided customer data to Government entities.

The warning you would normally get during a MITM attack:

Content Limitations

Lastly, Cloudflare, by default, only proxies or tunnels certain ports. If you want to forward an unusual port through Cloudflare, like 70 or 125, you would need to use a paid account or install additional software. Cloudflare's Terms of Service also limit the type of data you can serve through their free proxies and tunnels, such as prohibiting the streaming of video content on their free plans.

Alternatives

Are there other ways to get similar benefits?

• If only a limited number of users need access to your server or network and you have a public IP address, the best solution by far is to use a VPN service, such as Wireguard. Wireguard allows you to securely access your network without exposing any attack surface for bots or malicious actors.
• If you don’t have a public IP, there’s a service called “Tailscale” which uses the same outgoing-connection trick to bypass firewalls and CGNAT, but instead of acting as a MITM privy to all of your traffic, Tailscale simply coordinates the initial connection and then allows the the client and host to establish their own secure, encrypted connection.
• If you do want to expose a service to the world (like a public website) and you have a public IP address, you can simply forward the needed port/s from your router to the server (typically 443 and possibly 80).
• If you want your service to be exposed to the world , but are stuck behind CGNAT, then a Virtual Private Server (VPS) is a potential option. These typically have a small cost, but they provide you with a remote server that can act as a public gateway to your main server. They can also provide a degree of DoS protection, since they’re using a much larger network and you can simply turn off the connection between the VPS and your server until things calm down.
  
  

Closing thoughts

Cloudflare offers some great benefits, but if you're particularly security-minded, you may want to look into alternatives. Even though Cloudflare is trusted by numerous customers around the world, you still have to decide if you want to trust them with your data. While there are other alternatives, though, Cloudflare's offerings are mostly free and comparatively easy to use.

2. NAT translates private IP addresses in an internal network to a public IP address before packets are sent to an external network.

This blog is also available via gopher! And here's how:

Gophernicus

As I mentioned in my previous post, my gopher server of choice is Gophernicus. One of the benefits of Gophernicus is that it will automatically generate gophermaps based on the files in it finds in a directory. This means that adding entries is as simple as dropping a file into the directory. The next time that server is accessed, the new file will appear automatically.

Mirroring

All that remains is finding a way to easily add files to the gopher directory. Since I already have this blog, I decided the first thing I wanted to do was to mirror these posts into the gopherspace. This blog uses Dotclear, which provides a plugin to access an RSS feed for the blog. By fetching (and then parsing) this feed, I can export the contents into text files accessible to gopher. I wrote a perl script to accomplish this and created systemd units to execute that script on a recurring schedule to pull in new entries. The full source code is available on my github. The script uses LWP:Protocol:https to fetch the RSS feed and XML::Feed to extract the title, date, and body of each entry. The date and title are used as the file name, and the body is reduced down to plain text and then written to the file.

If you'd like to use the script, it should, probably, maybe work with other RSS and ATOM feeds, but some feeds can be a bit loosey-goosey about how they handle their XML, so no guarantees.

Background

After setting up my home server, I found that I wanted a way to securely access and manage my system when I was away from home. I could have opened the SSH port to the internet, but the idea of having such a well-known port exposed to the internet made me wary not only of attempts to gain unauthorized access, but even just simple DOS attacks. I decided the far better option was to use a VPN, specifically, Wireguard.

Wireguard is a simple, lightweight VPN, with clients for Linux, Android, Windows, and a handful of other operating systems. In addition to the security offered by its encrypted VPN tunnel, Wireguard is also a "quiet" service; that is, the application will entirely ignore all attempts to communicate with it unless the correct encryption key is used. This means that common intelligence gathering tactics, such as port scans, won't reveal the existence of the VPN. From the perspective of an attacker, the port being used by Wireguard appears to be closed. This can help prevent attacks, including DOS, because an attacker simply won't know that there's anything there to attack.

Now, when I set about trying to install and configure Wireguard, I found that many of the online guides were either overly complex, or so incredibly simple that they only provided a list of commands to run without ever explaining what those commands did. The overly complex explanation turned out to be too confusing for me, but on the other hand, I'm also not someone to run random commands on my system without understanding what they do. I did eventually figure it out, though, so I thought I'd try writing my own explanation to hopefully offer others a middle ground.

Introduction

Wireguard uses public key encryption to create a secure tunnel between two "peer" machines at the network layer. It's free, open source software, and is designed to be simple to use while still providing a minimum attack surface.

In essence, wireguard creates a new ipv4 network over an existing network. On a device running wireguard, the wireguard network simply appears to be an additional network interface with a new IP address. On a typical laptop, for example, one might have several network interfaces, like lo, eth0 and wlan0, with an additional interface for wireguard appearing as wg0.

Installation

Installing Wireguard is typically done through your package manager, i.e. apt, dnf, pacman, etc, or optionally installed via a container package, via podman, flatpack, snap, appimage, or docker.

For example, apt install wireguard-tools. Depending on your system setup, you may need to preface this and some of the other commands in this walkthrough with the word "sudo".

Configuration

Once installed, wireguard will need to be configured. Wireguard considers devices to be peers, rather than using a client/server relationship. That means that there is no "primary" device that others connect to, rather, they can all connect to each other, just like a typical LAN. That said, for the sake of explanation, it is sometimes easier understand if we use the server/client labels, despite the devices not technically functioning in that relationship. In other words, I'm going to say "server" and "client" because I find it less confusing than saying "peer A" and "peer B".

We'll start on the "server" (aka. Peer A, not technically a server, yada yada) by running wg genkey | tee private.key | wg pubkey > public.key.
wg genkey produces a random "Private key", which will be used by the peer to decrypt the packets it receives. tee is a basic linux command that sends the output of one command to two destinations (like a T shaped pipe). In this case, it is writing the private key to a file called "private.key" and also sending the same data to the next command wg pubkey. This command creates a "public key" based on the "private key" it was given. Lastly, > is writing the output of wg pubkey to the file public.key. After running that command, we now have two files: private.key and public.key.

We can view the actual key by running cat private.key or cat public.key.

We'll repeat those same steps on the "client".

Next, let's decide what IP network address we want to use for the wireguard network. Remember that this needs to be different from your normal local network. There are several different network ranges available, but the most common is 192.168.1.0/24. Those first three "octets", 192, 168, and 1 define the network in this IP address, while the last one defines the device's address within that network. So, what's important here is that if your primary network is 192.168.1.0/24, that we assign the wireguard network a different address, like 192.168.2.0/24. By changing that 1 to a 2, we're addressing an entirely different network.

Now, we'll create the configuration files themselves. On the server, we'll create a configuration file by using a text editor, such as nano. nano /etc/wireguard/wg0.conf will create a file called wg0.conf and open it for editing. The name of this configuration will be the name of the network interface. It doesn't need to be wg0, but that's a simple, easy to remember option.

Here is an example configuration file for the server:

[Interface]
PrivateKey = (Your server's private key here)
Address = 192.168.2.10/24  
ListenPort = 51820

[Peer]
PublicKey = (Client's public key here)
AllowedIPs = 192.168.2.0/24
Endpoint = (see below)

PrivateKey is one that we generated on this server.

Address is the IP address that we're assigning to this server on the wireguard interface. This is our wireguard network address.

ListenPort is the "port" that wireguard is going to "listen" to. If you're not familiar with ports, think of it this way: an IP address is used to address packets to a specific device, and a port number is how you address those packets to a specific application on that device. In this case, Wireguard will be "listening" for packets that come in addressed to port number 51820. That number is somewhat arbitrary, but for uniformity, certain applications tend to use certain port numbers and 51820 is typical for wireguard.

The [Peer] section defines the other system/s on the network, in this case, our "client" system.

PublicKey is the public key created on the client device.

AllowedIPs tells wireguard which IP address it should forward to this peer. Here, we're telling it that any packets addressed to the 192.168.2.0/24 network should be forwarded out to this peer (our client system).

Endpoint is how we reach this peer. Because wireguard creates a network on top of an existing network, we need to tell wireguard how to get to the peer on the existing network before it can use the wireguard network. If you're connecting two devices on your LAN, we would just enter the normal IP address of the peer, such as 192.168.1.123. If you're connecting to another device over the internet, you will need to have a way to address packets to that device, such as a public IP address or a domain name. This will be specific to your network situation. Following the IP or domain name is a colon and the port number. For this example we'll assume the other machine is on your LAN at address 192.168.1.123 so we can just enter the IP address and the port number, like so: 192.168.1.123:51820. Lastly, save the file.

We'll then reverse this process on the 'client', assigning it a different IP address on the same network, such as 192.168.2.10/24 and using the clients PrivateKey and the server's Public Key. Once we've completed the configuration files on both device, we can use the command wg-quick up wg0, telling wireguard to start (bring up) the wg0 interface we just configured.

Final Notes

It should be noted that the "Endpoint" setting is only required on one of the two peers. This can be useful if you want to use wireguard while you're away from your home. My "server" at home may keep the same public IP all the time, but my portable laptop will have a different public IP address every time I move to a new network. In this case, I would remove the "endpoint" setting entirely from the configuration file on the server, and only use that setting on the laptop. Remember, the one system needs to be able to connect to the other system over an existing network before wireguard can create the VPN connection between them. Most of the issues a user might encounter while trying to set up wireguard isn't actually related to wireguard itself, but rather the underlying network. Firewalls and Network Address Translation (NAT) are the most common causes of problems, but the steps to address these issues will vary significantly depending on your network situation.

It recently occurred to me that as I update each Linux container or VM, I'm downloading a lot of the same files over and over again.  While the downloads aren't huge, it still seems wasteful to request the same files from the repo mirrors so many times... So why not just download the update once and then distribute it locally to each of my systems?  

That's the purpose of a caching proxy.

I chose apt-cacher ng as it's very simple to setup and use, so I spun up a dedicated LXC and installed apt-cacher ng via apt. Once it was up and running, it was just a matter of following the included documentation to point all of my other systems to that cache.

After upgrading just a couple of systems, I can already see the cache doing it's job:

Those "hits" are requests that were able to be fulfilled locally from the cache instead of needing to download  the files from the repo again. Since this is caching every request, it actually becomes more efficient the more that it's used, so hopefully the efficiency will increase even more over time.

So what exactly is happening?

First, this is not a full mirror of the Debian repos. Rather, apt-cacher ng acts as a proxy and cache. When a local client system wants to perform an update, it requests the updated packages from apt-cacher instead of the Debian repo directly. If the updated package is available in apt-cacher's local cache already, it simply provides the package to the requesting client. If the package is not in the local cache, then the proxy requests the package from the repo, provides that package to the client, and then saves a copy of the package to the cache. Now it has a local copy in case another system requests the same package again.

Some packages, like Crowdsec, are only installed on a single machine on my network, so the cache won't provide a benefit there. However, since most of my systems are all running Debian, even through they may be running some different services,  they will still all request a lot of the same packages as each other every time they update, like openssh or Python.  These will only have to be downloaded the very first time they're requested, and all of the subsequent requests can be filled from the proxy's local cache.

Do you use a cache in your homelab? Let me know below!

So, while surfing the web and reading up on how I wanted to configure my new BananaPi R3, I encountered a sudden issue with my network connection:

It seemed like I could reach the internet, but I suddenly lost access to my home network services. I tried to SSH into my webserver and was met with a "Permission denied" error.  Since I had earlier attached an additional USB NIC to connect to the BPI, I thought that perhaps the laptop had gotten the interfaces confused and was no longer tagging packets with the correct VLAN ID for my home network. Most of my servers are configured to refuse any requests that don't come from the management VLAN, so this explanation made sense. After poking around in the network settings, all of the VLAN settings appeared correct, but I did notice that the link was negotiated at 100 mbs, instead of the usual 1000. I tired to reconfigure my network settings, manually setting the link to 1000 mbs, resetting interfaces, changing network priorities, etc.  I then tired the classic "reboot and pray" technique, only to find that my wired network connect was down entirely. I wasn't receiving an IP from the DHCP server and the laptop kept reporting that the interface was UP, then DOWN, then UP again, then DOWN again.

Now I started to think that perhaps the issue was hardware related. My usual NIC is built into the laptop's dock, so I thought I would try power cycling the dock itself. This didn't seem to have any effect, besides screwing up my monitors placement and orientation. My next thought was that there might be an issue with the patch cable. "Fast Ethernet" (100mbs) can theoretically function on a damaged cable, so that might explain the lower link speed, and if the damage is causing an intermittent issue, that could also explain the up/down/up/down behavior.  

Being the smart homelabber that I am, I disconnected both sides of the cable and connected my ethernet cable tester. All 8 wires showed proper continuity, though, suggesting that the cable was fine.  When I plugged the cable back in, however, I noticed that I was suddenly getting the normal 1gbe again, but the link was still going up and down. This lead me to the conclusion that the cable likely was the issue, despite it passing the cable test.  I tried replacing the cable entirely with a different one, and found that I now had a stable, 1gbe connection, an IP addresses, and I could now access my network like usual.

Looking back, I think replacing the cable should have been troubleshooting step 1 or 2.

Also, in retrospect, there were some clues that might have let me fix the issue before this point, if I had only put the pieces together. I had noticed another day that the link speed had dropped to 100mbs, but it seemed to correct itself, so I ignored it instead of investigating. While working, I found that Zoom and other applications had started to report my internet connection as "unstable" and that a "speedtest" showed that my internet bandwidth to be drastically lower than it used to be. I assumed this was due to my ISP just being unreliable, since reduced speeds and entire outages are not unusual where I live. 

In hindsight, I think these were all indications that there was a level 1 issue in my network. In the future, I'll have to remember to not over-think things, and maybe just try the simplest solutions first.

I've been using a consumer router from from 2016 (with OpenWRT hacked onto it) all the way here in 2024, and felt that it might finally be time for an upgrade. I settled on a BananaPi R3 because it was a reasonable price and seemed like it would be a fun project.

Here's the bare board as received:

You can see most of the physical features in this photo, including a USB3 port, two SFP ports, 5 RJ45 ports, an m.2 slot for a cellular modem, and a 26-pin GPIO header. On the bottom, there's also a m.2 slot intended for nvme storage, as well as slots for a micro SD and a micro SIM.  The CPU is a quad-core ARM chip paired with 2gb of RAM, and there's a handful of flash chips, providing NAND, NOR and eMMC.   Quite a lot of options!

My plan is to install OpenWRT to the NAND storage. I suspect the nvme might be useful if I wanted to run a small file server or something, but that's not in the plan for now.

The first step I took in assembly was to apply some thermal pads to the chips and then attach a cooler and fan.

The thermal pads are "OwlTree" brand, but I don't have any specific preference to them, I just happen to already have them on-hand from a previous project. The CPU is a 0.5mm pad applied, and I applied 1.5mm pads to the remaining chips.

After applying  pads to all of the chips, I attached the cooler and plugged in the fan.

The next step was to install the board into the case. I went with the official BPI-R3 case. The quality is surprisingly nice and looks great once assembled. After installing the board I then installed the pigtails for the eight (yes, eight) antennas and applied some basic cable management.

Now, I can't finish putting the case together quite yet, since I'll need access to the UART pins to install Openwrt to the NAND flash. The  UART header can be seen on the right side of this photo, but there is no way to access it once the case is assembled.

But, that's enough for today. I'll post an update once I make some progress towards getting OpenWRT flashed.